Notes from the Consultant’s Jungle

Data
Center consolidation and Data Center outsourcing are
top of mind for many CIO’s these days. Many companies
have ‘90s vintage IT facilities that not only do not
have the availability to align with the Business’ operating
model but also are struggling to keep up with the power
and cooling demands of contemporary computing systems.
The deployment of multi-core processors and blade-based
systems has pulled the rug out from beneath many a facility
manager. The rapidly growing consumption and cost of
energy due to the Data Center have caused many a CFO
to define facility operational costs as an IT problem.

Whatever
the motivation for the Data Center project, one will
have to become familiar with the spirit of the TIA-942
tier classifications as well as the nuances thereof
to exercise the proper degree of due care necessary
in planning these very expensive projects. In many
cases, the enterprise may be focused on the minutia
of TIA-942 because of the desire to align the Business
with the proper availability of the IT facility, but
this may take place at the expense of pragmatism.

The
Uptime Institute has done a great job of defining
a framework for defining the topology and architecture
of the MEP infrastructure of a data center that should
yield a certain level of facility availability. It
is true however, that some facilities that would score
high on the Tier rating scale fail to achieve the
target availability performance. Similarly, there
are many facilities that would score low on the Tier
rating scale that have exceptional availability performance.
Said another way, there are lots of Tier-2 sites that
exceed Tier-4 availability performance and lots of
Tier 3 and Tier 4 sites that fail to meet their availability
targets.

Why
is this so? Well it’s one thing to have the right
levels of redundant paths and backup components but
it’s another thing to operate them in a way that ensures
high availability. Many data centers have suffered
outages because equipment settings were not what they
were supposed to be. Inadequate equipment maintenance
and operational procedures also account for higher
frequencies of outages and extended downtime. In other
words, it often comes down to the “3 P’s,” that is
People, Processes, and Procedures that ensure the
quality of operation in mission critical facilities.

A
recent survey
of AFCOM members found that 81 percent of respondents
had experienced a data center failure in the past five
years, and 20 percent had been hit with at least five
failures. It’s common to find reports of high profile
data centers that endure unexpectedly long outages because
of failures of backup generators to start, improperly
configured ATS equipment, contamination of diesel fuel,
and many other similar nuances of system configuration
and operation. Sad but true, the majority of failures
in data centers are caused, triggered, or exacerbated
by human error. These exposures are mitigated with a
disciplined operational ethic.

One
will often hear of “SOPS and MOPS.” This is the Standard
Operating Procedures and Methods Of Procedures that
are the operating bible for the facility staff. The
exceptional performance of high availability sites
is a testament to the quality of the staff managing
and operating those sites.

In
2008, Switch
and Data scored an impressive five nines in its
facility uptime and some individual sites even exceeded
that. This surpasses Tier-4 guidance and is a result
of strong people, processes and procedures. It’s important
to move the Tier Level discussion to one of availability,
performance history, and value. The interest in knowing
a prospective colocation site Tier Level is driven
by a desire for a minimum threshold of facility availability
in order to align with Business goals. This can be
accomplished outside of the context of Tiers.

The
right to certify the Tier Level of data centers is
reserved by the Uptime Institute. It is a very expensive
process, and as of this writing there are only 13
Tier-3 and two Tier-4 facilities that have been certified.
While a consulting firm can do a thorough assessment
to determine a facility’s Tier rating, only the Institute
can “certify” it. All of these 15 sites are enterprise
facilities and none are multi-tenant colocation provider
sites. With that said, there are lots of data centers
that are “said to be” of a certain Tier Level, but
the extent to which that is true can only be determined
by thorough analysis of the MEP infrastructure.

Why
is it that colocation providers are not rushing to
achieve Tier certification? The reasons are beyond
the simple cost of undergoing certification. The levels
of MEP infrastructure necessary to achieve the highest
Tier Levels are extremely capital intensive. The cost/benefit
and ROI outcomes are difficult to justify. Given that
equally high availability is achievable by strong
operational ethics by a company in the business of
providing mission critical IT facilities allows exceptional
value to be delivered to the Customer.