Logicalis and Hurricane Ike

Attention: This is Not a Drill. The Logicalis Cincinnati-based EBOC Stood Eye-to-Eye with Hurricane Ike...and Never Blinked.

Logicalis EBOC operations manager, Tom Sweeney, knew he had a good disaster recovery plan in place. But he didn’t really know how good it was until Hurricane Ike roared through Ohio on Sunday, September 14, 2008, ripping gaping holes through the Cincinnati metropolitan area power grid. By Monday, Ohio Governor Ted Strickland had declared a state of emergency and reported that nearly two million customers were without power in Ohio. There were power lines down, transformers bursting into flames, traffic signals knocked out, and roads closed around the state, while inside the Logicalis EBOC, it was network monitoring and management as usual.

Sweeney summarized the situation, saying, “We had a plan and it worked; and that’s exactly what I expected.” According to plan, a back-up natural gas generator promptly took over from the downed power grid, and kept all of Logicalis’ customer network monitoring and management equipment fully powered. During the 30 hours it took for commercial power to be restored, the generator never went beyond 13% of its capacity. As a matter of course, all customers were immediately notified of the situation outside the EBOC, but there were no interruptions of service.

More than Technology
Three huge monitors dominate one wall in the EBOC, giving the facility the feel of a command center. There is more than technology involved in managed services, however. There are as many as 22 service delivery engineers available 365 days a year. Logicalis service delivery engineers hold among them the highest level of certifications in Cisco, Microsoft, Unix, IP telephony, and other technologies. Some have as many as 20 years of experience in IT operations.

As soon as the storm hit, senior management contacted everyone on the Managed Services team to maintain constant status updates. Not everyone was able to navigate through the devastation to get to work, but according to plan, those who had power at home were able to log in remotely and carry out their responsibilities. The Managed Services team was also available via cell phone or Blackberry, so communication was maintained at all times.

Those who could make it to work safely, swapped shifts with those who could not, and senior management worked 12-hour shifts at the EBOC so that they would be available to make critical decisions. Senior level engineers were also enlisted from Logicalis’ Columbus, Chicago, Phoenix, Seattle, and De Pere offices.

One of the additional responsibilities to be carried out, was the notification of clients with operations in the region where the monitoring equipment showed that they had lost power. As a matter of policy, any time a connection goes dark, Logicalis engineers make calls to the cell phones and private lines of designated individuals, persisting until news of a power outage is confirmed.

Uninterrupted
Meanwhile, across town where Logicalis maintains a co-location site for its clients, a similar level of foresight kept all systems running throughout the storm and its aftermath. A diesel generator was employed here, and according to plan, a full 13 days worth of fuel was on hand to ensure uninterrupted service. Each day that the generator ran, an additional day’s fuel was booked to maintain a 13-day buffer. Logicalis clients in the Cincinnati area who were co-located at this facility, might have lost power at their headquarters, but their IT infrastructure hummed on uninterrupted.

“We don’t keep all our eggs in one basket,” Sweeney notes. Had it been necessary, Logicalis could have switched over to a duplicate site at its facility in Indianapolis that provides full replication of the equipment used to monitor and manage our customers. No plan can cover all contingencies in a catastrophic event like a hurricane – that’s when training and experience count the most. “One thing we all learned from this is to be flexible, and respond to the situation at hand, rather than concentrating on preconceived notions of what could happen,” Sweeney says.

A Post-Mortem Review of the Day
Hurricane Ike descended on Cincinnati, and did reveal merely one shortcoming in the overall disaster plan; while flashlights were available when the lights went out in the EBOC restrooms, no one had thought to hook up the electrical outlet for the coffee pot to the back-up generator. It would have been nice to have been able to make some fresh coffee during those late-night shifts. That oversight, Sweeney says, has since been corrected.