Power Outages Do Happen

Posted July 10th, 2007 by

Finally had one today. It was great. The generators kicked on. The building-wide UPS worked. The NOC and SOC still had power. The other working areas ran out of AC, power, and people before I got there.  No problem, the engineers can go elsewhere to work as long as the operations people still have what they need.

Before you ask, no I didn’t create the problem as a BOFH DR test, and the outage did not occur immediately after a line like “so, what does this big, shiny, red button labeled ‘EPO’ really do?”.  =)

Similar Posts:

Posted in Risk Management, The Guerilla CISO | 5 Comments »

5 Responses

  1.  Saso Says:

    You say:
    “The other working areas ran out of AC, power, and people before I got there.”

    People? Did you dump the bodies outside and threw extra random pieces of furniture at the doors?[1]

    Big shiny red buttons are the only way to test DR. What is the use of all the DR preparations if people don’t actually pull the plug unannounced? Last I checked, disasters didn’t quite announce themselves way in advance and asked when would be a good time to strike. 😉

    [1] It’s Wednesday here; obligatory zombie reference.

  2.  rybolov Says:

    I didn’t see you come in and kick the generator dead, did I?

  3.  Saso Says:

    Me? Never. Who do you think I am? Dolphin?

    Soooo… are you going to press that big red button in a couple of weeks time, just to see if generator got refuelled after this little outage?

  4.  rybolov Says:

    You don’t understand, when you push the shiny red button, even the generator can’t help you anymore.

    It’s designed for those hopefully rare days when the data center is on fire or knee-deep in water.

  5.  Saso Says:

    Ah, *that* red button. Yeah, I missed the fact you said *EPO* somewhere there.

    Talking about data centres knee-deep in water. Have I ever mentioned a company, a global investment bank, that had world’s first underwater data centre? It was all by accident, of course, but rumour had it that facilities people couldn’t believe their luck when they scored that several-football-fields-sized data centre for next to nothing.

    Except that besides the cost being next to nothing, the facility itself was next to a river. 😉

Leave a Comment

Please note: Comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.

Visitor Geolocationing Widget: