Big Red Button or Time to Panic!

Image

Nothing says “push me” like a big red button.  One of the office supply stores even created a big red button that says “EASY” on it, to advertise how they can simplify your work life.

One of the data centers that I was responsible for had such a button.  It was covered with a little plastic rectangular box that said “emergency shut off” on it.

I have been in many data centers during my career.  There were several that had a big red button by the door with it sole purpose to release the magnetic latch on the door, to open it.

Like any other location, security in a data center is paramount.  Not only are network security firewalls and such important but physical security as well.  Only those who needed access to the data center, could access it with their security card.  Not even the CEO had access as he did not need it.  Their entrance was logged and in fact throughout the building one could forensically track any employee’s movements as this card was necessary to gain access to just about anywhere.  With the technology available today, I could design such a better system, but that is beyond the scope of this document.

One day, a vendor was visiting with a proposed solution to a problem.  Like any other vendor, if access to the data center is required, they are escorted at all times by one of, if not more of my staff or me.  The data center was in the middle of a retrofit and redesign while keeping the company running in parallel.  (This is much like changing the tires on a race car while it is moving down the track.)  On their way out of the data center, just as quickly as anything, the sales guy in front reaches up to the left of the door pops the cover open and pushes the big red button!  By the time that the sound of “NO” had left my lips, there was an eerie quite in the room.

The chain of events that this action triggered, were phenomenal.  Lights went off, the air handling unit went off, the Battery back-ups clicked on and for the moment; it looked as though the carefully engineered back-up power supplies were working.  I should mention that the look on this guys face was priceless, and I am just about certain that he had to change his shorts afterwards.  It dawned on me that no one had tested this button, and nobody knew where all of the circuit breakers were; well almost no one.  As I was the one that specified the power requirements for this data center and oversaw the installation of the new transformer, I knew where the main breaker was.  Within moments I had most of the power back on however; there was one legacy system that was still not on main power.

In another closet in another part of the building were still more circuits for this room.  I did not have a key to this and getting building maintenance involved was time consuming as they typically think like union employees; (don’t care if the place is on fire, when it is time for a break, they take it.)  Before the UPS was totally drained for that system I had gained access to that closet and found one tripped breaker.

I had inherited a mess of a data center that was put together on a shoestring budget.  Not because the company could not afford to do it right, their boss was cheap beyond reason.  They had cut corners at every place they could, including splicing old type 3 wires to cat 5 wires and running 16mg token ring over it.  They could not understand why 5250 and 3270 traffic would constantly be garbled and why connections to the server would be dropped frequently.  When I say spliced, I literally mean wires twisted together and a wad of electrical tape stuffed in the wall and or ceiling.  (Another story)

It did not take me long to get that circuit changed over and documented with everything else.  I also got to check off the list “test emergency shut down.”

Moral of the story; if you have a big red button, find a time to test it.  Secondly make certain that the button is labeled in big white letters on a red sign etc EMERGENCY SHUT OFF!

I am a stickler for documentation, which IT personnel are loath to do.  A living document should exist within each and every company that explains the ins and outs of everything, so if need be, someone else can take over.  It is part of the audit process for a disaster recovery plan and is one of the deliverable s.

-Best to you and all those that you care about!

Advertisements

2 thoughts on “Big Red Button or Time to Panic!

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s