Tag: vendor

Hubris in IT

Image

 

It would seem that “Pride cometh before a fall” is something that is lost on most people who work in IT.

 

As someone who has been working with computers from about the time Bill Gates was buying an operating system from some poor guy in Washington State, and Steve Jobs was phone phreaking; There is just not much that escapes me.

 

I was doing some consulting for a company that was simply put together with bailing wire and scotch tape.  They had a huge pipe to the internet and were getting a dribble through by the time it hit the desktop.

 

Loading WireShark (a free protocol analyzer) examining the broadcast packets it was easy to see why.  The OS was literally working with NetBIOS to route packets.

 

A quick examination of the “server room” found the switches all tied together with Fiber and, patch cords going from one switch to another causing untold amount of routing loops etc. While the picture above is a stock photo the room in question looked very much like this.

 

My job however was not to fix their networking issues as this was the task of the guy I was “helping.”  He was the System Administrator.  I sent e-mails to him alerting him to my findings so he could take the appropriate steps, which for some reason he discounted and did not do.

 Image

The weeks went on and the problems persisted several times a day where people were kicked off of the network or files were corrupt or lost etc.  His response/fix was to release and renew the IP address.  Putting one band aid on the problem day after day I guess gave him a sense of accomplishment but the problems were looming and like the 500 pound gorilla in the closet, soon to get out.

 

One of the things that I learned many years ago is to work with VARS.  Value added resellers have years of experience to draw upon.  They know which products are buggy and to stay away from and which are tried and true.  If you are a business don’t try and save money via internet stores as you will get what others can’t sell for one reason or another.  They are on sale for a reason…

 

When I asked him for his vendor contact list to include in his DR plan, there were no VARS on the list.  Everything was from internet companies or local retail locations.  He in fact had no fallback plan if the $hit hit the fan.

 Image

The hardware purchased looked as much.  There were no standards anywhere.  There were high end SANS tied to cheap switches.  The workstation of choice was whatever he got a good deal on making mass deployment of anything just about impossible.  Hardware was way past its lifecycle and the list just went on.  Because of his pride; he was not willing to listen to anyone regarding anything IT.  If he does not change it will be his undoing.

 

This is not my first rodeo and certainly not my first encounter with arrogance.  As a manager I can deal with it, as a consultant one must work around it and if it becomes too big of an impediment, bow out.  There is no reason to sully your name with a situation like this when the outcome will likely somehow be your fault.  

 

Always hire people smarter than you are and have the humility to acknowledge that you are not the end all be all.  There is simply too much information out there to know it all.  Wisdom is; knowing that you need help and to leverage VARS and consultants is simply smart.

 

-Best to you and those that you care about!

Big Red Button or Time to Panic!

Image

Nothing says “push me” like a big red button.  One of the office supply stores even created a big red button that says “EASY” on it, to advertise how they can simplify your work life.

One of the data centers that I was responsible for had such a button.  It was covered with a little plastic rectangular box that said “emergency shut off” on it.

I have been in many data centers during my career.  There were several that had a big red button by the door with it sole purpose to release the magnetic latch on the door, to open it.

Like any other location, security in a data center is paramount.  Not only are network security firewalls and such important but physical security as well.  Only those who needed access to the data center, could access it with their security card.  Not even the CEO had access as he did not need it.  Their entrance was logged and in fact throughout the building one could forensically track any employee’s movements as this card was necessary to gain access to just about anywhere.  With the technology available today, I could design such a better system, but that is beyond the scope of this document.

One day, a vendor was visiting with a proposed solution to a problem.  Like any other vendor, if access to the data center is required, they are escorted at all times by one of, if not more of my staff or me.  The data center was in the middle of a retrofit and redesign while keeping the company running in parallel.  (This is much like changing the tires on a race car while it is moving down the track.)  On their way out of the data center, just as quickly as anything, the sales guy in front reaches up to the left of the door pops the cover open and pushes the big red button!  By the time that the sound of “NO” had left my lips, there was an eerie quite in the room.

The chain of events that this action triggered, were phenomenal.  Lights went off, the air handling unit went off, the Battery back-ups clicked on and for the moment; it looked as though the carefully engineered back-up power supplies were working.  I should mention that the look on this guys face was priceless, and I am just about certain that he had to change his shorts afterwards.  It dawned on me that no one had tested this button, and nobody knew where all of the circuit breakers were; well almost no one.  As I was the one that specified the power requirements for this data center and oversaw the installation of the new transformer, I knew where the main breaker was.  Within moments I had most of the power back on however; there was one legacy system that was still not on main power.

In another closet in another part of the building were still more circuits for this room.  I did not have a key to this and getting building maintenance involved was time consuming as they typically think like union employees; (don’t care if the place is on fire, when it is time for a break, they take it.)  Before the UPS was totally drained for that system I had gained access to that closet and found one tripped breaker.

I had inherited a mess of a data center that was put together on a shoestring budget.  Not because the company could not afford to do it right, their boss was cheap beyond reason.  They had cut corners at every place they could, including splicing old type 3 wires to cat 5 wires and running 16mg token ring over it.  They could not understand why 5250 and 3270 traffic would constantly be garbled and why connections to the server would be dropped frequently.  When I say spliced, I literally mean wires twisted together and a wad of electrical tape stuffed in the wall and or ceiling.  (Another story)

It did not take me long to get that circuit changed over and documented with everything else.  I also got to check off the list “test emergency shut down.”

Moral of the story; if you have a big red button, find a time to test it.  Secondly make certain that the button is labeled in big white letters on a red sign etc EMERGENCY SHUT OFF!

I am a stickler for documentation, which IT personnel are loath to do.  A living document should exist within each and every company that explains the ins and outs of everything, so if need be, someone else can take over.  It is part of the audit process for a disaster recovery plan and is one of the deliverable s.

-Best to you and all those that you care about!