Tag: management

Single Point of Failure

One of my recent clients was desiging their data center and during the process of doing so, the CEO had read one of my blogs and called me in to look things over.

It is sometimes awkward for me to go into someone else’s shop and poke around as the relationship between their staff and I, sometimes is seems tenuous, if not adversarial. Not on my part mind you; I am there to help. “I fix things,” that is what I do. I do ask a lot of “why questions.” which I think starts the ball rolling.

The resident staff however is the expert that their boss trusts, so they “the CEO” go along blissfully ignorant of their exposures because he or she “the CEO” is not an IT person. The resident IT people usually don’t encourage an outside person to come in “such as myself,” because of two reasons. I think first and foremost, it is an ego thing. They are the best at what they do and if you don’t believe them, just ask them. Secondly, they don’t want their boss to find out just how bad things are. Or possibly their staff has no clue just how tenuous the situation really is.

If you see any of this this in your data center or wiring closets , you need some help.

When I was working as a Manager, or Director of same, I always welcomed this kind of assistance and frequently brought it in as often times we get mired in the woods; and cannot see the forest for the trees. I know that this is oversimplified but as a manager, I always tried to hire people as smart, if not smarter than myself. As a manager, one gets tied up in the day to day business of the company, as well as the technology; and often times do not see what is in front of them, or, can even loos their objectivity.

If you decide to bring in someone like myself, your staff needs a heads up. They need to be told to make themselves available. They need to know that this person is there because you requested it, are paying for it, and expect them to work with this person as needed. This understanding up front, saves you money.

When planning a move to a new building; one can go in and design and install everything correctly the first time, much more cost effectively than going back in and re-do things later.

Notice the nice manner in how the cables are dressed and notice the Velcro cable ties on not “zip ties.”

Why not Zip Ties you asked?

Poor cable management and “zip ties” lead to this. This could cost you your entire company.

This is the charred remains of someone’s data center.

Having said that, during a data center design one of the things that is often overlooked are single points of failures (SPOF). When designing a data center, one builds redundancy into their infrastructure as to avoid a downtime, or a total shut down. Few people truly understand soft dollars; and how the loss of productivity effects the bottom line.

Most everyone knows of raid, redundant switches, vmware, the cloud, and this is about where it stops.

What about an alternate path to get data in and out of your building? What about an alternate path or source for power?

One of my clients, (a law firm) lost power due to construction in the street below. The UPS only held their servers up for 30 minutes or so and they had no UPS’s on workstations. Guess what, the courts do not care if you can or can’t get to your files, your problem, not theirs! If you cannot feasibly get power to your data center from an alternate location, you have a generator or a warm site.

In their case the power took several days to get back online. The elevators of course were not working so they had to physically remove and re-install their servers in another location trucking them down the emergency stairwell.

Failure to plan is planning to fail!

The same is true of your data lines. Generally one installs two trunks diametrically opposite of the other on either corners or sides of the building. If that is not feasible you look at RF links. Yes they are slower than fiber but slower is better than nothing.

Your individual needs will dictate your level of redundancy needed.

During the design of your data center, each and every risk must be defined and be part of the risk assessment. During a move, a site selection team must evaluate all sorts of factors regarding your data center before the lease is signed, or the building is purchased or even built, if you are going to go that way.

Data Centers large or small all should have the basic fundamentals covered. It is your company after all.

One of my clients had me travel around with their folks to look over prospective sites. They were a little edgy with their people that they had doing this so hiring me to go do this with them hedged their bet. When I arrived at the first location they were ready to sign the lease. They were not happy when I discovered that the ceiling was loaded with asbestos. The extra cost to get someone that was certified to run cable in such an environment was over the top, not to mention that it was in an earthquake prone part of California. There is a reason that the lease was “cheap.” On another site the cable plant had been added to as the previous tenant grew. They has spliced wires in the ceiling, which you just don’t do. Some of these had been spliced to cat 3 wires. I can well imagine what the data throughput looked like and the error rate that those persons suffered.

We start with the basics: power, voice/data, air handling. We look at the hardware required for all of this. We calculate the power requirements and the amount of air-conditioning needed. What about a redundant air conditioner? How about fire suppression? I like to install dust filtration systems, as this investment will extend the life of your equipment. How about remote monitoring of your data center. What about security both physical and well as data?

We look at risk factors, where is it geographically? What historic data can we find on that area regarding disasters? I also look at permit cost, union involvement cost etc. I work with the architect and GC to make certain that all of the I’s are dotted and T’s crossed.

In 35 years of doing this I have never had a “good surprise” and my goal is to keep the customer from having a bad one.

One thing that I recommend is a sensor that “sniffs for smoke.” This sensor triggers an emergency shut down of the UPS if smoke is detected. Why on earth would I want to do this?

“Where there is smoke there is fire.”

Maybe, maybe not however; this equipment runs 24X7 and 2/3 of that time is unattended. Generally, poor cable management which I see in more data centers than not, is the cause for smoke and fire. This sensor shutting the power off to the data center stops the fire in its tracks as it generally will start smoking before actually catching fire. This sensor hooks up to the big red button and serves as the emergency shut down if smoke is detected. It can also be wired to the building security systems and trigger a call to the fire department as well as sound alarms so people can get out of the building.

A lot of data centers use the cheapest fire suppression techniques out there “again designed by a neophyte.” So, a cable rubs raw, starts smoking, catches on fire and the suppression system is activated. Water… Water and electronics do not mix. You now have your entire data center ruined because your “people” were lazy and failed to properly dress the cables and you did not have a modern fire suppression system. The smoke sniffer is the next best thing. Stops the fire before the water starts.

The devil is in the details, and brother there are tons of details. As part of a DR; we cover as many as we can find. Once we think we have it, we hire an outside technical staff to re-create your data center in an offsite location, with your run-book, documentation and backups. We give them everything that we “think” they should need and then we sit back and take notes. You’re CIO and I watch, and take notes. We get them the answers that they need, note them, and then move on until it either works or fails. If it fails, we do a root cause analysis, take corrective action and try it again.

Yes, this is a shameless plug for Sungard.

DR plans generally do not work the first time. There are way too many details to catch them all but, as they do their thing and they hit a stopping point, we find the answer, make notes and then let them proceed until they get everything working as we expect it to be. Then we have a Skeleton crew come in and try to work. We make notes of everything that they run into, fix what we can through the rent a geeks, and press on.

Part of the process is to determine what an acceptable down time is and that can even be broken down into individual systems. “Payroll before sales?”

After that, we go back and alter the run book, provide more software or whatever was needed and then we get another team of rent geeks and do it again. Same process until we get it to such a point that the rent a geek can draw on their own knowledge to fill in any holes.

An inventory of the skill set necessary for the geeks is yet another bit on information to document.

Having a disaster recovery plan, like a data backup strategy; is only as good as your last successful test. Once you have a successful test, it is incumbent upon your people to devise a scheme to keep it updated. I do this through a process called change management.

Is this your disaster recovery plan?

We use rent a geeks as we must assume that your staff has all been part of the disaster. If we can make it work with technical people that you can hire from your VAR then, if there is a disaster and your staff was not involved in the disaster, your chances of a successful recovery are very high.

If you look at my other blogs regarding information technology, there is plenty of good advice out there. If you are in need of a DR plan or someone to assist your technical team get through a migration, a move or what have you; I have over 35 years of experience, and a valid passport. I speak English with a smattering of Geek.. 🙂

Yes, if these were real people and not actors, I could converse with them easily, I actually know what a Higgs-Bosson is

Artist idea of what it would look like.

If you are moving and need a data center designed in the new location that is probably one of the least expensive things that I do; assuming that your documentation is up to snuff. The size of your company and time frame will determine the scope of work (SOW,) and if I will need more than myself to accomplish the task. I work with some fine people here in the DFW area, and they too can travel. If you have a relation with a VAR already, I can work with them. I must admit that I will evaluate them on your behalf, as I do not believe in wasting your money or my time.

My character closely matches this guy although, I am easier going and my vernacular is well suited for all people, most of the time.

There is plenty of good advice here on my site, so feel free to peruse and glean what you may. I do reserve all rights to the information contained herein, so please do not copy or disseminate without permission. Thanks!

-Best

Disaster Avoidance

Consulting as a Disaster Recovery Specialist, I often find things that need to be changed to avoid a disaster, much like a loose rug over a threshold or too many things plugged into one circuit; which would be an issue in your home. In the business world it comes down to security issues both IT related and physical, as well as simple things like a lack of fire extinguishers or the wrong type of fire retardant system in the computer room. I am trained to notice the smallest of details including things like cable management issues.

When Best Practice scenarios are not followed by sys-admins or networking guru’s, they too trigger red flags. There is an art to designing data centers. I have designed and built many over the last 30 years complete from the ground up; from air handling to power requirements to working with ADA compliance issues. I have designed cable management for many companies that include the MDF and IDF’s and working with building management to handle communication through multiple story buildings making sure that they pass fire code. You would be amazed at how many data centers that I walk into that are under wired, lack proper air handling and have a sprinkler head above the equipment! The cable management looks like Spiderman installed it, nothing is labeled, and there is absolutely not one shred of documentation. And the boss / owner is oblivious to the immanent disaster, as he thinks his guys are pretty good!

When business’s start up, often times they don’t contact the brightest and best to build it as they are on a tight budget. When I am called, their data center is generally a candidate for one of those web sites that post “what not to do.” The exercise of unraveling the Gordian knot comes into play before anything can be changed. Many times a family friend is called to assist or the business owner has a home network and thinks that a business network is no different.

When these knots are constructed; most if not always there is limited or no documentation and the original creator has long since abandoned ship as he undoubtedly realized the ice berg ahead was not too far off. To that end there are many land mines that have to be discovered and diffused. This practice is akin to changing the tires on a racecar, while it is going down the track, and part of that track is in no mans land! The catch 22 is that no business can afford down time but, if they don’t address the issues they will have un-planned down time! Un-planned is always much longer than planned, and always more expensive!

As an SME on this and many subjects regarding IT, I can offer many things to mitigate any issues and put them on a road to setting things right. Whether that is working with their current IT staff, or bringing in hired guns to knock it out quickly!

The business must be willing to want to change, and have Executive buy-in as well as buy-in from the local staff. The process can take weeks to months depending upon the situation; but after it is all said and done, procedures and processes are put into place to keep up with change.

Some policies addressed are Change Management, Incident analysis; complete with root cause analysis, documentation with the introduction of the concept of a living document. The run book, what is it and how does it work? Testing the Disaster Recovery plan and then implementing changes from things learned. Other topics include SAM (software asset management,) and of course hardware management including lifecycle, and the budget process.

All too often the CFO or CEO is told that IT needs X thousands of dollars for this, that, or the other thing; not because it is a new project but because something failed! With proper asset management this can be mitigated greatly and things can be budgeted for.

Much like any other audit, I don’t guarantee anything will be pleasant other than the knowledge that when it is done you will have the documentation you need, your network will be running at peak efficiency and it will be secure. Depending upon your growth and company needs, a design can be implemented to make sure your data network is robust enough to handle changes and or growth!

The last thing that I can address for you is personnel. As a manager of and director of IT for 2 decades I know people. I know who is right for a job and who is not. If that type of expertise is needed; look no further.

-Best

Effectively communicating and succeeding as a Manager, using Speech and Non-Verbal Techniques

When you see the word “speech” it may conjure up that heart stopping moment when we are ask to get up in front of the class, a group of peers, or perhaps in front of hundreds of people and give a talk. Even if you are the SME (subject matter expert,) that does not make it easier to get up in front of a bunch of folks and talk. (With lots of practice it gets easier.)

Today; that is not the topic of this particular blog. Today I would like to address how we communicate as employers or managers to our subordinates.

I could not help but notice how some managers, “ask” their subordinates to accomplish some task. Really? .. ASK?

If I were “asked” if I wanted to sweep the floor, or some other mundane task, my immediate response (possibly just in my head) would be “hell no!” If I have a choice, the choice is “let someone else do it.”

If on the other hand, I am instructed to sweep the floor (not asked,) I will grab the broom and go about sweeping the floor.

My point is, not only in verbal communication do we “Weaken our speech” with seemingly innocuous phrases like “if you don’t mind, if it is ok with you, how would you feel about, when you have time to…” and the list goes on. We in fact loose authority and run the risk of sabotaging our project, when we weaken our speech.

So when you speak as a manager, use that authority that you have been granted with that title. Do not ask how someone would feel about this or that. Tell them what and when you want this or that done, and do not give them the option of telling you how to run your department or business. When they earn the title of boss, then they can tell their subordinates how they want things done, right now, it is your turn. This of course comes with the understanding that you have done your due diligence. That you know what is going on with each part of whatever it is that you are working on; and know what each and every employee is doing.

We also do this in e-mail or other interoffice memos. The main difference with e-mail and memos is that once in writing, it is there forever, for anyone who may be copied in on it, or it gets forwarded to.

We are in fact judged by how we speak, or write. Whether it is the politically correct thing to do or not, “we do it!” We all do it! Remember that old axiom “better to keep your mouth shut and people think you a fool lest you open your mouth and remove all doubt?” Sometimes we are forced to open our mouths, so educate yourself before you reach that point.

While e-mail has become the norm as far as communication with peers and alike, many of us did not take English class too seriously, and it shows! No time like the present to learn how to craft simple e-mails. Once you “pen it” and hit that magic send button, you have no idea the life that it will take on, where it will go, and who all will read it.

I once authored a memo that went to a corporation of over 30,000 folks. There was a typo in the memo (you instead of your) and while my supervisors were no literary geniuses, and took a screw-them if they can’t take a joke attitude, I was mortified. Treat e-mails and memo’s as if they are “IED’s or road side bombs with a hair trigger.” Do not get in a hurry when writing an e-mail that may be seen by your boss, or potential boss. Spell check, read it out loud, and if you have a trusted friend, ask them to proof it. One trick that I find that works is to print it, and read it from the paper. I realize that on “earth week” that is probably not the most politically correct thing to say, but it is true! I am not one for political correctness anyway; I think we have taken it way too far. That is another topic for another blog.

When I first entered into the corporate world, which seems like yesterday; the executive dictated a letter to his secretary. She wrote it down on a steno pad, in something called “greg shorthand” and then went about the task of typing it up. She would then put the letter (draft) into his in box where he would read it, mark it up, make changes and then she would once again type it up. This process could go on all day. There was a study done once that concluded the average business letter cost about $100, back in the 70’s.

Today we have no secretary to “fix it” and make it pretty. People from the board-room to the mail room have the same e-mail, which connects not only to everyone in the corporation, but to the outside world. We no longer write many formal letters, as e-mail, text (sms) and instant message is on our desktop. Are you beginning to get a sense of how important that English class was that you slept through?

How many times have you read something that someone has written and found a typo, or a grammatical error? I frequently find them in books that have reputable publicist. What is the first thing that you do or think? Yep, we judge them. We either think that they are not very smart, or very clever, or we may even question where they went to school, or if they did.

“The pen is mightier than the sword” is not simply something for writers to gloat about (which they should not do, as most writers could probably not even lift a sword,) it is in fact a powerful tool. Unfortunately, like Damocles sword it is double edged, and is indeed hanging by a thread. Be very careful and deliberate what you write; keeping the audience and secondary audiences in mind.

I heard an impressive lady the other day who said, she speaks her opinions like they are facts! They are indeed “her facts!” She claims that she is perceived as a bitch, and I can see that. Is that wrong? Should she care how she is perceived? I for one was very impressed by her talk, and I am not easily impressed. In management we are entrusted by our superiors to get the job done, and your employees become “your tools.” While I don’t use the phrase “tool” in the pejorative manner that we hear it used today, employees are in fact implements of and end to a means. The manager uses the expertise of his or her employees, to reach an objective or several objectives. If they start asking their subordinates to do this or that, their timelines may suffer as well as the project(s) as a whole. When you give up that authority to your employees, (when you have time) you are no longer an effective manager. Your employees usually don’t have the whole picture or the sense of commitment or urgency to the project or end goal which you do! Clear task with authoritative language broken down into milestones and expectations set by you are mandatory, if you are to succeed as a manager.

If you think about it, you are actually leading and mentoring by example. Employees (not just yours) watch you. They observe more than you think, and that includes the two hour lunch, or the fact that you passed gas on your way to the bathroom. You are held to a higher standard. It is probably not fair but it is the way of the hourly vs the salaried employee. They are long gone by the time the managers day usually ends; but they are not there for that.

So instead of “how do you feel about coming to work on time” vs. “the office opens at 8, and I want you here. If you can’t do that, I will find someone who can.” Will the latter earn you the title of bitch or bastard? Who cares? When they work for you, they play by your rules, not theirs. If you worry about how your employees perceive you, than your are in the wrong line of work.

To be fair, if they have issues getting to work on time, you probably need to find someone else to do the job anyway. Conversely, if you have good employees, as I have been blessed with on so many occasions, I will go out of my way to take care of them in compensation, training, bonuses etc. While there is no need to deliberately alienate your employees, they are not your friends; and you are not theirs. At the office there is an expected decorum that must be adhered to; not only by them, but by you as well. The phrase “it is lonely at the top is not just a phrase, but can be a way of life, at least from 8 to 5.

One last piece of advice that I will share that is a little off topic, “Never under any circumstance hire anyone that you cannot fire!” I want you to go back and read that again. Read that until it sticks! Make certain that you have no sacred cows working for you. By that I simply mean, everyone; “including you” is replaceable. If you own a company and you have certain employees that you cannot live without, change it fast! No one should be held hostage by having to keep someone around because they are the boss’s kid, or they are the only one who knows this or that program or system; or they are “your friend!” The largest screw-up that I see constantly is that there is no documentation, anywhere on the systems, processes, key players, vendors etc.

If you want to see how survivable your business is, run a disaster recovery drill with non-key players or bring in temporary employees from a staffing firm that have the skill set, just not the experience with your company. Then using your “living document,” re-create your business in a hot site. If that does not go well for you, and you want to fix it, call me! http://www.guard-protect.com

Hope this helps!

-Best to you and those that you care about!