audit – TheTimeDok

Consulting as a Disaster Recovery Specialist, I often find things that need to be changed to avoid a disaster, much like a loose rug over a threshold or too many things plugged into one circuit; which would be an issue in your home. In the business world it comes down to security issues both IT related and physical, as well as simple things like a lack of fire extinguishers or the wrong type of fire retardant system in the computer room. I am trained to notice the smallest of details including things like cable management issues.

When Best Practice scenarios are not followed by sys-admins or networking guru’s, they too trigger red flags. There is an art to designing data centers. I have designed and built many over the last 30 years complete from the ground up; from air handling to power requirements to working with ADA compliance issues. I have designed cable management for many companies that include the MDF and IDF’s and working with building management to handle communication through multiple story buildings making sure that they pass fire code. You would be amazed at how many data centers that I walk into that are under wired, lack proper air handling and have a sprinkler head above the equipment! The cable management looks like Spiderman installed it, nothing is labeled, and there is absolutely not one shred of documentation. And the boss / owner is oblivious to the immanent disaster, as he thinks his guys are pretty good!

When business’s start up, often times they don’t contact the brightest and best to build it as they are on a tight budget. When I am called, their data center is generally a candidate for one of those web sites that post “what not to do.” The exercise of unraveling the Gordian knot comes into play before anything can be changed. Many times a family friend is called to assist or the business owner has a home network and thinks that a business network is no different.

When these knots are constructed; most if not always there is limited or no documentation and the original creator has long since abandoned ship as he undoubtedly realized the ice berg ahead was not too far off. To that end there are many land mines that have to be discovered and diffused. This practice is akin to changing the tires on a racecar, while it is going down the track, and part of that track is in no mans land! The catch 22 is that no business can afford down time but, if they don’t address the issues they will have un-planned down time! Un-planned is always much longer than planned, and always more expensive!

As an SME on this and many subjects regarding IT, I can offer many things to mitigate any issues and put them on a road to setting things right. Whether that is working with their current IT staff, or bringing in hired guns to knock it out quickly!

The business must be willing to want to change, and have Executive buy-in as well as buy-in from the local staff. The process can take weeks to months depending upon the situation; but after it is all said and done, procedures and processes are put into place to keep up with change.

Some policies addressed are Change Management, Incident analysis; complete with root cause analysis, documentation with the introduction of the concept of a living document. The run book, what is it and how does it work? Testing the Disaster Recovery plan and then implementing changes from things learned. Other topics include SAM (software asset management,) and of course hardware management including lifecycle, and the budget process.

All too often the CFO or CEO is told that IT needs X thousands of dollars for this, that, or the other thing; not because it is a new project but because something failed! With proper asset management this can be mitigated greatly and things can be budgeted for.

Much like any other audit, I don’t guarantee anything will be pleasant other than the knowledge that when it is done you will have the documentation you need, your network will be running at peak efficiency and it will be secure. Depending upon your growth and company needs, a design can be implemented to make sure your data network is robust enough to handle changes and or growth!

The last thing that I can address for you is personnel. As a manager of and director of IT for 2 decades I know people. I know who is right for a job and who is not. If that type of expertise is needed; look no further.

-Best

Disasters Big and Small

As a Disaster Recovery Specialist, I walk into many companies that are one step away from disaster. Some of them have been living on a wing and a prayer for a long time and are absolutely oblivious to the precipice on which they are perched.

One of the largest challenges one faces in this line of work are people. By that I mean more specifically egos. People are threatened by someone that “knows more than they do.”

Let me tell you a secret. This is a Jack Palance type secret, (from City Slickers) “This is the one thing” that will save your keister as well as change your attitude.

I worked for a man who owned this business that was very successful. I was a young guy fresh out of school and this guy saw something in me that I remember to this day. As time passed he took me under his wing and helped me knock some of the rough edges off of my “perception” of the world as it was. He took me out one day to JC Penny and had some sales clerk measure me for a suite and then he picked out a couple of them. We went to the shirts and he purchased a few of them right down to the shoes. While these were not super expensive, they were not cheap and his generosity never escaped me. The only thing that he did not replace were my shorts! Some might have taken offence to this but I am no creature of fad or style and while I would not qualify for a candidate on “what not to wear,” I did know that style was not my strong suite. “Knowing your limitations” is good advice, but not the secret.

Later he had me take over the service manager position in one of his branches which came with a company car and credit card. This was before the tax laws changed. He told me to use the car as I wished and if I took it on vacation to at least “pay for some of the gas myself.” He took me over to the office which was a good drive from the Dallas office. He regaled me with stories of advertisement and marketing. He told me the story of the sign with the waterfall on it by downtown Dallas. Back then it was a Pearl Beer sign. This man was pretty close to deaf. He was from Georgia and his accent was still very thick. It turns out that he was a tank commander in WWII. He told me that the secret to survival is to “surround yourself with smart people.” That not only applies to war, but business and oh yes, life in general. If you want to be successful, surround yourself with people smarter than yourself and learn to humble yourself. It is only by this step of humbling yourself will you realize the advantage of being around these people. I have never forgotten this and to this day I still practice this.

I offer this advice to all IT people in that “you are not the end all be all.” You cannot know it all even though you think that you do. We become focused on what interest us and then the rest of technology passes us by. Learn to control your ego for it is your enemy. No doubt you have heard the phrase “you are your own worst enemy.” Think of the truth of this statement and then marry it, own it and then change it. When someone starts talking to you about something which you think you know about and you feel that “anxiousness” start to well up inside, recognize this for what it is, you’re undoing. Squelch the feeling, take a deep breath and listen to what this person has to say. It may be worthy of hearing or it may be total crap. Before long this will be habit and you will have trained your ego to stand down.

One of the first steps in the DR process is an AUDIT. In order to prepare for a disaster one has to know what one has. This is done by an audit of the technology, how it is configured and of course managed. We look at policies and procedures and just really get into your business in a big way. The more you work with us the more you will get out of it. Conversely the more truculent or evasive that your staff is, the more it will cost. This is a “by the hour” service and time is money.

Audits are never fun but necessary, in that no one is perfect. Audits uncover the “dirt” so to speak and no one wants to acknowledge that they have dirt. Nobody wants to look bad so they are either un-helpful or become very defensive and blame the guy before them and so forth. No one in their right mind would welcome an IRS audit because of this. You know that you are playing by the rules but the rules are thousands of pages long. What if? Individuals should budget for an accountant for this reason. Companies should have more than one accountant “even if it is a small company” in that they can check one another. (another story for another blog)

While IT audits wont land you in front of a judge, it could have an effect on the bottom line in that deficiencies could be uncovered which could end up in with un-budgeted expenditures. Having an up to date DR and BC plan will not only prevent this but, will keep your IT department on their toes and up to date. A fresh set of eyes looking at how things are done contrasted against your business processes and needs, often bear fruit in that there may be a better way to do things. Personally I subscribe to “best practice” methodologies and policies.

Some companies don’t take IT seriously and look at it only as a necessary evil. An attitude which must be changed as IT is much more than a necessary Evil. IT is a resource which ties the entire company together. This department is the glue that binds most departments together as well as the interface between the customer and the company. In looking at the want ads occasionally one might notice ads for IT people with the following “PC Wizard” needed. Really? Does this person come from over the rainbow? The simple facts are that some HR people are totally bereft of any ability to interview for this position and the company as a whole does not take the department very seriously. I would liken this to the “audio visual club” at school. Know this all you who mock them, the nerds will inherit the earth. I digress..

If you really look at the way that your technical infrastructure touches every person in your company and your customers; your attitude on this matter might change.

During the process of a disaster recovery plan, this becomes very clear in that one of the pieces of this plan is a Business Impact analysis. It is during this process that the lights turn on in the CEO’s, or CFO’s head. I have heard the question posed to the CIO or CFO on many occasions “why hasn’t anyone told me this?” The simple facts are that the CEO’s job is to run the company, not the IT department. He or she depends upon the CIO to look out for the company on all things IT and a DR plan is simply one small part of it.

Simple programs like asset management and S.A.M. “software asset management” are not only not in play, but not even thought of. How can one budget for new stuff if one has no clue what one will need down the road? A complete Asset management program should be SOP in any company. This program accounts for hardware from the cradle to grave.

The same is true regarding software. Often time’s, companies pay way too much for software as it is installed by policy on computers with users who will never use it. Users may bring in their own software and install it, leaving a liability for the company to contend with should there be a software audit and it is done by the SBA.

While there are no good surprises in business there are certainly no good surprises after an event has been suffered by a company. A fire in the data center could take the entire company out of the marketplace for good.

Fire caused by poor cable management practices.

Human error accounts for a large percentage of the events which caused companies to fail. Doing a root cause analysis on failed companies who suffered a disaster you find that they did not value such a thing as “it will never happen to me.” You don’t have to suffer a Sandy or Katrina type event to bring your business to its knees. A simple mistake from some employee, working for a company without a business continuity or disaster recovery plan can ruin your day, if not your career.

It is at this time many companies wish that they had spent the money on such a plan. Too Late… If you fail to plan you plan to fail.

You can purchase insurance which will assist with the closing of the company but, that is not the way to go out of business, with a whimper, because you failed to plan.

Updated documentation of your infrastructure otherwise known as a “living document,” should also be SOP. IT folk absolutely do not like documentation, more specifically creating it. There are many schools of thought on this reason, but I suspect that laziness along with a “need” to have proprietary information so they are not expendable weighs somewhere in their decision. If the latter is your reason for not doing what is right for the company you need to re-examine your life.

If you are taking the paycheck you owe your employer the best that you can offer. If you managers feel like you have people in your department who are not expendable you need to address this post haste! One rule of preventing a disaster is avoiding single points of failure; and that means people as well.

Part of disaster recovery is averting disasters to begin with! Through solid best practices in policies and procedures, a large percentage of disasters can be negated.

One last topic on the subject that comes up from time to time. “Do I have a legal obligation to have a DR/BC plan?

The answer is not as clear cut as one would like. The interesting thing however from a legal perspective is that there is legal precedence whereby companies were held liable for failing to provide a more error tolerant system. They in fact were found to be negligent and case law purports to award large sums of cash to the plaintiff. These cases not only hold the owners of the company negligent but any and all officers of the company are liable. Think carefully about that promotion and VP title.

While companies are apathetic towards spending the money on such a plan, doing so is not only moral, it is strategic and most likely a legal obligation. As Billions of dollars are spent annually on technology to maintain a competitive edge “standards of care” and due diligence are required of all corporations both public and private. Not having such a plan violates the fiduciary standard of care.

-Best to you!

staylor@guard-protect.com