Dealing with an IT Outage or Crisis

January 17, 2010

So the day you’ve been dreading as an IT Consultant has finally arrived. An outage has occurred with you or your client’s IT infrastructure. In a time of crisis it’s natural for your stress level to rapidly rise and you may even find that your life has just skyrocketed to Defcon 1.

Step Back, Calm Yourself

In situations like this, it’s real easy to hurt you or your companies reputation. In these times of high level stress, it’s important to focus on solving the issues at hand, but you also want to keep in mind that your emotions are on display for everyone to see and your reactions not only can be contagious but can also make matters worse, not better.

So what are some steps you can take to help you make it out of this world alive?

Here is a list I personalized based on my experiences and Steve Tobak’s How to manage a crisis, any crisis | Train Wreck – CNET News.

1. Be focused
Assess the situation, don’t give in to the urge to make quick and careless reactions. Have a goal in mind of what you want to accomplish.

2. Look around
Investigate, what events led up to the outage? What is the most likely cause of the outage? These questions can assist you in developing a clear picture of the situation.

3. Who is affected?
Notify all key stakeholders of the situation and be honest with them. If working with a client as a consultant, involve those parties that may be able to provide some assistance with assessing the situation and arriving at a positive outcome. Continue to communicate constantly with all through E-mails, online conferences, voice mail, etc. Make sure all parties that need to be updated are included.

4. Don’t be afraid
Call for help, being part of a strong team goes a long way. At Varrow, the company I work for, success is always derived from the right people and teamwork. There have been several occasions where I have had to rely on co-workers to help me through a situation, and let me tell ya’, they were always there for me.

5.Be like Batman
Once you have gathered all information and consulted with all parties required to move forward, construct a plan on how to tackle best, worst and typical scenarios. Be prepared for any obstacle you may face.

6. Execute
You have a plan, now put it in motion. you should feel more confident about what actions need to take place and how to react to any hurdles that might present themselves.

7. Remain confident.
Know what you need to do, but be flexible. Remember that your client or colleagues’ emotions maybe predicated by how you present yourself during this time of crisis.

8. Regroup
Now that the smoke has cleared and everything is hopefully back to normal, have a meeting with all parties involved to review the outage and what steps were taken to solve the issue. Also review any next course actions that need to take place for clean up.

9. Find the Root
Re-investigate what events happened leading up to the outage, what steps were taken to resolve the issues at hand. Review any logs that are helpful and interview any parties that had interaction that could have caused the outage. Come to a conclusion and create a root cause analysis report and review it with your colleagues and/or clients.

10. Learn
Do what we Humans do best, learn from mistakes. Were there any steps that could have been taken that would have prevented this crisis from occurring? Document them, adjust any internal procedures, so forth. Don’t forget what Mr. Benjamin Franklin said, “an ounce of prevention is worth a pound of cure…”.

Hopefully after following these steps, you will find yourself in a situation where you, your colleague, your company, or all of the above is admired for their honesty, determination and calm manner during this time of a crisis. It’s these traits that will be remembered in the end!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: