Enter if You Dare

There’s nothing scarier than not having the proper backup and disaster recovery (DR) strategy in place. But it’s also so much more than that. Now, it’s imperative that organizations do their due diligence to avoid costly recovery from natural disasters, user error, or other emergencies.

I’ve seen a lot of the classic, and sometimes fantastic, pitfalls in action and the poor teams that are left trying to pick up the pieces. I’d like to help you mitigate disaster, so your story isn’t one told around the sysadmin campfire. Below are just a few IT Horror stories from customers I’ve encountered over my 20 years spent serving IBM i customers. Let them serve as a warning—there’s something to learn from each situation.

1 Don’t Keep All Your Eggs in One Basket

Firstly, don’t ever think, “it can never happen to me”. That train of thought leads right into the station of scary situations. I had a customer with a production machine and an HA system which were both kept within the same data center. If the data center in question were to be compromised for any reason, due to a power outage, then you would lose both backups.

This, unfortunately, happened to this customer who kept both of these assets not just in the same data center but also in the same rack, and then that rack lost power. This meant that they simultaneously lost their production environment and their HA environment, effectively leaving them with no backups. A simple story like this can keep an IT Director awake at night.

2 Is That Really a Sound DR Plan?

Sometimes, the greatest horror or fear that an executive has regarding IT, is the unknown. Case in point, an IT executive who took their tapes offsite as part of their DR “plan”. They transported them in the trunk of their car and stored them in the “comfort” of their own home. This is the classic example of, “is that really a sound DR plan”?

Let’s break this example down. First, taking tapes offsite already puts them in danger. Next, placing it in a hot car or trunk leaves it at risk for collateral damage in an auto accident, exposed to temperature fluctuations, and also general absentmindedness. Backup media assets should always be kept in a cool temperature-controlled environment. If this customer were to ever try a restore, it probably would not work due to the fact it was exposed to heat.

It’s certainly safe to say this “DR plan” led the company CEO and president to really start questioning this executive and these questions are good to ask in any situation:

• Have you tested your DR plan?
• When was the last time you executed a full system save?
• Have you tried to restore the system from a full back up?
• Have you confirmed that the backup tape is in working condition?

Within a week, this customer conducted a DR test. They did a full system save and then tried to restore the system from tape only to discover that the media was indeed corrupted. Now imagine if it hadn’t just been a test. The stuff of IT nightmares.

3 Safe and Secure?

A large company I’d worked with as an IBM Business Partner had freshly acquired a small security company providing both cyber and on-premise security services including penetration testing and physical security threat analysis.

A sales rep had a conversation with the Director of IT about this new offering. The IT executive stated, “We have an AS/400 that cannot be breached, and our physical security is airtight. So, why would we pay for something that we know is already well in hand?”

After working with this customer for 10+ years, the sales rep thought otherwise. They knew that the client in question had all doors key-carded and kept locked/closed at all times. They also knew their building was equipped with an on-premise security guard and employed restrictive guest protocols requiring check-in and escorts. However, he also knew from experience that he was typically “waved on back” and would be free to wander the halls unsupervised.

The sales rep offered the customer in this example a physical security audit to start which would be followed by a simple pen test. The customer agreed.

For the test, a meeting was scheduled. While the sales rep, security guard, and the services delivery manager waited in the lobby for the Director of IT, another security professional had walked to the exterior of the building and gained entry via a door used by employees in a separate connected building to access vending machines and a small cafeteria.

As the Director of IT greeted his guests, his phone rang and on the other end was the second security professional calling him on his cell phone while sitting at the Director of IT’s desk accessing his desktop.

This example illustrates that a threat can be closer than you think. Sometimes the boogeyman is right next door.

4 Accidents Happen

While we constantly encounter customers facing challenges with IBM system support, most would expect this to be from folks retiring. However, sometimes it’s because a sysadmin is involved in an accident. From car accidents, to travel mishaps, unexpected ailments such as a surprise appendectomy – all have the power to put your sysadmin in the hospital for a few days.

Sometimes accidents are even more serious than a brief stint in the hospital. Sometimes people die unexpectedly. This is why it is important to be prepared for anything, even the unthinkable. I once worked with a Fortune 500 company whose only IBM administrator passed suddenly. He was the sole contact with access to passwords and user IDs.

This is among the many reasons why Connectria’s managed services are essential. As more companies lose their internal IBM sysadmin talent, this becomes a crucial service. Not only do managed services enable your team to focus on innovation over infrastructure, but it also insulates you from the unthinkable and ensures you have reliable access and coverage at all times.

Ask yourself what you would do today if your Sysadmin were to unexpectedly quit? What would you do if they announced their plans to retire in two weeks? Where would you be if they were unexpectedly fired? If you don’t have an immediate answer, call Connectria today to see how we can prepare you for tomorrow and every day after.

Connectria is Here to Help

A common theme in most IT horror stories is that the customer believes they have a sound disaster recovery plan. When in actuality, they do not. They’re either lacking full systems save or they assume their DR plan is inherently built-in when it isn’t. This is typically discovered when they test or try to utilize their DR plan and come to find that things don’t work as expected.

If you’ve had your own IT horror experience, you know it can be worse than some of the examples above. If you’d like to prevent experiencing your own story, contact one of our IT experts today to discuss options for securing, managing, and optimizing your environments.