A Tale of a Manufacturing System Outage - Ancient History or Your Future?

It was 2:20 am on Saturday morning and Dave was in bed after enjoying a relaxing Friday evening with his family. The big family ski vacation that everyone was looking forward to after months of planning, was to start the following day and everybody had gone to bed early to match the early start. Little did Dave know that tomorrow, that ski trip would be the last thing on his mind.

As the clock ticked to 2:21 am, the blissful silence was broken by the chirping of Dave’s phone. Bleary-eyed he could see that it was the out of hours call service. “This can’t be good”, he thought. Reaching for the phone to silence the alert, he noticed immediately that there were 8 missed calls from one of the plant managers and several email notifications beginning with “[HIGH PRIORITY ISSUE]”. It was time to go to work.

At its peak, this system outage brought down Dave’s SAP-based manufacturing landscape and resulted in lorries queuing for up to 40km outside of distribution centers. It created a complete inability to ship Product or continue with manufacturing activities globally. All of this, caused by the overlooking of signs that something was seriously wrong with the hardware underpinning these critical systems. The warning signs had been there for months, but nobody noticed.

Dave managed to source a temporary replacement and the systems began coming back online by lunchtime the following day. But the outage of nearly 14 hours had made its mark. The business suffered hundreds of thousands of dollars in lost revenue, penalties and more.

Download our Manufacturing Brochure >>

This story is all too familiar to operations professionals and should be consigned to the history books, linked to a time when IT systems were new and fragile. A time when “control - alt - delete” was a phrase commonly used to solve issues.

In 2021, we have the tools to proactively manage and monitor the critical systems that power our manufacturing landscapes. We must not rely on manual observation to spot potential issues but rather entrust these repetitive tasks to platforms like Avantra that can alert you before any outage and empower your teams to be ahead of the curve. Critical alerts must be routed to the right people at the right time and before the business is impacted.

If Dave had a solution in place like Avantra, he would have been alerted to an impending hardware failure months before any outage. The issue affected the Identity Management (IDM) portion of his manufacturing landscape which was a critical single point of failure. When this service was offline, nobody could log onto the SAP MES (Manufacturing Execution Systems) systems which brought the production line and associated logistics processes to a complete standstill.

So ask yourself, is Dave’s story one that you see in your future or is it, as it should be, something from the past that we tell our kids about as we sip a hot chocolate in the evening around a fire on the family ski trip.