Industrial Control Network Cost Millions
I never considered spending a million dollars directly on an industrial control system network, but I was wrong. I changed my view after we had a network incident that caused a production outage.
My production personnel knew exactly the demands of their service level agreement. I knew they would do every thing possible to stay operational and get operational if an event occurred. Unfortunately, in our networking decisions, we automatically assumed redundancy and additional security were too expensive without performing a proper risk analysis to consider what a network failure might cost.
After the event, I started calculating our cost and got to a million dollars quickly considering supply chain cost, inventory, finished goods, labor, stress, shipping, and lost capacity. My team argued that each production item only cost a few hundred dollars and overtime on the weekend was not that expensive. However, the CFO pointed out that not being able to produce for several days added up to a substantial bottom line hit at the end of the quarter. My curiosity forced me to dig further.
One report I found was from an automotive industry survey. They reported that a minute of downtime costs an average of $22,000 USD. At that rate, an hour would cost $1.3MM. The research had been commissioned by Advanced Technology Services, Inc. (ATS), and conducted by Nielsen Research. At first I dismissed this because of the per unit cost difference, but realized that in a week we would loose the same amount as they do in an hour.
I challenged my team to think about the problem differently and consider that the network would fail eventually and have significant financial impact. We should determine where the failure would likely occur and plan so we could produce appropriately utilizing strategies like redundancy, segmentation, and isolation. They continued to argue that it was not a concern because of their biased labor perspective and could not recognize the bottom line balance sheet perspective. We eventually upgraded the system to sustain some damage and still operate.
Meanwhile, I suspect someone on your team is making the same decision not to ask for budget to perform an independent industrial network performance and reliability assessment.