On March 7th 2008, the Hatch Nuclear Power Plant in Georgia was shut down for 48 hours due to a software update. This computer was used to monitor data from the facility’s primary control systems. The issue was that the computer was also on the plant’s business network. The patch was designed to synchronize data but when it rebooted, it also reset the data on the control system. The system interpreted this reset as a drop in the water level contained within the cooling systems. As a result it responded as it should have and triggered a shutdown.
Why is this important? Aside from the implications that this has for our nation’s critical infrastructure, this incident highlights the unintended consequences of updating systems. Now I’m not advocating the practice of not updating our systems. I think it is important to update our systems. What we should not do is update critical systems without prior testing.
This may seem obvious but there are quite a few companies out there that blindly update their systems without testing these patches first. Had there been a patch testing program in place, it is possible that they would have realized that the data store would have reset itself and then be able to deal with the situation without inadvertently causing a shutdown. It is also possible that this system was so unique that testing would not have discovered the problem prior to the patch “going live.” No program is full proof. There is always something that could go wrong – that is why it is called Risk Management rather than Risk Elimination. Having a Patch and Vulnerability Management program instituted not only can save you from having a loss due to an unintended consequence but it also shows that you were practicing due diligence when a loss does occur.
Entries (RSS)