Safety and availability requirements within operational technology (OT) environments and where connected Industrial Internet of Things (IIoT) devices such as actuators and sensors are becoming prevalent, creates significant vulnerability management challenges. A familiar meme applies nicely in this scenario: “One does not simply patch OT.”
Applying a software patch or firmware update often requires scheduled downtime and intensive regression testing. In areas where downtime isn’t tolerated, a faulty patch could result in the loss of hundreds of thousands of dollars per hour if a patch failure results in loss of equipment and services or affects personal safety. OT asset owners must ask themselves whether it makes sense to patch in their environment in the first place.
As you can see, the nuances of vulnerability management for OT and IIoT environments are much different from traditional information technology (IT) enterprise environments. For example, OT environments that now include IIoT assets that expose traditionally air-gapped networks to the internet create new vulnerability management challenges. A pragmatic approach to OT and IIoT vulnerability management is required to achieve cyber resilience in these environments.
Consider vulnerability management for OT assets in manufacturing, such as a manufacturing execution system (MES) for an automotive manufacturer. It may be possible to test, evaluate and schedule patching on Microsoft Windows-based Human Machine Interfaces (HMIs) running on a workstation between shift turnovers and factory down days. However, this turnover is not possible for programmable logic controllers (PLCs) because pushing firmware updates remotely is not an option in most cases. It would also be dangerous if a failure occurs and challenging to recover compared to an HMI running on a Windows operating system. The amount of time to swap out a DI-rail-mounted PLC and program logic on the fly is more difficult than hot-swapping a pre-programmed workstation.
Another example of OT vulnerability management challenges is within the chemical sector. From personal experience assessing polymer plants, an OT asset owner may operate one of the reactors at a plant for two-years before they can consider patching or upgrading devices within the process control systems. This is because controllers operate and monitor sensors and actuators that cannot be stopped for firmware updates or hardware upgrades during chemical processes. Any disruption to controllers, actuators, or sensors within chemical processing could be catastrophic and result in the loss of life, environmental damage, and federal fines. However, there is an opportunity for focusing resilience efforts on the offline line process control system by performing hardware/software upgrades and applying patches to controllers, sensors, and engineering workstations until the running chemical plant reactor goes into its maintenance period.
Industrial IoT sensors expand boundaries to include traditionally closed-off OT environments, which make them vulnerable to external attackers depending on the level of exposure to the internet. OT asset owners can implement on-premises servers to support IoT sensor monitoring without exposure to the Internet.
At the same time, some have permitted ports on the firewall for monitoring IIoT sensors within an external cloud service provider. Suppose an external cloud is used for IIoT sensor management and monitoring, such as in smart manufacturing environments. In that case, OT asset owners must develop a vulnerability management plan for patch testing, evaluation, and deployment with a digital twin of the domain. Having a digital twin domain reduces risk and improves resilience by providing an asset owner with a replica of the actual production environment for performing software updates, patches, or firmware upgrade testing and regression without risking disruption to operations. This can help the asset owner develop a resilient, tailored vulnerability management process for their environment.
Vulnerability management to enable cyber resilience for different critical infrastructure sectors may be slightly different due to the type of control system environment and business operation demands. These demands may not afford vulnerability and patch management on a timely basis in OT environments, as with IT enterprises. Patch deployment within an IT enterprise environment is usually automated through Microsoft Endpoint Configuration Manager (formerly SCCM) or other patch management software, which isn’t the same for OT environments.
Business operations are a major factor influencing when patching and upgrades are performed. A one-size fits all approach to vulnerability management is not possible for OT environments and needs to be considered based on the critical infrastructure sector.
This is starting to be addressed by efforts between the National Institute of Standards and Technology (NIST) and the National Cybersecurity Center of Excellence (NCCoE) for manufacturing with the NIST SP 1800-10: Protecting Information and System Integrity in Industrial Control System Environments: Cybersecurity for the Manufacturing Sector. This will take time to address the nuances of vulnerability management across each critical infrastructure sector.
In the meantime, OT asset owners face an uphill climb when it comes to vulnerability management because of all the dependencies around critical assets, an intolerance for downtime, and the possibility of system failure or even personal harm to operators or the public. A pragmatic approach takes into account all of these nuances in order to not only patch vulnerabilities but also build resilient systems and networks that can withstand an active incident and maintain process availability and integrity.
Dan Ricci is founder of the ICS Advisory Project, an open-source project to provide DHS CISA ICS Advisories data visualized as a dashboard to support vulnerability analysis for the OT/ICS community. He retired from the U.S. Navy after serving 21 years in the information warfare community.
