This is Part 2 of a two-part series on vulnerability remediation and patch management challenges related to industrial automation and control systems. You can read Part 1 here.
Industry 4.0 and converged information technology (IT) and operational technology (OT) environments have exposed previously isolated systems to a slate of new cybersecurity risks that in turn have forced CISOs to rethink vulnerability remediation and patch management for process-critical OT devices and control systems.
In Part 1 of our series, we explained the need for organizations to start with asset identification and vulnerability detection programs that considers requirements specific to both IT and OT. Continuous monitoring and reporting are crucial to ensure organizations limit their exposure to publicly known vulnerabilities.
We also covered the insecure design and technical limitations of OT devices and control systems. For example, much of operational technology lacks native security enhancements, including encryption and access control. Many critical devices and systems also lack update mechanisms that can accept security updates in a timely manner.
Security leaders must holistically approach OT security and include regular risk assessments and methodologies to identify vulnerabilities, and prioritize the implementation of security controls to reduce their exposure to attackers.
In Part 2, we’ll cover other challenges, and suggest a course of action to reduce your organization’s risk.
Operational technology (OT) devices are often reliant on vendor-approved or validated images to ensure the integrity and reliability of the IACS. As a result, it is recommended that organizations do not deviate from the vendor-approved patches, especially when it comes to operating system patches. Unlike IT systems, where administrators may be accustomed to a monthly server patching cadence, IACS vendors may not release approved patches at the same frequency. Additionally, legacy systems may need to continue running on outdated or end-of-life operating systems indefinitely to support continued system functionality, further emphasizing the importance of relying on vendor-approved security patches.
To ensure that patching policies are aligned with vendor patch approval schedules, it is best to identify these schedules and establish maintenance windows accordingly. This will help to ensure that patches are applied in a timely manner, minimizing the risk of security breaches. For legacy systems that are unable to receive patches, organizations should implement compensating controls, such as network segmentation, access controls, monitoring and logging, application, protocol and port whitelisting, and system hardening that are commensurate with the risk and criticality of the system.
By adhering to these best practices, organizations can ensure the integrity and reliability of their IACS, while also minimizing the risk of security breaches. It is important to recognize that IACS require a unique approach to patch management due to their specific integrity and reliability requirements. By working closely with vendors, and implementing compensating controls where necessary, organizations can maintain the security and functionality of their IACS, while also ensuring that they are compliant with relevant regulatory requirements.
Although IACS are designed to be highly available and resilient, patching these systems often requires downtime due to physical limitations or safety, regulatory, and quality requirements. Downtime can result in significant costs, with manufacturing outages costing an average of up to or even in excess of $260,000 per hour. As a result, organizations may need to adopt a maintenance schedule that is different from traditional IT maintenance schedules to minimize losses.
In some cases, systems may be unable to be shut down to address emerging vulnerabilities, and unplanned outages for patching may not be approved. To make informed decisions about patching, organizations must first identify asset dependencies, hourly costs of downtime, and risk tolerances for IACS. This information can be used to conduct a cost-benefit analysis and risk assessment to determine whether to patch or accept the risks of unpatched systems until the next maintenance window.
By considering the financial impact of downtime and the risks associated with unpatched systems, organizations can make justifiable decisions about patching. It is important to recognize that IACS require a different approach to maintenance scheduling due to their unique requirements. Organizations should work to balance the need for security with the need for system availability and make decisions that align with their risk tolerance and regulatory requirements.
Overall, organizations should prioritize the development of a comprehensive patch management strategy that is tailored to the specific requirements of their IACS. By doing so, they can ensure the reliability and availability of their systems while also minimizing the risk of cyberattacks.
Ensuring the safety, reliability, and integrity of IACS requires additional governance, oversight, testing, and validation due to the critical nature of these systems. In some industries, regulatory requirements may call for a level of procedural oversight that exceeds the requirements of normal IT patch testing. For example, patching control systems may involve greater change control scrutiny, testing on a digital twin or test environment, and risk assessments, as well as an update of design documentation.
The additional time and staffing overhead required for these procedures must be taken into account when creating standard operating procedures (SOPs) and budgeting for patch management. It is also important to consider these factors when evaluating out-of-cycle patching requests. The cost of delaying a patch due to the additional testing and validation required should be weighed against the potential cost of a breach, outage, or other security incident that could result from not patching in a timely manner.
By adopting a comprehensive approach to governance, oversight, testing, and validation of IACS changes, organizations can ensure that their systems remain secure, reliable, and available. They can also ensure that they are compliant with regulatory requirements and industry standards. However, it is important to recognize that these additional procedures come with an overhead cost that must be accounted for when developing patch management policies and procedures.
The need for vulnerability management in IACS can arise from various sources such as new developments in the threat landscape, newly released security patches, vulnerability disclosures, risk assessments, or other events. Vulnerability remediation, however, typically involves a change to the current operating state or configuration of the converged environment, which may have implications for the systems' safety, operability, integrity, or reliability.
Organizations that are responsible for patching and vulnerability management for IACS need to account for these OT-specific challenges. This can be achieved through separate standard operating procedures (SOPs) or specific language to differentiate between IT and OT systems in existing governance documentation. In addition, IACS should be designed with consideration of a longer mean time to patch compared to a typical IT system, given the potential implications of any changes made.
Moreover, the changes required for vulnerability remediation in IACS involve testing, downtime, and procedural oversight, such as risk and impact analysis, change control, and post-change validation activities. These additional work cycles and staffing requirements must be considered in SOPs and budgeting to ensure that the remediation process is performed effectively, efficiently, and without undue risk.
Overall, organizations should adopt a comprehensive approach to patching and vulnerability management for IACS that accounts for the specific challenges associated with these systems. This approach should be guided by a framework that considers the requirements of the systems' safety, operability, integrity, and reliability, as well as regulatory requirements and industry best practices. By doing so, organizations can ensure that their IACS are secure, available, and compliant, while minimizing the risk of cyber-attacks and other security incidents.
Juan Piacquadio is the CIO & VP, Information Technology at Phlow Corporation.
Tim Hall is the Director of Information Security at Phlow Corporation.
