Topics:
During a consulting engagement with a major automotive manufacturer, we patched thousands of Windows-based HMIs and engineering workstations across the assembly, welding, paint, and quality-assurance (QA) lines in one weekend maintenance window.
Pushing the patches was the easy part, but deployment didn’t mean the job was done. That’s because validation—comparing every machine’s patch state against the previous window’s record—would have caused us to miss a number of actual occurrences.
We found machines that had been offline during the cycle and missed the updates, and we scheduled them for smaller windows during non-production periods.
We also found newly imaged machines that had been put on the plant floor without earlier patches or required software, and those needed special remediation.
If we had called the job done when deployment finished, we would never have known about any of them. This is where a lot of OT programs fall short. Documentation, a clean audit, or a vendor attestation shows what was supposed to happen. None of it shows that the control is working on the plant floor today, and that is the question validation answers.
In OT, validating a control takes more than someone looking at it once. It is a chain of evidence that three groups build together. For example, let’s look at a firmware update. The engineer applies it and records it in the maintenance log. Operations ties that work to the change-control ticket they approved and the maintenance window it was performed in. The security team then confirms, independently, that the update was applied and the vulnerability can no longer be exploited. The validated update is recorded in the configuration management database (CMDB), so it can be audited if the organization’s OT systems fall under NERC CIP in North America or the NIS2 Directive in the EU, where Article 21(2)(f) calls for “policies and procedures to assess the effectiveness of cybersecurity risk-management measures.”
The way to get every team on board is to treat validating a security change as part of completing maintenance, because that is what it is. An engineer doesn’t half-finish maintenance, do it improperly, or skip the sign-off. Patch and firmware work should be held to the same standard, and falsifying a patch record should carry the same consequences as falsifying any other maintenance record.
Validation needs time, and that time has to be planned into the maintenance window up-front rather than squeezed in at the end. Depending on the environment (a manufacturing execution system (MES), energy management, building automation, battery management, or process control), you may get a 12- to 24-hour window. Where continuous operation is the norm, offline systems get patched during long outages, and the next opportunity might be one to six months out, or one to two years. A change that fails or never gets verified stays exposed for that whole stretch.
An engineer doesn’t half-finish maintenance, do it improperly, or skip the sign-off. Patch and firmware work should be held to the same standard, and falsifying a patch record should carry the same consequences as falsifying any other maintenance record.
Safety sets the method. Security teams should never run vulnerability scans against PLCs, RTUs, or other devices that aren’t running Windows or Linux. A scan can take them offline, lock up open ports, or in the worst case brick them. Engineers usually know these devices best, so security should agree with them on a safe way to confirm versions, often through the vendor’s SCADA software or engineering tools, checking the running version against what was on that model before the update. On Windows-based OT hosts, engineers can run local scripts built on standard commands such as PowerShell’s Get-HotFix (or systeminfo on hosts too old for PowerShell) and hand security a dated export to diff against the previous configuration report. If nobody has recorded these versions before, that first reading becomes the baseline. New firewall rules, and the IDS detections that feed the SIEM, get tested in a virtual environment before they go anywhere near live operations.
The standards say much the same thing. ISA/IEC 62443-3-3 asks for the capability to verify that security functions operate as intended (SR 3.3). NIST SP 800-82r3 recommends testing patches under field conditions on a test system, where possible, before installing them on the OT system. NERC CIP-010-4 requires entities to verify after a change that required security controls were not adversely affected, document the results, and update the baseline within 30 calendar days. For high-impact systems, it also requires documenting how the test environment differs from production. That last piece deserves attention, because a control that passed testing in an environment that doesn’t match production can still break a live process.
Plan for the patch that fails. Every patch and compensating control plan needs a written backout step, so the change can be reversed immediately if it affects safety or operations. A root-cause analysis should follow right away to determine what went wrong and whether the cause was human or electro-mechanical. If a patch fails and there’s no room left in the window to try again, reschedule it, record why it failed, and decide whether a compensating control is needed until then. That decision depends on how critical the system is and how exposed it is, for example whether operations and maintenance (O&M) or engineering staff access it remotely on a regular basis.
In a well-developed program, the person who deploys a patch shouldn’t be the one who validates it. Separation of duties is part of what gives the program its integrity. Many small and mid-size organizations don’t have the people for that, so two-person integrity is the practical alternative: a second qualified person witnesses the work and both sign an attestation, with the level of formality set by how regulated the environment is. That second person doesn’t have to come from security; a qualified engineer from another shift, an operations supervisor, or a contracted integrator can fill the role. Where a union represents the workforce, accountability terms for maintenance records have to be negotiated in writing before they can be enforced. It’s a nuanced issue, but a real one. Final authority sits with the people who approved the change control: the CISO, operations management, and the business unit manager.
Done starts with an entry in the evidence log confirming that a specific patch or firmware version is running on a specific asset. If the patch wasn’t applied, the asset goes on the risk register, and the entry should capture the asset ID, site, advisory, affected products, status, the rationale and who made the call, the compensating measure, notes, a target mitigation date, the record date, and the signatures and titles of the approval chain.
That register entry is also how the exception gets communicated. The people who approved the change sign it. Operations managers brief their crews on any compensating control that changes how they work. Open entries get reviewed before each maintenance window, since every window is another chance to close one out.
Some systems reach end of life with every available patch applied and nothing on the market to replace them. For those, done means isolating the system, monitoring it, and maintaining a documented supply chain risk management plan that keeps it running. That’s what I call “Done, Done.” The IT mindset that legacy systems must be replaced isn’t always practical, cost-effective, or realistic in OT. Virtualizing platforms that still depend on Windows XP or Windows 7 can add a logical layer of protection, but those systems still need defense in depth when the next advisory comes out with no patch available. NIST CSF 2.0 describes the target outcome the same way: “Software is maintained, replaced, and removed commensurate with risk” (PR.PS-02).
To know whether the program is working, CISOs should watch three numbers: the percentage of changes closed with validation evidence inside the same maintenance window, the number of risk register exceptions past their target mitigation date, and the percentage of OT assets with a current recorded baseline. The security team pulls these together after each maintenance window, and the CISO reviews the trend with operations and engineering leadership every quarter.
Your job during validation is to check the evidence, not to touch the devices. Work out safe collection methods with engineering, diff each new baseline against the last one, and test detection and firewall rules in a virtual environment before they go live. A denial of service caused by the security team is not a great way to build a relationship with operations and engineering, and you will need that relationship in every patch cycle after this one.
This will most likely reach you through your engineering manager rather than the CISO, framed as maintenance completion. You already use the right tools and PPE, and you already sign for work done properly. Recording the firmware version you installed, along with the version it replaced, is part of that same signature. Those records are what everyone measures against the next cycle.
Operations management signs off on the change control ticket before the work starts, which gives you the best situational awareness when the window closes: did the work finish in the time allotted, and does anything look off once the process is back up? Report disruptions and unusual behavior after every window. A compensating control done right should be invisible to you, unless it blocks something you were previously approved to do, like remote access into the OT environment. Your managers should know that ahead of time and brief the crew so it isn’t a surprise on shift.
When a CISO brings security, engineering, and operations leads together, the message is this:
“Validation is not one person’s responsibility. It belongs to this team. We treat patch and firmware updates on OT systems as routine maintenance, performed by qualified people, tested in advance by the vendor, third parties, and our own engineers, and validated by the roles we designate. We put safety first and make no changes without the planning and approvals we all agree on. When any of us objects to a patch, we build a compensating control together that keeps the plant safe and running.”
Dan Ricci is founder and CEO of Industrial Data Works, and the founder of the ICS Advisory Project, an open-source project to provide DHS CISA ICS Advisories data visualized as a dashboard to support vulnerability analysis for the OT/ICS community. He retired from the U.S. Navy after serving 21 years in the information warfare community.
