Technical Debt in OT Environments: How It's Different and Why it Matters

Most people in tech are familiar with the concept of technical debt as it pertains to software development. But rarely do people outside IT personally experience the cost of technical debt and when they do (most of the time), they certainly don't realize it. Technical debt is generally something that the IT department and developers need to deal with. 

Two recent events in the airline industry showed how costly technical debt could be. In early January, the Federal Aviation Administration found a corrupt system file that led to a ground stoppage. A source told CNN that a tech refresh had been "pushed off because of budgetary concerns and budget flexibility. I assume now they're going to find money to do it."

A couple of weeks prior, Southwest Airlines canceled thousands of flights from just days before Christmas to New Year's Day. While poor weather may have been the catalyst, it was widely reported that outdated systems made a challenging situation significantly worse.  

They are certainly two unfortunate examples of how technical debt can have costly and real-world repercussions. The term initially applied to software development, which meant expediting code production at the expense of code quality. In these cases, the developed code would have to be refactored, and that refactoring is paying down "technical debt." Today, technical debt broadly refers to having old technology in place that causes problems later on, such as increased maintenance, lower productivity, and software or system reliability.

How does this technical debt concept differ in operational, medical, and IT environments? And what are the best ways to manage it away when it does creep into operational environments?

OT Technical Debt Found in Infrastructure

While technical debt can be created within environments by not upgrading or updating systems for long periods, they fall behind current standards, such as modern APIs, network and communication protocols, and security standards. Experts say, unlike traditional technical debt, which resides within developed software and technological platforms and on infrastructure, most of the technical debt in OT environments resides within the infrastructure. 

Technical debt crept into operational environments because IIoT devices were installed with discrete use cases in mind, such as to measure temperature, pressure, or resource availability. These systems were theoretically air-gapped and only allowed specific access. However, over time, organizations wanted to become more aware of their environments, and they increased the number of sensors in use. 

Primarily due to concerns around negative impacts on production, the devices, infrastructure, and supporting applications are rarely updated in these environments. 

The result is an aging infrastructure and devices heavily sunken with technical debt and difficult to secure with modern tools. How are these systems patched? How is it connected? Most of the time, organizations try to put IIoT devices on a separate VLAN. "Due to technical debt, many switches and access controls don't allow for that segmentation," says Jonathan Townsend, vice president of engineering at Set Solutions. "The organization often then relies on their vendor for patching. Now that third-party has internet access," he adds.

Technical Debt Increases Costs

The result? Security teams in OT environments with considerable technical debt find that they can't keep the systems as secure as they'd like without the organization engaging in costly upgrades and modernization efforts. Unable to secure their systems as tightly as they'd like, they turn to trying to enforce security standards on their hardware suppliers and service providers. 

"Rather than fix a lot of their problems, we often see teams turn to supply chain security and vendor compliance," says Townsend. "It becomes a matter of looking at the vendor's security posture and how well the vendor follows good security and regulatory practices." 

Such practices would include looking at the vendor's security practices, vetting for IEC 66243 certification, SOC Type 2 certification, ISO 27001, and specific hardware specifications, when applicable, that pertain to security and regulatory compliance.

"They are still failing to address the technical debt inside their operational environments directly," adds Townsend. 

To effectively manage IIoT technical debt, experts recommend the following:

Regularly Evaluate Asset Inventory and the System Architecture: It's critical to take periodic asset inventories so that the organization understands the devices in place. Regular reviews of the IIoT system architecture can also help identify areas that may require updates or modifications. The review would include hardware and software components, communication protocols, data flow, network gear, and data storage. "There are tools in this space that focus on passively reviewing inventory so that you understand if your IoT devices are properly patched, configured properly, communicating over insecure protocols, among other traits, so that gaps can be identified," says Townsend. 

Prioritize Technical Debt: When paying down technical debt, it's essential to prioritize the most critical environmental areas first. This often includes a calculation involving the asset's business criticality, the level of risk it poses, and the chances that the device is likely to be targeted by threat actors. "When you can start taking a more risk-based approach to technical debt, you can start addressing the technical debt in a more measured way. You can also articulate a plan based on an accurate asset inventory and risks identified," says Townsend. 

Embrace Open Standards: Using open standards for communication protocols and data formats can help ensure interoperability and facilitate future upgrades.

Plan for Scalability: Always consider the future scalability of the IIoT system, including the ability to add additional infrastructure, a modern management plane, new devices, sensors, and analytics capabilities. A scalable system (built on open standards) is less likely to accrue technical debt over time.

Get Organizational Buy-in to Pay Down Technical Debt: It's tough to pay down technical debt if the organization doesn't want to invest. "There are plenty of technical issues that we discuss regarding technical debt. It's not really where the biggest challenges are," says Townsend. "It's organizational pushback that is one of the biggest challenges. Ultimately you need the resources and the budget to update the IIoT infrastructure. It's hard to tell someone who has a preconceived notion that security will impact operations negatively to implement security when the operations are literally what is creating the revenue for the business," says Townsend. 

Townsend recalls a time when he was onsite with a company that had OT regulatory requirements that they had to meet, and the CISO believed the OT security program was underfunded. 

"He just sat down with his board and drew two bars. One bar was about three times the size of the other bar. The CISO asked a simple question: which one of these is our IT security budget? Which one of these is our OT security budget? Because the IT security budget, which is the more expensive, is focused on securing IT assets, the back-office operations. The bar that's one-third the size of that bar is our OT security budget. That's the budget that secures the actual revenue for the company," recalls Townsend.

The demonstration worked in getting much of the budget the CISO needed.

By ensuring that business leaders understand that technical debt in the operational environment is a business risk, they will be much more likely to provide the funding necessary to pay down that debt. After all, technical debt isn't just a technical problem and can bite an organization with real-world consequences. Just ask the airline industry.