Topics:
RunZero Director of Applied Research Rob King joins the Nexus Podcast to discuss the security risks and exposures introduced by digital transformation to operational technology environments.
As many OT and cyber-physical systems assets are connected online, there could be signification exposures introduced to these internet-facing devices and systems. Rob also discusses the effectiveness of popular mitigations such as virtual network segmentation and other controls.
MIMOSO 0:14
All right, welcome back to the Nexus podcast. Rob King is my guest. Rob is the director of applied research at Run Zero, and lately he's doing a lot of work on the security of OT assets, and we're going to spend a lot of time getting his perspective on some of the things that he's seeing out there. So good to meet you. Thanks for doing it too. Thanks for having me. Let's kind of start at the sort of the semi-beginning here with convergence, digital transformation. Is this another facet of technology where we've kind of moved too fast without dragging security along?
KING 0:46
Yeah, I mean, yes and no, like anything. So, you know, OT used to be and sometimes still is, this utterly grade-separated network, different physical media, different protocols. If you want to talk to it, you have to be on the factory floor, connected via, you know, a modem or a radio link or whatever. And that is still the case, you know, in some situations. But the economies of scale with commodity protocols like IP, TCP, uh commodity media like Ethernet, it's really, you know, we're we're chasing that uh the abil that scale and that price point. So we've started lifting OT protocols into the TCP world. And that's great because it makes them more accessible. It makes them easier to see, it makes them easier to talk to. That's worrisome because it makes them more accessible, it makes them easier to see and easier to talk to. So if you can see it, your attacker can see it. Uh, it now could theoretically talk to the whole public internet, which is a whole thing.
MIMOSO 1:48
So obviously you're seeing some really big risks being elevated. Is access to these assets the the big one, uh, or is it just kind of like the front door?
KING 1:58
Well, so OT devices are interesting. And and I want to say there is a huge amount of work in improving security in OT, a huge amount of protocol work, a huge amount of uh you know OS work, patching, and all that. But OT devices have a very long life. You know, it's not unusual to see an OT device that's 25 years old. That's not weird. And those devices, maybe they can't be patched, maybe they can't run the newest software, maybe they can't run the secure protocol. Uh maybe they can only be patched once a year when the factory goes down, you know, that particular line goes down or whatever. Maybe the vendor's gone out of business. That's a thing that happens. That happens, yeah. Um and so while so while we're doing while security for OT is improving leaps and bounds, you still have this long, long tail of devices that are kind of hanging out. And so those can't be patched. And so there's still this huge attack surface, and it is shrinking slowly, but it's certainly not going to shrink particularly quickly. Um and also there's this is a a sea change in the way that these devices are connected. And so a lot of times security is I don't want to say it's an afterthought, it's just a different um priority, right? Like this device needs to make sure the motor spins, everything else is secondary.
MIMOSO 3:18
So and then you have I guess yeah, I have to advocate a bit to engineers when you're talking about OT in terms of securing devices and just kind of understanding what they do every day, and then they then understand what guys like you do every day to protect these these assets, right?
KING 3:35
Yep, yep. And but it's interesting too because these devices, because they are what they do, um to to a greater extent than just about anything, you know, there's no uh in in the IT world, there's this uh history of huge security breaches. Uh and it's been sort of beaten into the industry that security breaches are bad. I mean they are bad, right? Um whereas with OT, that's still sort of maturing uh from a practitioner standpoint. And that's not because the practitioners aren't trying their best, they absolutely are. It's just it was a different focus for the industry.
MIMOSO 4:14
So and in terms of you know, attacks against IT for you know two, three decades now, we understand them and we know how to deal with them for the most part, or at least try to with OT, we still not seen enough, you know, at any kind of scale attacks against OT to uh I dare to say take them seriously, but you know what I'm going here.
KING 4:36
Uh so again, everybody, I mean pretty much everyone is absolutely doing their best. But fac i it's very easy to from you know, relatively speaking, to go in and create a new cloud instance of something running the latest patched software. Nobody has a spare factory. Nobody wants to spend the millions of dollars an hour that a factory is going to be costing when it's not producing something to go in and update upgrade devices and patch devices and test devices. So the fact that security is being thought of at all is, I think, great. Um and I think that the the easiest bang for the buck is gonna be segmentation. Um just because these devices, you're always gonna have these older devices, you're always gonna have these unpatched devices, the only way to protect them is to not let attackers get to them. And how successful is that today?
MIMOSO 5:33
What's the reality about segmentation?
KING 5:36
Um it's not as good as you would think. Uh it's better than you fear and worse than you think, maybe is the way to put it. Um there's still, to this day, hundreds of thousands of OT devices just connected directly to the public internet. Um the other day I scanned for every device speaking Ethernet IP on IPv4 and found several thousand. So they're they're there, and I could talk to them, and they were very happy to just tell me whatever they wanted, whatever I asked. Um don't try that at home, kids. So uh there's that's basically zero segmentation. Then there's the segmentation problem of a lot of devices will bridge networks and you won't even know. Um, for example, if you have a, say you're a developer and you have a laptop and you're running Docker, Podman, or or something, a lot of times uh various runtimes will enable routing on your laptop so that those machines can talk to each other. But that also turns your laptop into just they don't limit it to just those machines, so now your laptop is a router. And if you connect to multiple networks, you're now a bridge. Um we see that with printers, we see that a lot with SIP devices, uh Ethernet IP devices. You know, you might have multiple links to two different places and they don't show up in a top in a topology diagram. So segmentation is important, but segmentation only works if you can verify it and actually know what is bridging those segments.
MIMOSO 7:02
And is that a gap in terms of capability, that verification?
KING 7:07
I I think so. Um networks are hard. Yeah. Uh they've gotten so much better. Um I can tell you all sorts of war stories from the early 2000s with multi-protocol routing that would uh would make you cry. But it's gotten significantly better, but it's still hard to tell. Uh especially if something doesn't talk or doesn't talk much. Um you basically have to go in and verify configuration or do something like an active scan to see what can be reached.
MIMOSO 7:34
Yeah. Let's talk about the real bugaboo too in terms of vulnerability management, exposure management. Is it smart to think uh about these issues in terms of exposures versus vulnerabilities when it comes to OT and why or why not?
KING 7:48
I I think so. I think that's a really good way to put it. Um because everything's always gonna have vulnerabilities. And and the thing I keep harping on is that you know OT doesn't get patched as often and sometimes can't be patched. So it's always going to come down to exposure. It's always gonna come down to can I actually reach the device to exploit it? And even if there's not a vulnerability in the classical sense, um, these devices are controlling and affecting things in the real world. And so even just overwhelming them with traffic, even if nothing's malicious, can still cause serious real world problems. So, yeah, I would say exposure is probably the best way to think about OT security. So is that how you're defining exposure, for example?
MIMOSO 8:30
Is it kind of the reachability part of it?
KING 8:32
Pretty much, yeah, reachability and um I love talking about segmentation violation, and I love saying it that way because it sounds like you know the old segfault Unix thing, so I can see I get to conflate the terms, it's always fun. Um but yeah, segmentation violations are uh are a big deal to me. And I think I think that they are gonna be the next uh the next frontier, which is a great choice of words for um for finding for for protecting and finding uh holes in your armor for your IT network.
MIMOSO 9:00
Alright, so let's talk about the offensive side of this, what attackers are doing to kind of leverage these exposures. And you you hinted at it. Sounds like they're fairly easy to enumerate, given Shodan, Census, whatever tool you you prefer. Um how low are these barriers to entry?
KING 9:18
So security by obscurity is not security at all. Um a lot of these protocols are paywalled or hidden or unusual. And it's kind of interesting in that there's a lot of uh older protocols, proprietary protocols, there's a huge amount of common knowledge that floats around about how these protocols work. And sometimes sometimes that's correct, and sometimes it's not. Um but what it's led to is a I'm trying to think of the best word, not not amateur, but uh non-practitioner view of how to enumerate these devices and how to find them and how to discover them and how to talk to them. And sometimes it's not safe, right? Like you we've seen devices where trying to enumerate them using what is the the quasi-standard out there will damage the device, will cause it to go into a fail protect mode, and it's a huge deal. Um so having these protocols be relatively obscure. They're not just HTTP. They're they're SIP. SIP's a whole thing. SIP's not even just one protocol, right? Uh variants on Modbus, variants on DNP3, um, even just accidentally mis uh addressing your DNP3 discovery could cause something to stop talking because it'll only talk to one primary at a time. You know, it's a huge deal. So yeah, finding these devices, enumerating these devices does come with risks, but it can be done safely.
MIMOSO 10:50
What kind of device asset information is being returned? You you said you can you can if you can talk to a device, what's it going to tell you? Is it strictly version number, stuff like that, or does it go a little deeper?
KING 11:04
So it's a lot of these devices. So what's interesting is the older you go, basically there's zero uh self-identification going on. Um like if you go and look at just pure classic modbus, the best you're ever going to get is on a serial network, it'll say I am device ID two, which tells you nothing. Um but then you get on the other end of the spectrum, uh things like SIP, which is a virtual protocol that has multiple different adaptations onto different network layers, Ethernet, CAN bus, device net, all that. With SIP, you can get an immense amount of information. You can say, give me your uh identity, your serial number, your manufacturer, your model name, your product ID, uh, you can ask all, you can introspect all of the uh IP addressing on it, you can enumerate the backplane and find out what's connected serially to that device, which is uh super fun, actually, because you can see some really neat things. Right. Um I love doing that. Uh so you can get a lot of information back, but it's not always super useful. The number of devices that I scan, and it's you know, they all have serial number 000 or uh manufacturer unknown or um some just random long string of digits. There's a certain OT manufacturer who I will not name, who their uh their internal IDs do not correspond to anything you will ever see in a product catalog at all. You know who I'm talking about.
MIMOSO 12:22
So I guess that was my next question is like, what what is an attacker looking for? What's useful, what's not? So how do you mask this stuff if you're right, right?
KING 12:32
And it basically you don't mask it. You segment off this network. Um what attackers are looking for is these devices at all. Um if they can find one of these devices, that already gets them in. Maybe it's useful as a pivot, maybe it's multi-homes, maybe it can maybe they can transit traffic across the back plane. Um maybe it's just now they have a foothold where they can uh you know get into the network and and and persist. Or now they just have their attack target. Um these are so high value uh in a lot of cases that even just being able to claim I can take this device down or uh or knock it offline is enough to get you know some ransom or some fame, you know.
MIMOSO 13:16
Sure. And so are we seeing attackers looking for assets, collecting assets versus organizations, I guess?
KING 13:28
So I mean, yeah, attackers are since the since the early days of the internet, back in the last millennium, for everyone who is uh old like me, that's terrifying to say it that way. Um, you know, people have been scanning the public internet. It used to be thought you can't scan the whole internet, and now it's trivial to do, right? It's trivial to scan the whole thing. So uh state actors and uh attack groups, yes, they absolutely have um persistence and lists of devices that are ready to go at the touch of a button, uh, which is technologically impressive and a little scary. And what's holding them back, I guess, is the natural follow-up question there. Um so I think so. I think the thing that's holding them back is A, they're not necessarily being held back. We see active attacks, um, we see vulnerabilities on uh OT devices getting added to the CISA Kev list, the known exploited vulnerabilities list. So these are being exploited in the wild, often by state actors, um foreign militaries, you know, spy agencies, three-letter agencies, all that. Um but another thing that's holding, I think, some people back is for, and again, I don't want to use amateur, but bang for the buck for certain things like ransomware or whatever is a lot easier to do and a lot faster on an IT environment like, you know, a hospital or an office, right? If your goal is just to make money and not sew chaos, there might be easier targets out there.
MIMOSO 15:04
So most of the attacks might be disruptive versus destructive.
KING 15:10
Disruptive versus destructive, uh, which is my new band name, actually. And um they also are definitely going to be used as uh the even just the act of discovering itself is like I was saying earlier, it can be dangerous, right? So, and they don't care. They really don't care. Yeah. So segment your networks, people, please.
MIMOSO 15:32
And in terms of the attack groups themselves, you brought up state actors a couple of times. Are you seeing you know, lesser skilled groups kind of jumping on the geopolitical bandwagon, and you know, I don't like what XYZ countries doing are gonna help them out even if I'm not on the payroll necessarily?
KING 15:51
Um everyone who says they're not on the payroll ends up on the payroll eventually. But uh but yes, I do think that there is um, you know, everybody talks about anonymous, nobody knows who anonymous is. Anonymous is not one thing. Um, but there are people who do threaten these devices, and there are people who just go and, you know, they'll probe a device and maybe knock it offline and not care, uh, and then go and brag to their friends, you know, hey, I I took down what you know, this thing at the tire factory. Uh and you know, it might be so simple as just they go in and they reset the device, and the factory is fine, and you know, but they add redundancy, but it might be that there was an incident and now it has to be filed, and it's you know, cost money, and now you've got to figure out what to do. And it's uh and that's that's what we're paid to do.
MIMOSO 16:40
So these groups aren't necessarily a front, they just get recruited eventually, or um I mean it it all kinds.
KING 16:48
Um and also there's the whole point of plausible deniability, right? Like a lot of these groups are, you know, I am not affiliated with XYZ, because, you know, sure sure you're not, buddy. Sure you're not.
MIMOSO 16:59
And just from your experience, why are these assets online exposed to begin with?
KING 17:05
Is it a remote maintenance kind of issue or just um it's yeah, so some of it is, you know, convergence happened in the OT world by OT standards, ITOT convergence happened extremely quickly. Right. So a lot of these people were like, we're told, you know, now we're gonna use this and we're gonna get this online. And they don't necessarily know that if you connect it like in a certain way, it's going to be exposed to the public internet. They don't know, they don't need to know, that wasn't their job. So for a lot of it, it was they did everything correctly, but things like IP management and IP network segmentation were not ever part of the playbook. Um and now they are. Like I said, people are working very hard and doing a great job, but it's still relatively new in the grand scheme of things.
MIMOSO 17:53
And talk to me a little bit about some of the legacy protocols and then the security issues that you see there. Um I know there are a lot that have been there for decades and just kind of they work, nobody wants to touch them, but there's some definite gaps.
KING 18:09
Yep, yep. You uh you can't see me out there in in Radioland, but I'm smiling because I really love writing protocol parsers. It is like the best part of my day. Um so these protocols are extremely uh convolute. So every OT protocol is either one byte header and one byte command, or an entire XML object model that has, you know, deep trees and whatever. There's no in between, and I'm only being slightly facetious. So these protocol implementations tend to be very, very uh boutique, right? There's not a huge market for Ethernet IP SIP implementations. There's not a huge market for modbus implementations. And they were coming from a place of when they were written, these systems need to be online, and if you get a message, you need to turn the rotor 90 degrees and don't worry about anything else. So these protocols don't have encryption. They do now, some of them do now, but these older protocols they don't have encryption. They have um things like DMP3 when running over TCP are basically the entire physical and addressing layer and everything else lifted and put in top of a TCP session. So now you have two addresses. Um it's getting out of hand. And uh so this all leads to, you know, parsing issues, um, vulnerabilities where even just uh exposing configuration, exposing um uh management interfaces, and when I say management interfaces, I don't mean the interface, the physical interface, I mean just there's a SIP object that you can touch and it does things. And and that actually kind of brings me to another thing where a lot of these protocols are, and I'm not saying that I could do better, but a lot of these protocols are simultaneously under-specified and over-specified. There's a lot of things that you would think you should be able to do that you just can't in the standard protocol. It's just not possible. And so there's a lot of proprietary extensions to these, and every proprietary extension comes with a risk. Um, you know, not every SIP object is going to return the same attributes. Or if you want to enumerate, uh if you want to get the number of backplane slots on an RSlogics controller, the easiest way to do it in standard SIP is to kind of enumerate everything. It's a little weird. Uh, but there are proprietary um vendor objects that will tell you more information. But those are hidden and secret and undocumented, and you got to use the the vendor's tool, um, which is super fun, don't get me wrong. It's like I love working with that sort of thing. But it's a cre it increases the attack surface, it sh it uh it gives you more things to test, and not necessarily everything gets tested, because that's from a security standpoint, because that's just not the priority. Is it relatively easy pickings? I mean it's depends on what you're trying to do. Yeah. Um basic identification of OT devices and basic speaking of these protocols has come an enormous way. Um, Ethernet IP, I love you. I can just send a broadcast UDP and I get back so much information, you're so great. Um, and then you get to you know, actually talking SIP and virtual objects and classes and interfaces, and and I I I still love you SIP, but man, you're making it hard. Uh so but it's it's a lot of fun to write these, write these parsers and these probes. I do love it.
MIMOSO 21:31
Aaron Ross Powell And from a uh a factory's point of view, or you name it, is it a huge lift to go from Modbus to the more secured version of Modbus that includes encryption and and whatever else?
KING 21:42
Aaron Ross Powell So yes, in that a lot of these devices, and it it all goes back to these devices are often old, often underpowered, or not, I'm sorry, not underpowered. They have to be exactly what's going to work in a hot and noisy, vibrating environment, right? So you're not going to get the late. And greatest are going to get hardened and simple. And that's that's a trade-off that is important to make. But that means that they're not necessarily going to have a lot of extra CPU time to encrypt messages or things like that. Even if they can be upgraded, right? If they're old, if they're new enough to even have that patch supported. And there's a lot of devices that only tangentially support these protocols. The number of proprietary protocols and proprietary tools is just massive. Even devices that claim to support standard protocols, often you can't do something important via that standard protocol. You still need to use that vendor's tool.
MIMOSO 22:39
Alright, so before we wrap up, let's talk a little bit about kind of from the defensive point of view. We've talked about segmentation, an important tool. What else? I mean, is it is it a matter of just keeping these things off the internet?
KING 22:53
I mean, definitely do please keep these things off the internet. Um segmentation is definitely, I think, the most powerful arrow in your quiver. Um but just like you know, these these devices are kind of sort of IT devices now. Patch management is a thing. Um, vulnerability management is a thing, risk acceptance is a thing, and risk acceptance has always been a thing in OT networks, but now you have to add the dimension of you know cyber attack to your risk assessment. Is it worth not patching this device until the next, you know, factory downtime maintenance window in three months? Yes or no? And oftentimes the answer is going to be it's not worth it, right? Because shutting down a factory and stopping making, you know, tires or blocks or or whatever your factory makes is more important. Yeah, because that's the goal.
MIMOSO 23:44
It's probably a lot easier to quantify in a factory, too, if we're down for an hour. It's I know it's gonna cost me exactly six million dollars. Yep. Yep. No one has a spare factory. Right. All right, Rob. Thanks so much. Appreciate it. It was great to be here.
KING 23:56
Thank you so much.
Michael Mimoso is Director of Influencer Marketing at Claroty and Editorial Director of Nexus.
