The Supply Chain Reaction: Lessons from SolarWinds

Hank Schless  00:00

Hey everybody, happy New Year. Welcome to a new episode of Endpoint Enigma. I'm your host, Hank Schless. And we're excited to be kicking off 2021 with you. One of the things that is increasingly complicated for security professionals these days is how connected everything is, especially with the number of third party tools, applications, integrations, everything that people have. You know, for example, and in the mobile world, you may have a single app developer, able to push updates to millions of devices around the world. Now recently, we've been seeing a lot about a threat actor who took advantage of that interconnectivity, and was able to push malware out to hundreds of government and private organizations. And my guess is you can guess that we're talking about: SolarWinds. So joining us today to talk about this a bit is Tim LeMaster, who is Lookout’s director of systems engineering and a veteran of networking, cybersecurity, and infrastructure as well. So,  welcome to the show, Tim. So, you know, this all kind of unfolded right around the holidays. We're still learning more every day, it seems there's a new extension on the report of, you know, maybe who was responsible or who was affected, what was done? Can you kind of bring us up to date on what happened and what we do know?

‍

Tim LeMaster  01:33

Yeah, I can try to summarize. Where we're at, as of today, like you said, is an evolving threat and evolving research and our analysis of this compromise. But what we know so far is that earlier this spring, a company called SolarWinds, was compromised, the compromise actually probably occurred in late 2019, I don't have the exact dates. It looks like the threat actors were able to compromise their built environment, such that when a new customer, or existing customers or new customers, downloaded the SolarWinds software, they got a Trojan horse, which allowed this threat actor to compromise that agency or that company, and move laterally throughout their environment. What's impressive about it is just how sophisticated this threat actor was and how long this went on before it was detected. And…

‍

Hank Schless  02:32

It seems like that happens a fair amount, you know, someone's able to get in and sort of evade detection for a while. And what I'm curious about, honestly, is both as a practitioner and someone who speaks to a lot of people about things like this, what your initial reaction was… Shock? Was it surprised? But how did you feel?

‍

Tim LeMaster  02:50

You know, it's an interesting question, when I first heard about it, you know, you hear about these things frequently, especially in our space, where we're reading about them. We're hyper focused on these kinds of compromises. So when I first heard about it, I presumed it was specific to SolarWinds, I didn't really understand the breadth of the impact. And in fact, what I heard about first, was the FireEye compromised? FireEye is such a well known brand name, and for them to get compromised themselves, because they're one of the top forensic companies in the business, was significant news. And then as time went on, a few days or a week later, we learned that that compromise was probably a result of SolarWinds. And we began to piece together more components and more understanding of it. And then I went and read FireEye’s analysis of the malware and the compromise. And then as it continued to evolve, I began to understand the breadth and the sophistication of this and it… it fascinated me, it was actually, sadly, in some ways, kind of exciting, you know, to read about.

‍

Hank Schless  03:56

Yeah, and actually, same here, I was… I'd heard about the FireEye one first and then kind of came back to SolarWinds. And one thing I thought was actually pretty impressive on the part of FireEye was how quickly they came forward and said, “Hey, you know, here's what happened, here are the tools that were compromised.” You know, just kind of gotta give an a-plus and kudos to them. It's something that no one ever wants to have happen to them. And, you know, they definitely did the right thing in coming forward, helping other people stay informed. And now, on top of that, you know, I think that the security community in general has done a pretty good job of supporting them, really kind of showing that it's a team sport. 

‍

Tim LeMaster  04:32

Yeah, that's a great point. Yeah, team sport is a good way to describe it. Although all these companies compete at some level, or many of them do. You know, at the end of the day, we're talking about customers and government agencies and large enterprises against an adversary that is, at least within the press and the analyst community, thought to be a nation state, Russia specifically. So we all want to obviously do the right thing and do everything we can to detect and prevent this.

‍

Hank Schless  04:58

Absolutely. So we'll… Before we dive into really how sophisticated this threat was, or is, because it's still, you know, it's still out there. What exactly did the threat actor do to get the malware installed? But really, you know, the term, the term “backdoor” has been thrown around a lot in the media. What does that refer to? And then on top of that, do you think there's anything else that people need to know about what the actor did once they were able to install this malware?

‍

Tim LeMaster  05:26

Yeah, so a backdoor is a term that's commonly used in the threat community, typically to describe some alternate access method to, you know, a software environment or solution or maybe even hardware. But, you know, on alternate access methods, let's say it's sometimes sort of associated with a developer leaving a backdoor for himself or herself to come in later and access the environment; if the user gets locked out, or something like that. In this particular case, though, what was interesting was, this was really a supply chain compromise. So, the bad actors, they compromised this built environment at SolarWinds, so that when the users downloaded this update, or new version, of SolarWinds, the threat actors would, you know, have their Trojan horse malware inserted. And then they could leverage that position to spread laterally and access other solutions, other information in that organization. So this concept of a backdoor is really about creating an environment that allowed that attacker to come in later and access a broader range of services and information.

‍

Hank Schless  06:47

And not just a broader range of services and information, but a pretty broad range of targets as well. I mean, you think about the web that is spread out when one software update gets distributed out to you know, X number of, you know… SolarWind says they have about 18,000 customers, but nothing's been confirmed that every single one of those is compromised. It seems now about 250 of them, I think, is the latest number. Is that right?

‍

Tim LeMaster  07:11

That's a number that has been used, or I've read. I think anytime an adversary has a savvy win like this one; leverages this kind of a backdoor to access an environment; they don't want to burn their access too quickly. And so they want to be very selective about where they exercise that access privilege, and only use it in a place where they see the most value. So, you know, a small organization –– that wouldn't be a high-value compromise for them. They wouldn't even bother to exercise that malware. Whereas a large government agency or fortune 500 company or something like that, where the value of that information might be much higher, that's where they're going to apply or leverage their access to get more information.

‍

Hank Schless  07:54

Right. It's almost a question of resources to, you know… where are you going to focus your efforts, the resources that you have, you know. Even if it's a nation state that's backing all this, everyone's got some limit on resources. And to your point, that's something that, actually, when friends and family have reached out to me about this, saying, “Oh, I saw this thing in the news,” you know. Especially over the holidays, people were asking me about it, you know: “There are all these companies, is Russia gonna take control of everybody?” Obviously, it's, you know, everyone who has this type of software, may be at risk. But if you're a smaller company, a large country isn't going to spend the time and resources to do that.

‍

Tim LeMaster  08:29

Most likely, not just they don't want to risk getting burned, of course. Now they have been burned. And, of course, the word is out around SolarWinds. And everybody, almost everybody's trying to remove that. But now we're learning as incident response teams continue to analyze this, we’re learning other vectors, other methods that this actor has been using to get access. So, as this continues to evolve, as you pointed out at the very beginning…

‍

Hank Schless  08:53

Right, absolutely. So, to bring it back a little bit. One term that you use that I think is, it's been thrown around, “supply chain,” right? People think about supply chain and they think about physical goods. But the fact is that there's a software supply chain as well, you know, developers developing an application, delivering it to the end user. So where does that software supply chain play into this? And has this kind of brought light to that as a threat vector?

‍

Tim LeMaster  09:21

Yeah, supply chain is an issue that has long been considered not a viable, legitimate threat vector. For a long time, it was primarily thought of as more of a hardware issue, right? If you're building routers or switches or firewalls or something like that. But in the software community, it's long been understood that there are supply chain concerns there as well. Both software, you know, solutions as a whole solution is not developed all internally in-house by one software development team. It often takes in open source code and components from here and there and bundles that into an overall solution. And in fact, we saw an example of this a compromise of supply chain back in 2015, I think it was. There was a compromise of the iOS Xcode development environment. It’s not exactly sure it wasn't the development environment. Let me describe that a little bit further. Xcode is the software that's used to develop iOS applications. And when Apple releases new versions of iOS, there's often the scramble for developers to recompile their application to operate in that new environment. So when they go from 11 to 12 or 13 to 14, or something like that, the developers of those apps want to make sure their application works correctly in that new environment. They want to do that quickly. And so they'll look for the Xcode for the new version, and download it and try to recompile your app using that. And in 2015, some application developers did that and downloaded a version of Xcode, not from Apple directly but off the internet, that had been compromised with a Trojan. And so as they recompiled their apps, they did exactly what we're talking about here. They compiled the malware into that code. And so that was an example of where we saw a supply chain compromise applied to mobile back in the 2015 timeframe. And that continues to be a possibility with almost all applications, because over the years, certainly one of the things we've almost always seen is very few mobile apps are developed from scratch, almost all of them use additional libraries, advertising SDKs and other sources of code that they have to bring together to build their applications. Yes, absolutely.

‍

Hank Schless  11:49

There's a lot to unpack with all that. I mean, we could take this a number of different directions. But we'll save that for another episode. But in talking about iOS, you know, I did have the question, and it was actually one of the first things that my father, who loves to reach out to me about these things, called me and said, “Hey, is this, you know… Can something like this, this type of supply chain attack, happen with a mobile device?”

‍

Tim LeMaster  12:14

Yeah, I think it is. You know, one of the things that is also well understood in the security community is that a well-resourced attacker with lots of time and patience and focus, dedication, whatever term you want to use, you know, can often find ways to attack and compromise software. It's just a matter of focus and resources. And so, is it possible? Absolutely, yeah; we saw it in 2015. We've seen it in other examples as well. If you think about the number of advertising SDKs, for example, that go into most applications that we all use, most of the applications we use on our phones are third party apps that we don't pay for; they're free. Because we don't we don't like to pay for them, we download a free version of Evernote or a free version of whatever it might be right. And those applications, most of them monetize themselves through advertising. And those advertising SDKs… we as consumers don't often know much of anything about the makeup of that app. And if it updates tonight, when I go to bed and plug my phone in, I get a new version of it. I don't even know what changed. So something could have been compromised in that, that new version, and I would have no insight into that, no visibility to that typically.

‍

Hank Schless  13:43

It almost ties, I think, to something else that we talked about a lot. Which is… which is your trust, like you said. You know, we trust our phones to be safe. We turn on automatic updates, you know. We do these things, and that's fine. Like there's nothing wrong with doing that. In fact, we should be doing that. Because a lot of times those updates are security related or, you know, something that improves the experience and… But looking at something like zero trust, does that tie in here and does it kind of help with that approach that everything needs to be validated before allowing it to touch –– even personally on a personal level –– but also corporate resources?

‍

Tim LeMaster  14:23

It really does. Zero trust is a concept that is again, you know, sort of talked about a lot in the community. But it is applicable here. If you think about the idea of zero trust, the concept is that you need to extend the security analysis from a traditional perimeter approach out to the endpoint. And don't inherently trust a user or a device or a software service just because it has traditionally always accessed or it has the right privileges or seems to have the right authentication token or something like that. You want to inspect that further and apply this, you know… Don't inherently trust anything inspected every time continuously.

‍

Hank Schless  15:11

And so how do you implement that? That sounds like a big, big lift.

‍

Tim LeMaster  15:16

Yeah. So you know, it depends a little bit on your environment. Obviously, you know, in the mobile space, where we spend the majority of our time, what that means is pushing the security out to that end device, that mobile device, and validating the state of that device before you allow it access to resources, whether it be email or other valuable services or something like that –– internal data stores, that sort of thing. So, you want to validate the state of the device. Is it secure? Is there, you know, malware on it? Has it been compromised in some way? So, you want to do that as well as checking other aspects of it, or the accessing from… Maybe you would check the IP address for a source address validation kind of thing or other context associated with that device before you allow access to sensitive data stores. The more, you know, variables you can analyze, the more confidence you can have that this is a trusted device or a trusted user of the device.

‍

Hank Schless  16:28

One thing that I've seen more recently is something like, you know, people have obviously been using multi-factor authentication or SSO or things like that for a while. But something that I've seen pop up a bit recently has also been the move to password list technology. But the fact is, just for your point about making sure the device is secure, is that those types of authentication technologies often use a smartphone as the second validation. So, you know, that is an element of zero trust –– having MFA in place –– but whatever the device is you're using, it's that second validation token, I've just in conversations with kind of people in the in the industry recently, that’s helping them understand that even that device can be a compromising factor. If it's not, you know, if it's not monitored, it's not locked down.

‍

Tim LeMaster  17:18

That's exactly right. In fact, in this particular compromise, there has been some reporting that this adversary was savvy enough to compromise or essentially forge a token associated with a second factor of service. And they leveraged that to give themselves access to email services and that sort of thing. There's been some reporting around SAML certs that SAML tokens were used to allow themselves access to other resources inside the environment, once they were in there. So a second factor is absolutely still necessary and a really smart security service. But again, on its own is not sufficient. You need that multiple layers of defense kind of approach.

‍

Hank Schless  18:05

Yeah, absolutely. Absolutely. All right. Well, look, Tim, I think that's most of what we have time for today. But thanks so much for joining us. Thanks for your time, your expertise. It's always a pleasure.

‍

Tim LeMaster  18:17

Thank you, Hank. It was my enjoyment to be here.

‍

Hank Schless  18:20

Glad to hear. Thanks so much, Tim. And for everyone listening, thanks for stopping by. To learn more about mobile security, be sure to visit our blog@blog.lookout.com We do have one up there regarding what we talked about today. And follow us on LinkedIn and Twitter, @lookout, as well. Thanks so much for joining us, and we'll see you next time.

‍