Why The xz utils Software Vulnerability a Big Deal
How did it happen, what are the implications, and what can you do about it?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: Data breaches | Application Security | Cybersecurity
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You may have heard of the xz utils vulnerability by now if you are on social media, following me or anyone else who works in security. You can follow me @teriradichel on x. I generally write about things that happen in the news but not immediately, because I find that when an incident first happens, there is an exceeding amount of noise, drama, incorrect perspectives and speculation. I like to wait for the facts and enough details to develop an informed opinion.
In this case, we may not entirely have all the facts yet, but evidence from top researchers seems to point that this was a nation state effort — meaning some government carried out the actions that led to this code vulnerability — rather than some random developer in China. And who knows if it even came from China or if someone connected to a Chinese network and took actions from there to make it appear as if it was from China. We may never know for sure. But here’s what we do know.
The xz utils vulnerability problem in a nutshell
There is an open source (free software) library called xz utils that is used in a lot of operating systems like certain versions of Linux and Windows, as well as other software such as Home Brew that deploys software on Apple Macintosh computers.
Someone who was helping write that software library inserted some malware into that code. Whomever knew that malware existed would be able to take over your system and control it or steal data from it.
Since that software was included in so many different types of software including the operating systems that run on so many computers, and software used to download and install other software, the potential impact or blast radius (another term in my book) is huge. Think of all the systems out there that could be infected and infiltrated with this software and how many people or companies could be affected.
The impacted systems also include network devices and your internet of things — iRobots and refrigerators and Apple TV and Roku devices or Ring doorbells if they use that particular software package. (I don’t know if they all do, but chances are high.)
So that’s why it’s a really, really big deal!
No one is liable?
The other significant problem with the xz utils software vulnerability is that there is no vendor to assume the liability for the harm this software might cause. Liability refers to who you can sue and seek damages (money) from in court for the problems this software vulnerability causes your business.
What if software with a vulnerability allows an attacker to infiltrate your network, install ransomware on all your systems, and you can’t get your data back? What if your customers sue you for losing their data?
If the software with the vulnerability is open source (meaning free software on the Internet maintained by “the community” as people like to call it instead of any one company) then who can you in turn sue for the damage caused by that vulnerability.
The European Union has been exploring this topic of liability and at one point late last year, they were going to make the people who maintain the repository where the free code exists liable for damages. The people who maintain the code might not write all the code. They facilitate hosting the code in a code repository and determine who can add new code to that repository.
The maintainers are usually working for free and have more work than they can possibly do to support this free and open source software. They also might not be well versed in security and relying on various sources to review the code or they may not review the code at all.
If the European Union had held these maintainers liable, that would likely cause a huge amount of problems in the industry since people who maintain critical software might completely stop supporting it for free as they do now — and the people you don’t want maintaining it like the person who inserted the malware into xz utils package will gladly step up to “help.”
That might cause more security problems than it solves. These maintainers do not always have the resources required to write secure software and leverage secure development software practices. That doesn’t mean the software is bad, it just means there are reasons why these vulnerabilities may exist. We want people to be able to share ideas with the world, but at the same time, someone has to review this software before using it in their systems and software products.
Ultimately the EU didn’t go through with the initial plan to make the maintainers liable. So who is? Your company is likely liable for choosing to use that software. That also could apply to software you pay for, if you do not perform a proper security assessment when selecting that software. Is a true effort to review and select secure software and good intent enough to avoid liability? Well that’s going to depend on many factors and I’m not a lawyer, so consult with one well-versed in such matters.
But here’s another related twist. I saw that in a post on X / Twitter that a smaller company was trying to sue a large company that had infringed on their software patent and used their software without permission. However, the large company using the software claimed that the people maintaining the free (open source) software from the Internet software had incorporated that copied software into the code. The larger company had, in turn, included the free software in their own code. The larger company claimed the free software and the people who created it were to blame, not them. But who is liable for the infringement?
https://x.com/fielding/status/4043046259?s=46&t=tSBek9kyVQEcvnDNq-7qdw
That was my question as you can see from my comment on the above post. Whether or not an actual infringement exists, the question remains. Who would be liable in that scenario?
As I just mentioned, the free software maintainers aren’t being held liable currently for software vulnerabilities. So who will be liable for infringing on patents or using proprietary code in the other software? No one? How can these open source maintainers possibly review and ensure there is no stolen code in any commits made by the myriad of strangers on the Internet who request to submit code to their software repository?
So here’s the implication if that claim holds true. If a big company wants to use some software belonging to another company they just get it rolled into some open source software and then they can use it — and then neither the big company nor the free software maintainers are liable.
So what can the person that owns the software do? Sue the developer who inserted the copied code into the free software?
Maybe — if you can figure out who they really are — which is one of the problems with the code inserted into the xz utils security vulnerability.
There’s a name of a person who inserted that software but is that really the person who did it, or is it a fictitious name and email address?
Here’s what Brian Krebs, a famous security reporter and researcher, had to say after doing some research on the person who made the commits to the xz utils repository:
Some thoughts about attribution in the XZ backdoor, having just wasted so many hours digging into the details. The email addresses used for a couple of years at least by the parties involved have absolutely *zero* trace in any kind [of] data breach or database beyond Github/Gitlab, and maybe Tukaani and Debian and a few mailing lists. Normally when I see this, the assumption is that we’re dealing with a single-use or single-purpose email address that was created either for fraud or b/c someone is super paranoid about privacy. The people in the latter camp who do this tend to have other tells that give them away, or at least *some* trace or home base in the online world. Especially if we’re talking on the order of years using that address. Either way, very few people do opsec well, and for every year you’re operating under the same name, nick, number, email, etc you dramatically increase the risk of screwing up that opsec. And almost everyone does, eventually. To see this complete lack of presence in breached databases once or twice in the course of an investigation is rare, but to find it multiple times suggests we’re dealing with an operation that was set up carefully from the beginning. And that almost certainly means a group project (state-sponsored).
Here’s another story from Wired looking into that question, and you’ll probably be able to find many others like it online.
The jury is still out as to who this developer really is, and by the way, if they are in another country like China, good luck with that lawsuit.
I repeat: I am not a lawyer, so consult with one. But you may ultimately be liable for vulnerabilities in open source software you choose to use, depending on your contracts with your customers.
On the flip side, customers using software from vendors need to consult with their lawyers on the terms in their contracts with software vendors. Can you sue the vendor if there’s a problem with the open source software they use in their own software? Or has the contract excluded any liability for third-party software the vendor uses in their software, products, or services that you use from that vendor?
How do you know if that library is included in the software you run?
First, software developers are moving towards creating a software bill of materials or SBOM. In fact, including an SBOM has been mandated by the U.S. federal government for software sold to them.
They are also proposing enforcement of SBOMS for contractors.
That software bill of materials tracks all the third party software and versions used in the code and you would be able to look at that SBOM and see if this library is used by that code.
Some argue that SBOMs don’t work because the software vendor may have mitigated the vulnerability found in a software package in some way. But as I’ve written on X (Twitter), in that case, the SBOM can display the package and explain how the vulnerability has been mitigated. That allows customers to understand what is included in the software and make a reasonable assessment as to whether that mitigation is sufficient instead of having a software vendor say, “Trust us, everything is OK.”
What if your software doesn’t include an SBOM? Well, now you get to review the code yourself and try to figure out if that library is included anywhere. How you do that depends on what particular software you are inspecting and is beyond the scope of this post.
Just to show you what a code review might entail, I was looking for the xz utils repository on GitHub and found this:
Which led to this:
Where is the original source for xz utils now? You might find the answer here:
But remember that anyone can update and edit Wikipedia, governed by humans, the same way humans govern the code that gets inserted into open source software. Even finding the original source of the code and verifying that you are reviewing the correct code might be tricky. And how do you review the code every single time it is updated when you rely on countless software packages?
Whenever I am performing a penetration test or security assessment for a client I often find many different versions of the same software packages because one library might include another library and then you include the latest version of the second library but the other library you included has included an older version of that same library. What to do? It’s a very tricky problem and one that I will be exploring further.
But probably one of the easiest thing to do is to start by limiting the software you include in systems to a set of defined standards as I explained in prior posts, and reduce the code to what is minimally required, the same way you should be limiting what ports are accessible on a zero trust network.
I said it’s one of the easiest things to do…not that it is easy…it’s NOT. But it’s easier than reviewing every line of code in every software package you use and every line that changed every time the software packages get updated. I want to explore further on this blog this year as time allows. Some of my prior posts on this topic are found here.
You can also run this software in a locked down network so malware with a C2 backdoor in it cannot reach the attacker’s network. A C2 (command and control) channel is something that can connect to the Internet and send commands to run on your system or that the attacker can use to exfiltrate data. Proper network architecture can prevent that communicate and might save you when some malware inadvertently makes its way to your applications or operating systems.
Which of those options is easier? I tend to like the latter approach as a first line of defense and then implement the former as much as possible and improve on it over time.
What if I’m not that technical? What can I do?
Here are some steps you can take to protect yourself from malware like this if you are not that technical.
Turn on automatic updates everywhere
If you are not a very technical person, the best thing to do is to make sure all your software is up to date. Wherever you run software that allows you to turn on automatic updates, do it. That includes any Windows or Apple devices and any software you run on your computers or phones like Google Chrome, Apple TV, and Microsoft Office products. My husband almost makes a game out of checking to see how many application updates he has on his phone each day.
When you can’t auto-update
You can make a list of all the software you run that doesn’t offer auto-update capabilities. Make a habit of checking if it is up to date at least once a month if you can’t turn on automatic updates.
Upgrade very old hardware and software
Some very old hardware and software has vulnerabilities that can’t be fixed. Sometimes the vendor has declared that particular device or software is end-of-life. In that case, the vendor may not fix any software vulnerabilities. The good news in this case is that the new vulnerability introduced into the code may not make it to your system. The bad news is that there may be many other vulnerabilities in that code that never gets fixed, making your system a prime target for attackers. If you are running very old networking equipment that is probably the first thing you want to update as soon as you can.
Make sure your router uses the latest version of certified Wifi
You can find out the latest version of Wifi from the Wifi Alliance. Your Wifi device should be certified in the latest version. If it’s not, consider buying a new Wifi device.
Use DNS servers that block malware
I wrote about a simple change you can make to your laptop and systems to use DNS servers that block malware here.
Avoid hardware and software from sketchy sources
In addition, you can simply choose not to run software that doesn’t have automatic updates. Before you buy a device or software look at the documentation to see if it has automatic update capabilities. Also, avoid installing sketchy software you don’t need from unknown sources. Check to see who developed the software, where they are from, and their background if you can. It’s hard to figure this out sometimes and if you don’t want to do all that, stick to well known names you know and trust.
This post has some other tips:
What if I’m a software developer and I’m using open source code?
If you are a software developer, inspect the software an its source before you use the software.
- Who originally developed it?
- Who maintains it now?
- How frequently is it updated?
- Does the update process include security reviews?
- Does the software provide an SBOM?
- Ask the same questions for each item in the SBOM.
There are many other questions you can ask but that is a good place to start. In fact, if you are using open source code, you should go ask those questions about any software you are currently incorporating into your projects.
I’ve seen some vendors advertising services that analyze and try to report on the risk related to various open source software but you can perform a basic risk assessment yourself. If you rely on the vendor and the vendor is wrong you still may be liable.
You should also be developing your own SBOM for the software you write.
Back to the original problem
Again, the problem with xz utils is that its use was ubiquitous — practically everyone was using it. So you can’t just hold one or two companies liable for using sketchy software. What do we need to do to solve this problem?
This is a complicated problem with no easy answer.
But perhaps the government needs to get involved and scan open source software for vulnerabilities and weed out some of the obvious malicious code on the Internet — not denying its use by penetration test or security researchers — but rating it so that companies can make informed decisions about the security of that software from a trusted source.
The scanning process may not catch everything but at least it can identify “known bad” and risky software and make it clear to people that if they choose to use it they are putting their business at risk. The process could be similar to the process NIST uses to provide security guardrails and restrictions for government agencies that in turn become recommendations to companies related to the encryption algorithms they choose to use.
Perhaps there’s a mechanism like PCI compliance or the Wifi Alliance to prove that your software is certified. If software is not compliant or certified, that doesn’t mean companies can’t use it. It just means they should apply a bit more scrutiny to those software packages.
As with compliance, certified software won’t mean that software is “secure” but at least companies would know it had undergone some basic security scanning and testing. Perhaps to become compliant it has to follow basic security practices like code reviews by people who are well-versed in security, similar to the way a penetration test is required for PCI compliance and other types of regulatory compliance.
Those are just some ideas. I don’t think there’s an easy answer to this question, but doing nothing is probably the wrong approach.
What if the vulnerability is present and I can’t remove it?
If that vulnerability in the xz utils is included in your software, then someone who knows about the vulnerability could infiltrate your system, run commands on it, and exfiltrate data. The primary way to stop it short of removing the vulnerability is to use a tightly controlled network that would disallow connections to attacker servers.
More on network security here and how it helps in cases like this:
Hopefully that gives you a general overview of some of the issues related to the xz utils vulnerability. There are some other topics I’d like to cover, time permitting.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2024
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab