What Security Teams Can Learn from the xz Utils Attack

On Good Friday, Microsoft developer Andres Freund dropped an Easter bombshell. Whilst troubleshooting some innocuous-looking performance issues on a Debian Linux system, he stumbled on what has been described as the “best executed” supply chain attack ever seen.

It will have major implications for IT security teams, and how they manage open source risk going forward.

What Happened?

The attack was incredibly complex. But it appears to have been a state-sponsored attempt to insert a backdoor in a popular open-source component known as xz Utils. The data compression utility can be found in almost all Linux systems. The backdoor and associated vulnerability, CVE-2024-3094, is designed to inject malicious code into an OpenSSH server (SSHD) running on a victim’s machine. There are more details here, but the tl;dr is that it would enable remote attackers in possession of a specific private key to remotely hijack a targeted machine.

In short, it’s about as serious as it gets, which is why the CVE was given a CVSS score of 10.0. Fortunately, the attack was spotted before the malicious xz Utils update was merged into major Linux distributions. At that point, it could have given the threat actor behind it remote access to an unknown number of global machines.

The sophistication and patient planning that went into this attack points to nation-state backing. We can deduce this because:

• The backdoor itself has a complex execution chain comprising multiple stages
• The backdoor was introduced over multiple commits
• These commits were only included in source code tarball releases rather than pushed to the public git repository – in order to keep them hidden from scrutiny
• The operation was at least two years in the making. That’s when the malicious ‘developer’ known as ‘Jia Tan’ joined the open-source project
• It appears that the group behind Jia Tan deliberately pressured the original maintainer, Lasse Collin, into bringing Tan on board. Likely fake personas, including ‘Jigar Kumar’ and ‘Dennis Ens’ all piled on the pressure by bombarding Lasse with feature requests and bug complaints

The Open Source Security Challenge

The bad news is that Jia Tan is known to have worked on several other open-source projects. It’s unclear whether malicious code has already been inserted covertly into these and what the impact could be.

Open source is both the problem and the solution here. Its “many eyes” approach to software development should, in theory, mean that problems are spotted in time. But as seen with this attack, threat actors are going to great lengths to sneak malicious code unseen into projects.

The challenge for security teams and developers using open-source components is the intransitive dependencies which they often unwittingly introduce into code – as highlighted by the Log4j saga. Gaining visibility into such risk is critical, according to Jamie Scott, founding product manager at Endor Labs.

“When you trust one Maven package, on average, there are an additional 14 dependencies you implicitly trust as a result,” he tells ISMS.online. “This number is even larger in certain software ecosystems such as npm, where you, on average, import 77 other software components for everyone you trust. This trust relationship is established with all maintainers of all software components.”

So what’s the answer? On one level, there needs to be more done by governments, large software companies that use open source, and the open source community itself.

“Industry and the open source community needs to work more closely to secure the wider cybersecurity software ecosystem,” Cyberhaven CSO, Chris Hodson, tells ISMS.online. “The lines between commercial and open source software are opaque. It’s in everyone’s best interest to do better.”

Endor Labs’ Scott adds that industry could help by adopting artifact signing.

“Artifact signing provides some resilience to this type of attack by ensuring that only authorized pipelines can produce valid, signed artifacts,” he argues.

“While this won’t provide perfect protection, it significantly raises the cost for an adversary to get malicious packages adopted, and also aids in effective incident response by allowing responders to quickly and accurately block use of the malicious package once it is identified.”

Other much-needed industry initiatives include the adoption by default of security metadata like software bills of material (SBOMs) and Supply Chain Levels for Software Artifacts (SLSA), which can help CISOs better understand risk in their supply chains. There is also more funding for organisations like the OpenSSF, which in turn funds important security initiatives.

Next Steps for Security Teams

Ultimately, for CISOs and DevSecOps teams, the key is gaining visibility into their open source code, especially intransitive dependencies, according to Hodson.

“It’s important for CISOs to appreciate the chain of trust and interdependencies of a software library and an executable. Xz Utils, in isolation, feels like a fairly low-risk software component, but adversaries will look to exploit such software to action on their objective,” he argues. “CISOs need to obtain a much better understanding of their supply chain – not just the vendors they’re using for commercial software, but the open source components across their applications and services. Not to mention those of their third parties.”

SoSafe CSO, Andrew Rose, shares this to-do list for managing risk, with ISMS.online:

• Create a library of approved software and assets to ensure clear guardrails for developers, and limit unapproved code and software. Software should be validated and approved, perhaps from risk-assessed suppliers
• Create fake data lakes for test purposes and implement network segmentation. This means all developer environments, or places with uncontrolled builds, should have no access to real data, or production systems and services
• Use only approved, released builds in production and review these regularly to ensure any suspicious activity is monitored
• CISOs should stay connected to developer communities to learn of vulnerabilities, backdoors and issues as they arise
• Regularly scan environments, even after CISOs have cleaned the house. A ‘cattle not pets’ model should be applied to manage services or software, which means continuously assessing and rebuilding to make sure the right approved versions are being used

Endor Labs’ Scott adds the following defence-in-depth tips for reducing the likelihood of exploitation in the event a software supply chain is compromised:

• Implement least privilege: In the case of xz Utils, a least privilege approach to server and container configuration would have helped. Backdoor SSH keys allow you to access SSH when SSH services are accessible. Please don’t expose SSH ports to the internet
• Create governance policies for open source usage: These could range from “Pin all versions of open source software used so that you don’t download the latest version always” to “don’t use versions of open source software less than 60 days old
• Remove unused software to reduce supply chain risk: Unused software can create unnecessary bloat in your software and introduce supply chain risk over time.It’s only a matter of time before the next xz Utils-type attack is discovered. By taking more precautions now, organisations can proactively build resilience and ensure their path to recovery will be quicker and less traumatic.