Attacking the Linux supply chain: Simmering

April 3, 2024

Introduction

Supply chain attacks continue to increase significantly. We know that attackers always look for the weakest link in the chain as a target to initiate their attacks. In this case, an open-source library maintained for a long time by a single developer was the target of a premeditated and lengthy harassment and takedown.

Back in 2018 we already published an article in this blog about the frustration of the open-source maintainer as an attack vector, considering what happened in the well-known Log4Shell case at the end of 2021, and in this March we can conclude that the validity of this risk remains unresolved.

All of this should make the global technology ecosystem reflect on the validity, effectiveness and insufficient coverage of government initiatives, foundations such as the Linux Foundation and the large technology companies created to mitigate the scarce resources available to the developers of some of the open-source core libraries widely used by operating systems.

What happened?

Last March 29th Andres Freund (a Microsoft developer) investigating an unusual CPU consumption behavior in his failed SSH access tests, discovered "by chance" a backdoor mechanism in xz utils, a library that supports lossless compression. The library is extremely popular and is used in most major Linux distributions and in a lot of Linux and macOS applications..

This meme describes what happened after the discovery

Officially named CVE-2024-3094, it has the highest possible CVSS score of 10 out of 10 (as did Log4Shell). In the CVE Red Hat reports that the malicious code modifies functions within liblzma, which is part of the xz utils package.

This code could have posed a very serious threat to Linux if it had not been for its early detection, which has prevented Linux distributions from including this poisoned library in their stable production distributions, as it could have compromised ssh authentication on millions of servers globally.

What did this back door allow to do?

The backdoor allegedly (as its scope is still being actively investigated) intercepts RSA SSH key decryption operations, which are redirected to the backdoor code. This would allow the attacker to manipulate the SSH authentication operation and execute code on remote systems if they use the attacked library. Master key functionality was probably also added to allow attackers to access all affected servers at will.

The backdoor code was found in xz utils versions 5.6.0 and 5.6.1.

The backdoor was active for approximately one month before detection.

Why did no one else detect the backdoor?

While Jia Tan (the attacking user) added some code to the xz utils project, most of it was never added to the project's GitHub repository. The real backdoor resided in the xz utils tarballs. These are tarballs that are automatically generated whenever developers release a new version of a library.

Third-party software projects do not usually, for simplicity's sake, pull the source code and compile the entire project, they simply extract the tarball. Jia Tan modified lines in the tarball configuration to load the backdoor, which was hidden in binary test data files.

Developers usually audit the source code, mistakenly thinking that tarballs perfectly reflect the code.

In reality, tarballs and other files in software releases can be manipulated quite easily, and other threat actors have done so in the past. In fact, some members of the developer community are calling for the removal of automatically generated tarballs and other potentially vulnerable files from project releases.

Simmering

What makes this attack most special, in our opinion, is the way in which it was orchestrated and the extreme planning and patience with which the attack was executed.

It appears that the attacker initially identified that xz utils was effectively a dependency of OpenSSH. OpenSSH being a critical remote administration tool used on servers around the world.

This dependency did not exist because of a design decision by the OpenSSH developers, but because of a patch added by some Linux distributions to integrate the tool with the operating system's new orchestration service, systemd.

The alleged attacker in question, armed with that knowledge about xz, invented the persona of Jia Tan, a developer with no prior history who appeared out of nowhere in October 2021 and began making useful contributions to xz utils.

The xz had a single unpaid maintainer from 2009 until that time, Lasse Collin, who was dealing with health issues and was lagging behind on maintenance tasks. Shortly after Jia's arrival, several accounts, possibly of stooges supporting the attacker himself, began pressuring Lasse to pass the baton: he appears to have relented sometime in 2023.

Since then, Jia effectively continued the maintenance work, finally perpetrating his attack in February 2024 with the introduction of a sophisticated and well-hidden backdoor within one of the library's build scripts.

Some time after inserting the backdoor, Jia, along with a new cast of puppet accounts, began contacting Linux distribution maintainers to package and distribute the compromised library to end users.

Conclusions

If we carefully review the events, we can conclude that this is clearly not the modus operandi of an amateur. Nowadays, if you have the technical ability and patience to pull this off, you can easily get a steady job that will support you for life without risking prison time.

In other words, all indications are that this was a well-paid professional operation, and it would not be surprising if it was funded by a state actor.

The xz backdoor is not merely a technical problem and probably cannot be solved by technology alone. To a large extent, it is a counterintelligence challenge, squarely within the purview of governments and a handful of commercial tech giants with global ecosystem-level surveillance capabilities.

This time, fortunately, it seems that those affected versions were not incorporated into any production versions for major Linux distributions, but it is a desperate wake-up call:

"If it hadn't been discovered, it would have been catastrophic for the tech world."

To conclude we rescue the mythical XKCD's Dependency meme about the reality of building modern software that must be taken very seriously if the root problem is to be tackled.