Monday, April 1, 2024

xz Backdoor

Thomas Claburn:

Red Hat on Friday warned that a malicious backdoor found in the widely used data compression software library xz may be present in instances of Fedora Linux 40 and in the Fedora Rawhide developer distribution.

The IT giant said the malicious code, which appears to provide remote backdoor access via OpenSSH and systemd at least, is present in xz 5.6.0 and 5.6.1. The vulnerability has been designated CVE-2024-3094. It is rated 10 out of 10 in CVSS severity.

Dan Goodin (Hacker News):

xz Utils is nearly ubiquitous in Linux. It provides lossless data compression on virtually all Unix-like operating systems, including Linux. xz Utils provides critical functions for compressing and decompressing data during all kinds of operations. xz Utils also supports the legacy .lzma format, making this component even more crucial.

Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL offerings, was recently troubleshooting performance problems a Debian system was experiencing with SSH, the most widely used protocol for remotely logging in to devices over the Internet. Specifically, SSH logins were consuming too many CPU cycles and were generating errors with valgrind, a utility for monitoring computer memory.

Through sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils. On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.


This code allowed someone in possession of a predetermined encryption key to log in to the backdoored system over SSH. From then on, that person would have the same level of control as any authorized administrator.

See also:


Plans to literally “hack the planet” foiled due to 500ms of latency that Andres instinctually investigated.

The latency was due how the malicious code parsed symbol tables in memory.

Amjad Masad:

You know that annoying person on your team who insists every bit of perf regression needs to be investigated. One day they will save the world.

Perry E. Metzger:

I’ve always thought that your CI and monitoring systems absolutely have to flag performance regressions and that you have to investigate them quickly, but I never thought of it as a security issue until now.

Rob Mensching:

Lots of analysis of the xz/liblzma vulnerability. Most skip over the first step of the attack:

0. The original maintainer burns out, and only the attacker offers to help (so the attacker inherits the trust of the project built by the maintainer).

Gynvael Coldwind (Hacker News):

Someone put a lot of effort for this to be pretty innocent looking and decently hidden. From binary test files used to store payload, to file carving, substitution ciphers, and an RC4 variant implemented in AWK all done with just standard command line tools. And all this in 3 stages of execution, and with an “extension” system to future-proof things and not have to change the binary test files again. I can’t help but wonder (as I’m sure is the rest of our security community) – if this was found by accident, how many things still remain undiscovered.

Rob Mensching (Hacker News):

This thread is a microcosm of the interactions in Open Source projects. Consumers make demands (some polite, some not-so-polite) of one maintainer (rarely two) that does everything.

Make no mistake. This is the way it works.

Glyph Lefkowitz:

For most maintainers, Tidelift pays a sub-hobbyist amount of money, and even setting it up (and GitHub Sponsors, etc) is a huge hassle. So even making the transition from “no income” to “a little bit of side-hustle income” may be prohibitively bureaucratic.


Specifically, every employer of software engineers should immediately institute the following benefits program: each software engineer should have a monthly discretionary budget of $50 to distribute to whatever open source dependency developers they want, in whatever way they see fit.


This sub-1% overhead to your staffing costs will massively de-risk the open source projects you use. By leaving the discretion up to your engineers, you will end up supporting those projects which are really struggling and which your executives won’t even hear about until they end up on the news.


The xz backdoor was the final part of a campaign that spanned two years of operations. These operations were predominantly HUMINT style agent operations. There was an approach that lasted months before the Jia Tan persona was well positioned to be given a trusted role.


Every intelligence agency in the world could run this campaign, design and execute these operations. There is a serious level of technical acumen on display as well, the Jia Tan persona has to be able to do the work and talk the talk, but the core of this campaign is HUMINT.


It is important to remember that Lasse is blameless in this. There is no individual, and very very few organisations, able to detect, let alone resist!, the directed interest of an intelligence agency.

Lukasz Olejnik:

We should… maybe… resist the temptation of portraying XZ as alleged evidence of underfunded OS. Could it rather be THE evidence that resisting orchestrated and well-organised and funded campaign is hard?

Saagar Jha:

The problem is that nobody can read all this code. That’s it. You can make the code 50% clearer or reduce the number of libraries loaded or increase auditing but there is so many orders of magnitude more code being written than is properly reviewed that this can’t be fixed

It makes me so sad because I want this to be fixed and I want to go “oh if we paid maintainers some money the problem would go away” but, like, it just doesn’t seem to work. There is just so much code. We are drowning in it. The complexity of our stacks is insane

Juliano Rizzo:

Jia Tan’s git commit to turn off Landlock sandboxing one week after Lasse Collin improved it.

Note the extra period.

Simple Nomad (Hacker News):

This xz backdoor thing reminds me of a story I heard from friends that worked at a tech company that made cell phones. They had a great coder that worked on the project, he had put in work as a contractor for a few months, and due to the quality of his work he was hired in full time. After two months he simply stopped showing up to the office.

An investigation turned up the following interesting items. His account had accessed all files including source code to all cellular projects - in that he had apparently downloaded a copy of everything. He had committed a large amount of contributions to the project he was assigned to. None of his paychecks were ever cashed. A wellness check to the house he had rented was performed and the house was completely empty.

Isaiah Carew:

xz is an inflection point. people are going to lose their collective shit.

the idea that instead of breaking 4096-bit keys with gigawats of compute, or infiltrating hardened machines with 1337 haxors…

just shotgun backdoors into 1000 libraries that everyone uses.

Feross Aboukhadijeh:

The xz package backdoor is just the tip of the iceberg.

There’s a CONSTANT low-level stream of malware and spyware being uploaded to npm, PyPI, and Go registries.

I want to share a few examples from the 20,000+ malicious packages we detected so far[…]


Update (2024-04-02): Russ Cox (via Hacker News):

This post is a detailed timeline that I have constructed of the social engineering aspect of the attack, which appears to date back to late 2021.

Mark Atwood:

The xz attack was not because it was open source. The attack failed because it was open source. The way this attack works for non-open source is the attacker spends 2 years getting an agent hired by contract software development vendor, they sneak it in, nobody finds out.


The xz fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion dollar corporations expect free and urgent support from volunteers.

@Microsoft @MicrosoftTeams posted on a bug tracker full of volunteers that their issue is “high priority”

Update (2024-04-03): Russ Cox (Hacker News):

At a high level, the attack is split in two pieces: a shell script and an object file. There is an injection of shell code during configure, which injects the shell code into make. The shell code during make adds the object file to the build. This post examines the shell script.

See also:

Update (2024-04-08): blasty (via Hacker News):

the xz sshd backdoor rabbithole goes quite a bit deeper. I was just able to trigger some harder to reach functionality of the backdoor. there's still more to explore.

John Gruber:

Another is that it was very subtle: the ultimate goal was a back door in OpenSSH but the attacker(s) put their code in a compression library that was sometimes a dependency for another library that was itself only sometimes a dependency of OpenSSH.

See also: Jordan Rose.

Update (2024-04-11): Oxide Computer Company (via Adam Leventhal):

Andres Freund joined Bryan and Adam to talk about his discovery of the xz backdoor. It’s an incredible story… so great to get into the details with Andres. We started by ranting about the coverage in the New York Times… coverage that explicitly refused to dig into the details! It’s all the more shocking because the big story here is how Andres’ penchant for digging into the details is what saved us all from what would have been a pervasive and damaging attack!

Bruce Schneier:

It’s impossible to count how many of these single points of failure are in our computer systems. And there’s no way to know how many of the unpaid and unappreciated maintainers of critical software libraries are vulnerable to pressure. (Again, don’t blame them. Blame the industry that is happy to exploit their unpaid labor.) Or how many more have accidentally created exploitable vulnerabilities. How many other coercion attempts are ongoing? A dozen? A hundred? It seems impossible that the XZ Utils operation was a unique instance.

Solutions are hard. Banning open source won’t work; it’s precisely because XZ Utils is open source that an engineer discovered the problem in time. Banning software libraries won’t work, either; modern software can’t function without them.

Update (2024-04-24): Joab Jackson (via Hacker News):

At any rate, how the backdoor got so close to so many production systems may be a cautionary tale over the state of the internet infrastructure.

Kaspersky (via Hacker News):

Unlike other supply chain attacks we have seen in Node.js, PyPI, FDroid, and the Linux Kernel that mostly consisted of atomic malicious patches, fake packages and typosquatted package names, this incident was a multi-stage operation that almost succeeded in compromising SSH servers on a global scale.

11 Comments RSS · Twitter · Mastodon

Backdoored xz was in one of my MacPorts installations. I read on Mastodon that it was also in HomeBrew. But they were probably not vulnerable because that required an opensshd patched for Systemd notifications, that's where xz hooked in.

Still, do: sudo port update && sudo port upgrade outdated

Old Unix Geek

Less & Simpler code is always better.

xz is decoding a file format. It should be small and changing very little.

What will be even more fun, is when do LLMs start learning such obfuscated hacks from github and start outputting them. Then it'll be even easier to backdoor most projects.

@Ed Yes, agreed. Some of my Arch devices received the backdoor update as well and the patched OpenSSH daemon with the required hooks seems to be required for the vulnerability to function, possibly only effecting Debian and Red Hat and similar distros who incorporated those prior patches for linking into Systemd. Either way, I just updated everything over the weekend to get rid of the vulnerable code.

While I delved deeper to see which of my systems had the vulnerable version installed and did check to see what my xz package was linked to just to be sure, the bottom line, I wasn't going to wait to patch either way.

Like everyone else, I now can't help but wonder just how many widely used libraries, open source or otherwise, have backdoors in them.

I also haven't forgotten the Snowden revelation that the US government can coerce individuals to add backdoors to their software products and services, and then forbid them -- with threat of imprisonment! -- from ever telling anyone they were ordered to do it, including anyone higher up in their company. How many times has that happened? How many times has a similar sort of thing happened in other countries?

It appears that the homebrew version remains unaffected and has been downgraded in order to enhance security.

> To be clear: we don't believe Homebrew's builds were compromised (the backdoor only applied to deb and rpm builds) but 5.6.x is being treated as no longer trustworthy and as a precaution we are forcing downgrades to 5.4.6.

"This thread is a microcosm of the interactions in Open Source projects"

Reading these comments, it almost feels as if people were actively pushing xz's maintainer to hand responsibility over to Jia Tan. I guess this could mean they were accomplices, but the way people in oss talk to each other is so degrading that it's impossible to tell the difference between a good-faith actor and an attacker trying to emotionally harm a maintainer until they give up.

I wish people like Linus Thorvalds would understand the harm they cause by normalizing this kind of behavior, and would take steps towards improving the situation.

Quoth Plume:

Reading these comments, it almost feels as if people were actively pushing xz's maintainer to hand responsibility over to Jia Tan.

I assume that Jigar Kumar is a sockpuppet account judging by his behavior.

I wish people like Linus Thorvalds would understand the harm they cause by normalizing this kind of behavior, and would take steps towards improving the situation.

I disagree. My reading is that Jigar Kumar was acting like an entitled brat even though he is owed nothing — not even an explanation. While I can understand why Lasse Collin only pushes back lightly, saying he's having some mental-health issues plus other stuff, the response Jigar Kumar deserves is something like Linus Torvalds, in 2012, reiterating that Linux is not to break userspace.


> I wish people like Linus Thorvalds would understand the harm they cause by normalizing this kind of behavior, and would take steps towards improving the situation.

You couldn’t have picked a worse example. Torvalds did recognise his behaviour, apologised, and aimed to do better.

Systemd, FFS. Not content with subverting the health of the Linux ecosystem by monopolising and subverting better and more carefully written alternatives through brute force displacement of their functions, now it's directly responsible for enabling a backdoor by requiring daemons to link in bloated, dependency-rich libraries. And Debian and RedHat and all those who enabled this have questions to answer, frankly. Seriously considering going back to Gentoo after this, so I can pear things down to size. Less code, less risk.

@OUG Unfortunately in this instance the problem was the test files shipped in the tarball and malicious build scripts. Not saying it wasn't a devious case and a close call, but that's clearly a "blind spot" that should not be overlooked in future and there are interesting discussions about how you avoid it, perhaps having reproducible archive builds direct from the VCS instead of hand-crafted externally. Still, good thing a Microsofty found it, even by accident. Many eyes, eventually, etc.

Nathan: "the response Jigar Kumar deserves is something like Linus Torvalds, in 2012, reiterating that Linux is not to break userspace."

Linus: "Mauro, SHUT THE FUCK UP!"

You don't see how this is what creates the environment where shitty behavior is normalized?

Quoth Plume:

You don't see how this is what creates the environment where shitty behavior is normalized?

You haven't convinced me that it bad to be rude to people who were rude and entitled to you first. Unwise, perhaps, from a professional standpoint, but you haven't convinced me that Kumar or Ens didn't deserve stronger rebukes, both to get them to stop and to make clear that this sort of hectoring of maintainers is not tolerated.

Leave a Comment