xz Backdoor

Thomas Claburn:

Red Hat on Friday warned that a malicious backdoor found in the widely used data compression software library xz may be present in instances of Fedora Linux 40 and in the Fedora Rawhide developer distribution.

The IT giant said the malicious code, which appears to provide remote backdoor access via OpenSSH and systemd at least, is present in xz 5.6.0 and 5.6.1. The vulnerability has been designated CVE-2024-3094. It is rated 10 out of 10 in CVSS severity.

Dan Goodin (Hacker News):

xz Utils is nearly ubiquitous in Linux. It provides lossless data compression on virtually all Unix-like operating systems, including Linux. xz Utils provides critical functions for compressing and decompressing data during all kinds of operations. xz Utils also supports the legacy .lzma format, making this component even more crucial.

Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL offerings, was recently troubleshooting performance problems a Debian system was experiencing with SSH, the most widely used protocol for remotely logging in to devices over the Internet. Specifically, SSH logins were consuming too many CPU cycles and were generating errors with valgrind, a utility for monitoring computer memory.

Through sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils. On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.

[…]

This code allowed someone in possession of a predetermined encryption key to log in to the backdoored system over SSH. From then on, that person would have the same level of control as any authorized administrator.

See also:

• Andres Freund (Andres Freund, Hacker News)
• Filippo Valsorda (Hacker News)
• Joey Hess (Hacker News)
• Devon Eriksen
• Evan Boehs
• Rhea Karty and Simon Henniger (Hacker News)
• xzbot (Hacker News)
• The Linux Backdoor Attempt of 2003 (via John Regehr)

HaxRob:

Plans to literally “hack the planet” foiled due to 500ms of latency that Andres instinctually investigated.

The latency was due how the malicious code parsed symbol tables in memory.

Amjad Masad:

You know that annoying person on your team who insists every bit of perf regression needs to be investigated. One day they will save the world.

Perry E. Metzger:

I’ve always thought that your CI and monitoring systems absolutely have to flag performance regressions and that you have to investigate them quickly, but I never thought of it as a security issue until now.

Rob Mensching:

Lots of analysis of the xz/liblzma vulnerability. Most skip over the first step of the attack:

0. The original maintainer burns out, and only the attacker offers to help (so the attacker inherits the trust of the project built by the maintainer).

Gynvael Coldwind (Hacker News):

Someone put a lot of effort for this to be pretty innocent looking and decently hidden. From binary test files used to store payload, to file carving, substitution ciphers, and an RC4 variant implemented in AWK all done with just standard command line tools. And all this in 3 stages of execution, and with an “extension” system to future-proof things and not have to change the binary test files again. I can’t help but wonder (as I’m sure is the rest of our security community) – if this was found by accident, how many things still remain undiscovered.

Rob Mensching (Hacker News):

This thread is a microcosm of the interactions in Open Source projects. Consumers make demands (some polite, some not-so-polite) of one maintainer (rarely two) that does everything.

Make no mistake. This is the way it works.

Glyph Lefkowitz:

For most maintainers, Tidelift pays a sub-hobbyist amount of money, and even setting it up (and GitHub Sponsors, etc) is a huge hassle. So even making the transition from “no income” to “a little bit of side-hustle income” may be prohibitively bureaucratic.

[…]

Specifically, every employer of software engineers should immediately institute the following benefits program: each software engineer should have a monthly discretionary budget of $50 to distribute to whatever open source dependency developers they want, in whatever way they see fit.

[…]

This sub-1% overhead to your staffing costs will massively de-risk the open source projects you use. By leaving the discretion up to your engineers, you will end up supporting those projects which are really struggling and which your executives won’t even hear about until they end up on the news.

thegrugq:

The xz backdoor was the final part of a campaign that spanned two years of operations. These operations were predominantly HUMINT style agent operations. There was an approach that lasted months before the Jia Tan persona was well positioned to be given a trusted role.

[…]

Every intelligence agency in the world could run this campaign, design and execute these operations. There is a serious level of technical acumen on display as well, the Jia Tan persona has to be able to do the work and talk the talk, but the core of this campaign is HUMINT.

[…]

It is important to remember that Lasse is blameless in this. There is no individual, and very very few organisations, able to detect, let alone resist!, the directed interest of an intelligence agency.

Lukasz Olejnik:

We should… maybe… resist the temptation of portraying XZ as alleged evidence of underfunded OS. Could it rather be THE evidence that resisting orchestrated and well-organised and funded campaign is hard?

Saagar Jha:

The problem is that nobody can read all this code. That’s it. You can make the code 50% clearer or reduce the number of libraries loaded or increase auditing but there is so many orders of magnitude more code being written than is properly reviewed that this can’t be fixed

It makes me so sad because I want this to be fixed and I want to go “oh if we paid maintainers some money the problem would go away” but, like, it just doesn’t seem to work. There is just so much code. We are drowning in it. The complexity of our stacks is insane

Juliano Rizzo:

Jia Tan’s git commit to turn off Landlock sandboxing one week after Lasse Collin improved it.

Note the extra period.

Simple Nomad (Hacker News):

This xz backdoor thing reminds me of a story I heard from friends that worked at a tech company that made cell phones. They had a great coder that worked on the project, he had put in work as a contractor for a few months, and due to the quality of his work he was hired in full time. After two months he simply stopped showing up to the office.

An investigation turned up the following interesting items. His account had accessed all files including source code to all cellular projects - in that he had apparently downloaded a copy of everything. He had committed a large amount of contributions to the project he was assigned to. None of his paychecks were ever cashed. A wellness check to the house he had rented was performed and the house was completely empty.

Isaiah Carew:

xz is an inflection point. people are going to lose their collective shit.

the idea that instead of breaking 4096-bit keys with gigawats of compute, or infiltrating hardened machines with 1337 haxors…

just shotgun backdoors into 1000 libraries that everyone uses.

Feross Aboukhadijeh:

The xz package backdoor is just the tip of the iceberg.

There’s a CONSTANT low-level stream of malware and spyware being uploaded to npm, PyPI, and Go registries.

I want to share a few examples from the 20,000+ malicious packages we detected so far[…]

Previously:

• On the Insecurity of Software Bloat
• The Lack of Compensation in Open Source Software Is Unsustainable
• Mac Software Updates Open Up sshd
• SolarWinds Breach

Update (2024-04-02): Russ Cox (via Hacker News):

This post is a detailed timeline that I have constructed of the social engineering aspect of the attack, which appears to date back to late 2021.

Mark Atwood:

The xz attack was not because it was open source. The attack failed because it was open source. The way this attack works for non-open source is the attacker spends 2 years getting an agent hired by contract software development vendor, they sneak it in, nobody finds out.

FFmpeg:

The xz fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion dollar corporations expect free and urgent support from volunteers.

@Microsoft @MicrosoftTeams posted on a bug tracker full of volunteers that their issue is “high priority”

Update (2024-04-03): Russ Cox (Hacker News):

At a high level, the attack is split in two pieces: a shell script and an object file. There is an injection of shell code during configure, which injects the shell code into make. The shell code during make adds the object file to the build. This post examines the shell script.

See also:

• Daroc Alden (via Hacker News)
• Rachel Kroll
• Bruce Schneier

Update (2024-04-08): blasty (via Hacker News):

the xz sshd backdoor rabbithole goes quite a bit deeper. I was just able to trigger some harder to reach functionality of the backdoor. there's still more to explore.

John Gruber:

Another is that it was very subtle: the ultimate goal was a back door in OpenSSH but the attacker(s) put their code in a compression library that was sometimes a dependency for another library that was itself only sometimes a dependency of OpenSSH.

See also: Jordan Rose.

Update (2024-04-11): Oxide Computer Company (via Adam Leventhal):

Andres Freund joined Bryan and Adam to talk about his discovery of the xz backdoor. It’s an incredible story… so great to get into the details with Andres. We started by ranting about the coverage in the New York Times… coverage that explicitly refused to dig into the details! It’s all the more shocking because the big story here is how Andres’ penchant for digging into the details is what saved us all from what would have been a pervasive and damaging attack!

Bruce Schneier:

It’s impossible to count how many of these single points of failure are in our computer systems. And there’s no way to know how many of the unpaid and unappreciated maintainers of critical software libraries are vulnerable to pressure. (Again, don’t blame them. Blame the industry that is happy to exploit their unpaid labor.) Or how many more have accidentally created exploitable vulnerabilities. How many other coercion attempts are ongoing? A dozen? A hundred? It seems impossible that the XZ Utils operation was a unique instance.

Solutions are hard. Banning open source won’t work; it’s precisely because XZ Utils is open source that an engineer discovered the problem in time. Banning software libraries won’t work, either; modern software can’t function without them.

Update (2024-04-24): Joab Jackson (via Hacker News):

At any rate, how the backdoor got so close to so many production systems may be a cautionary tale over the state of the internet infrastructure.

Kaspersky (via Hacker News):

Unlike other supply chain attacks we have seen in Node.js, PyPI, FDroid, and the Linux Kernel that mostly consisted of atomic malicious patches, fake packages and typosquatted package names, this incident was a multi-stage operation that almost succeeded in compromising SSH servers on a global scale.

Compression Exploit Linux Open Source Open-source Software Programming Security SSH

11 Comments RSS · Twitter · Mastodon

Leave a Comment

Name

E-mail (will not be published)

Web site

Δ