Friday, February 14, 2025

Asahi Linux Lead Resigns

Hector Martin (via Hacker News):

When Apple released the M1, I realized that making it run Linux was my dream project. The technical challenges were the same as my console homebrew projects of the past (in fact, much bigger), but this time, the platform was already open - there was no need for a jailbreak, and no drama and entitled users who want to pirate software to worry about.

[…]

Unfortunately, things became less fun after a while. First, there were the issues upstreaming code to the Linux kernel, which I’ve already spoken at length about and I won’t repeat here. Suffice it to say, being in a position to have to upstream code across practically every Linux subsystem, touching drivers of all categories as well as some common code, is an incredibly frustrating experience.

[…]

No matter how much we did, how many impossible feats we pulled off, people always wanted more. And more. Meanwhile, donations and pledges kept slowly decreasing, and have done so since the project launched. […] It seemed the more things we accomplished, the less support we had.

[…]

I consider Linus’ handling of the integration of Rust into Linux a major failure of leadership. Such a large project needs significant support from major stakeholders to survive, while his approach seems to have been to just wait and see. Meanwhile, multiple subsystem maintainers downstream of him have done their best to stonewall or hinder the project, issue unacceptable verbal abuse, and generally hurt morale, with no consequence. One major Rust for Linux maintainer already resigned a few months ago.

As you know, this is deeply personal to me, as we’ve made a bet on Rust for Linux for Asahi.

Previously:

Update (2025-02-16): Kevin Purdy:

Rust is a far more memory-safe coding language than the Linux kernel’s native C. But getting more than 1,700 maintainers, including branch bosses, to accept Rust code after decades of work in C is no small feat. Linux lead Linus Torvalds has shifted ever-so-slightly from a “wait and see” approach in 2021 to noting in the summer of 2024 that he expected Rust updates to be faster while admitting that it’s largely kernel developers’ familiarity with C standing in the way. At that time, Microsoft engineer Wedson Almeida Filho resigned from the Rust for Linux project, citing “nontechnical nonsense” as the motivation.

That conflict between the energy of Rust for Linux contributors and the strictures of kernel practices collided once more this winter in an early January kernel mailing list thread about a patch with the deceptively non-controversial name “Add dma coherent allocator abstraction.”

Kernel maintainer Christoph Hellwig opposed a patch that would have allowed drivers written in Rust to access the Direct Memory Access (DMA) API. “No rust code in kernel/dma, please,” Hellwig wrote. After some back-and-forth about suggested alternatives, Hellwig comes out with it: “Don’t force me to deal with your shiny language of the day. Maintaining multi-language projects is a pain I have no interest in dealing with.” A later post by Hellwig pushed further, and his attempt to clarify that it was “a cross-language codebase” he was comparing to “cancer,” not just Rust, did not likely soften its impact.

Update (2025-02-25): Greg Kroah-Hartman (via Hacker News, Michael Larabel):

The majority of bugs (quantity, not quality/severity) we have are due to the stupid little corner cases in C that are totally gone in Rust. Things like simple overwrites of memory (not that rust can catch all of these by far), error path cleanups, forgetting to check error values, and use-after-free mistakes. That’s why I’m wanting to see Rust get into the kernel, these types of issues just go away, allowing developers and maintainers more time to focus on the REAL bugs that happen (i.e. logic issues, race conditions, etc.)

I’m all for moving our C codebase toward making these types of problems impossible to hit, the work that Kees and Gustavo and others are doing here is wonderful and totally needed, we have 30 million lines of C code that isn’t going anywhere any year soon. That’s a worthy effort and is not going to stop and should not stop no matter what.

But for new code / drivers, writing them in rust where these types of bugs just can’t happen (or happen much much less) is a win for all of us, why wouldn’t we do this?

Kevin Purdy:

Torvalds’ response from Thursday does offer some clarification on Rust bindings in the kernel, but also on what die-hard C coders can and cannot control.

Maintainers like Hellwig who do not want to integrate Rust do not have to. But they also cannot dictate the language or manner of code that touches their area of control but does not alter it. The pull request Hellwig objected to “DID NOT TOUCH THE DMA LAYER AT ALL,” Torvalds writes (all-caps emphasis his), and was “literally just another user of it, in a completely separate subdirectory.”

“Honestly, what you have been doing is basically saying ‘as a DMA maintainer I control what the DMA code is used for.’ And that is not how *any* of this works,” Torvalds writes.

[…]

The leader goes on to state that maintainers who want to be involved in Rust can be, and can influence what Rust bindings look like. Those who “are taking the ‘I don’t want to deal with Rust’ option,” Torvalds writes, can do so—later describing it as a “wall of protection”—but also have no say on Rust code that builds on their C interfaces.

Update (2025-03-19): Michael Larabel (via Hacker News):

Asahi Lina announced today that she is pausing work on all of the Apple GPU driver development she had been pursuing for Asahi Linux with the open-source DRM kernel driver as well as Mesa contributions.

[…]

Asahi Lina had been leading the effort working on a Rust-written Apple DRM kernel graphics driver that has yet to be mainlined to the Linux kernel. Plus contributions to the Asahi AGX Gallium3D and Honeykrisp Vulkan drivers within Mesa too. As part of that, Asahi Lina was also involved in reverse-engineering the Apple M1/M2 GPU.

22 Comments RSS · Twitter · Mastodon


"As you know, this is deeply personal to me, as we’ve made a bet on Rust for Linux for Asahi."

And the major players who have to safeguard the critical operating system that runs most of the world decided to be conservative, and keep *world critical infrastructure* on a conservative strategy of keeping everything (as I understand it) in C.

They don't want Rust in the kernel yet, cry us a river. What's the goal of Asahi; Linux on Apple Silicon, or Rust in Linux, because it really sounds like giving up on the former, if he can't have the latter.

I've seen this "I'm only going to do this if it can be in Rust" thing a few times. It seems a very culty language.


Matt Leipham Ellis

As a non-developer, I have a hard time knowing how seriously to take those claims of a language unlocking an otherwise unattainable outcome. The author mentions the GPU driver in that light, dependent on Rust.


Unfortunately this is Linus' fault. He should have said he would not allow Rust into the kernel not because it is a bad language, but because most of his maintainers do not know 2 languages and are unwilling to learn them. This is not a technical reason but a social argument. Rust probably does simplify large driver development, and probably reduces the number of bugs the driver may have, but that simply doesn't matter.

I've actually been involved in another project that was supposed to be integrated into the kernel. Linus essentially killed it and I gave up on kernel development... and eventually moved to the Mac. His decision threw away a number of years of work and no Rust was involved.

So why is he going along with Rust, albeit halfheartedly? There was a point where Linus was "reeducated" with regard to codes of conduct. The simplest explanation that I can see is that he was told he could be removed from his job or he could do what he was told to do. My guess is that something similar is occurring here: some large important institutions want to move to Rust, so he cannot just ban it.

Linux has become so large that one cannot easily keep up with a fork. But the difficulty people now face to get their changes integrated is such a pain that I think only employees of large corporations will bother working on it. One can't expect corporate employees to spark innovation since they are measured on whether they deliver features some idiot upstairs decided were important. Therefore the most likely source of innovation (hackers doing stuff for fun) will be removed.

My conclusion is that unfortunately Linux is now a large cumbersome enterprise thing, and opensource doesn't magically solve the problems inherent with that.

As to Asahi, it's a shame, but it was a Quixotic battle anyway requiring them to reverse engineer hardware without documentation as well as building complex drivers. It's not surprising they tried to simplify the problem by using Rust. Moreover, as time goes on, it seems to me that people are only getting more and more entitled, not appreciating that someone made something neat, but demanding support or more features. Again, as a hacker, why bother releasing your stuff for that reward? The only remaining reason is to build a portfolio for CV purposes, but starting your own project can do that too.


It's too bad for Linux on AS, but really, Martin has a complex. There are obvious and important reasons for the Rust transition in Linux to be handled carefully, and Martin is just being entitled and stupid in response to being told to pipe down his enthusiasm. And yes, Rust is a very culty language—it attracts a certain kind of narcissist, just as other languages do. I wish people could just fucking get on.

And no, Apple Silicon is *not* an open platform. Don't be silly! Can run arbitrary code != open. Never was, never will be. Apple should simply support Linux directly to make their machines even more attractive to devs. Windows too, in order to give Windows apologists fits.


"They don't want Rust in the kernel yet"

Wno is "they"? It's mainly one person who is blocking this for no good reasons that he is able to articulate. And this isn't even about "Rust in the kernel", it's about one bit of Rust code that allows Rust drivers to integrate with the kernel.


"They don't want Rust in the kernel yet"

Which is not true. Linus has accepted Rust as an experiment in the kernel. There is even an official documentation page about it:

https://docs.kernel.org/rust/index.html

Note that the page says: "Currently, the Rust support is primarily intended for kernel developers and maintainers interested in the Rust support, so that they can start working on abstractions and drivers, as well as helping the development of infrastructure and tools."

An DMA abstraction (that is separate from the C code) seems well within the scope of 'abstractions and drivers'. And adding the abstraction would not make the kernel dependent on Rust, you can compile the kernel without Rust with a config option.

Following the discussion, it seems that it's mostly old hands only believe in C that are sabotaging the experiment by stonewalling Rust contributions.

Sure, there are also Rust zealots. However, the Asahi GPU driver in Rust has been a great success and written in a impressively short time frame. The developers have also shown convincingly the value-add of writing drivers in Rust. This has also led to new initiatives, like a new driver in Rust for NVIDIA GPUs by a Red Hat engineer.

The trajectory is clearly that the Rust for Linux project has booked successes. Some people are not happy about that.


"I have a hard time knowing how seriously to take those claims of a language unlocking an otherwise unattainable outcome."

It should be clear that it's not otherwise unattainable, Linux drivers have been written in C for well over three decades after all. It's just that, besides memory safety, Rust has better tools for strongly typing things like states, state transitions, etc. (e.g. through sum types).

The other aspect of it is that C is unlikely to generate enthusiasm from newer generations (for good reasons). If the kernel wants to appeal to new generations, as old hands retire, it needs to move with the times. It does not have to be Rust, but it is realistically the only alternative that is modern, fit for low-level code, has been stable for long enough (Rust 1.0 was released a decade ago), and has wide industry support.


The problem with supporting 2 languages is that the rule for kernel developers is: if you make a change, you fix everything that it breaks. This ensures less friction there's no coordination problems where one waits on someone else to fix the breakage (which is what the Rust people propose doing instead).

The result is that if you change C code, you have to fix the Rust code it breaks. If you don't know Rust and don't have time to learn it (because you already have too much to do maintaining what's there), you reject this solution.

To be snarky, if "newer generations" want to play with Rust... they could go write their own kernel. Like, say, Redox OS. After all, that's why the "older generations" even have a kernel to play with -- they wrote it themselves. But ultimately, I don't think this has anything to do with generational warfare.


Most of the old guard seem to either be fine with R4L, or don't really give a shit. the R4L people have already agreed to be responsible for unbreaking things in their own code. No C dev will be forced to fix any Rust code.

The issue seems to be that at least one person doesn't believe them, and thinks that they won't be around to maintain their own code. I guess this is plausible, but a more plausible outcome is that there will be fewer and fewer C devs and more and more Rust devs, and the opposite will happen. It's already difficult to find new Linux maintainers, particularly ones that aren't paid by Microsoft to capture Linux. This will only get worse, because who wants to voluntarily write C code in their spare time in [current year argument].


> because who wants to voluntarily write C code in their spare time in [current year argument].

*sheepishly hides my current hobby project of writing a classic Macintosh game in C89*


I like C. And I like many other languages. They are just different mindsets. So I don't find the argument "who wants to voluntarily code C in their spare time" particularly convincing.

C is a good choice for a portable project since there are many C compilers. I expect C will last longer than Rust. (See Lindy Principle).


Well, at least Hector and Filho will not be unbreaking any Rust code that a C developer breaks. So there is no reason for the maintainers to accept anything that they will be responsible for fixing, being the individuals where bucks stop.


"sheepishly hides my current hobby project of writing a classic Macintosh game in C89"

Okay, that's actually cool.

"They are just different mindsets"

They're different languages with different features. If you have the choice of not using C today, imo you should not use C, because you're intentionally depriving yourself of things that would make both your life and your application better.

"C is a good choice for a portable project"

Did C actually standardize the size of types across platforms now?

What you say used to be true in the 70s and 80s and probably 90s, but in my opinion, at this point in time, C is one of the worst languages you can choose for portable projects. Sure, if you are really good at C, you can make your code portable, but you could also use a language where you don't have to think about this in the first place.

"I expect C will last longer than Rust."

Neither of these two languages will ever disappear, for all reasonable values of "ever."

"there is no reason for the maintainers..."

I'm doing my absolute best to cause the problem I predicted would happen, and still failing because there are still people who want to maintain the Rust code, therefore I'm right.


"Karol Herbst announced this morning he's stepping down from his role as one of the upstream Nouveau kernel driver maintainers."

https://www.phoronix.com/news/Karol-Herbst-Nouveau-No


"Did C actually standardize the size of types across platforms now?"

Oh! It's not perfect! Shan't use it then!

Having programmed at all levels of the stack (firmware, drivers, OS, apps, ML) and on about 10 different architectures... I have good reason to claim that C was available for all of them. Rust on the other hand doesn't. https://doc.rust-lang.org/nightly/rustc/platform-support.html Now, you might not care about that, but that doesn't make your argument stronger.

As to the longevity of Rust, we'll see. C has 53 years to its credit. Rust has 13.

I quite like Rust. I don't like its fanboys and fangirls. They are too cocksure of themselves.


"I don't like its fanboys and fangirls"

Oh! Don't like some random people that have no impact on my life! Shan't use Rust then!

"As to the longevity of Rust, we'll see"

Presumably, neither of the two of us actually will.


@Daniël said "Rust [...] has been stable for long enough (Rust 1.0 was released a decade ago)" [yes, I'm eliding a lot of text here].

This is not true. On my FreeBSD server, I somehow got Cargo-C (and thus Rust) added as a dependency for a port. This meant that every time I tried to update some ports (including, importantly, PHP), I had to wait for ~ 3 hours as _every_ _single_ _time_, the Rust version had changed, and thus all of Rust had to be recompiled. I tried locking it, but then Cargo-C would complain that it needed Rust 1.40.2, and I only had Rust 1.40.1 (the numbers are pulled out of a had, but it was single digit differences in the third section).

(I finally managed to excise whatever it was that relying on Rust, but it took me a lot of trial and error because whatever it was, it didn't actually declare that it depended on Cargo-C; that's the ports fault though...)

1.0 may have been released, but if every few weeks you need to recompile the entire toolchain because of a minor point update, that's not stable.



I think the sentence that covers my feeling about Rust groupies is that "if you have a hammer, everything is a nail". An unwillingness to see other points of view.

The null hypothesis is still to expect C to outlive Rust. However another reason might be that if AI were to take over programming, something with more compiler level error checking than Rust would do better... like Idris or Haskell.


"if you have a hammer, everything is a nail"

This is such a confusing thing to say, because what actually happened is the exact opposite.

Until Rust arrived, people had no hammer. They wanted to build houses, and they had to use nails, but the best they could do was hammer them in with a bunch of bricks they had lying around. They hammered them in crooked and hit their fingers, but it did work, they were building a house.

At some point, somebody figured this was kind of stupid, and a bunch of people invested time to invent a proper hammer, so the house would not have crooked nails and people would not have bloody fingers, and by golly, they did it, they invented a fricken hammer.

And most people were happy. Some didn't want to learn to use the hammer, which was fine. Some wanted to use these new-fangled hammers, and that was also fine, their nails were straight, and they still sometimes hit their fingers, but much less often. But there was one person who was loudly yelling about how he did not want anyone using hammers on his house, because his house was built by hammering in nails using bricks, and if he saw anyone using a hammer, his day would be ruined.

Fortunately, the house owner stepped in and told him to shut the hell up. I can't link to that, because my post will get caught in the spam filter, but Linus shutting this stupid discussion down is pretty easy to find.


"People didn't have hammers before Rust arrived"

You must be trolling... or be very wet behind the ears. You might want to look up what the SeL4 kernel is written in.


It's written in bricks.

Leave a Comment