Revisiting ZFS for Mac
Hacker News is highlighting Adam Leventhal’s 2016 post (2016 comments) about Apple’s Leopard-era support for ZFS:
A few weeks before WWDC 2007 nerds like me started to lose their minds: Apple really was going to port ZFS to Mac OS X.
[…]
ZFS was to bring to Mac OS X data integrity, compression, checksums, redundancy, snapshots, etc, etc etc. But while energizing Mac/ZFS fans, Sun CEO, Jonathan Schwartz, had clumsily disrupted the momentum that ZFS had been gathering in Apple’s walled garden. Apple had been working on a port of ZFS to Mac OS X. They were planning on mentioning it at the upcoming WWDC. Jonathan, brought into the loop either out of courtesy or legal necessity, violated the cardinal rule of the Steve Jobs-era Apple. Only one person at Steve Job’s company announces new products: Steve Jobs. “In fact, this week you’ll see that Apple is announcing at their Worldwide Developer Conference that ZFS has become the file system in Mac OS 10,” mused Jonathan at a press event, apparently to bolster Sun’s own credibility.
Less than a week later, Apple spoke about ZFS only when it became clear that a port was indeed present in a developer version of Leopard albeit in a nascent form. Yes, ZFS would be there, sort of, but it would be read-only and no one should get their hopes up.
[…]
By the time Snow Leopard shipped only a careful examination of the Apple web site would turn up the odd reference to ZFS left unscrubbed. Whatever momentum ZFS had enjoyed within the Mac OS X product team was gone.
Here’s what I wrote in 2007. Revisiting some of the issues:
With Time Machine also being announced with Leopard, there was initial speculation that it was implemented using ZFS snapshots. Of course, that turned out not to be the case. Apple did eventually reimplement it using APFS snapshots.
ZFS was incompatible with HFS+ in that it didn’t support full-length Unicode filenames. It now supports filenames up to 1,023 bytes, which is more than APFS. So this is surely something Apple could have added if it wanted. (I’m not actually sure what the APFS limit is. Apple’s documentation doesn’t seem to say. Wikipedia says 255 UTF-8 characters, which I’m pretty sure is wrong because I think it supports at least 255 UTF-16 characters like HFS+. I was unable to set more using Finder. In any case, even 255 full Unicode characters encoded as UTF-8 is less than 1,023 bytes.)
ZFS was incompatible with Mac software in that it only supported bag-of-bytes filenames, not case-insensitivity or Unicode normalization. Apple was never fully consistent here, in that at the time it did support Mac OS X on UFS (which lacked both) and HFSX (which was case-sensitive). There was speculation that Apple could address this using higher level APIs. This is the path it took with the initial APFS release, which was no better than ZFS in this regard. By 2017, Apple had announced a new version of APFS that did support Unicode normalization.
I wrote that, unless Apple was already very far along in developing its own new file system, it would choose to get a head start by using a modified version of ZFS to address the above issues. This turned out to be totally wrong, as Apple instead scrubbed nearly all mentions of ZFS. APFS was not even available in developer beta until 2016, at which point it had been in development for only 2 years. It remains somewhat of a mystery what Apple was doing in the interim, aside from working on Core Storage.
ZFS had inefficient storage for extended attributes. I didn’t think this was a big deal at the time because Apple was barely using them. Since then, Apple has been using them a lot more, and APFS offers optimized storage for smaller xattrs.
It wasn’t until 2018 that Apple documented APFS, with the encryption documentation not coming until 2019.
Fast directory sizing was announced with the initial version of APFS but still hasn’t shipped in a form that works with Finder.
There are still very few tools available for working with APFS. This might have been better with ZFS.
I still wish that Apple had supported data integrity features like ZFS and think it’s bizarre that they suggested this was not needed due to their superior hardware. With APFS, I have continued to see file corruption, drives that mount but don’t work properly, and drives that suddenly don’t mount at all. In theory, ZFS could have offered more robustness or at least earlier detection.
Transparent compression would have been nice, too.
APFS’s awful performance on spinning hard drives continues to be an issue, with even small folders sometimes taking more than a minute to display in Finder. ZFS also uses copy-on-write, but it was designed before flash, so I wonder whether it would have worked better in this regard (e.g. keeping the metadata contiguous).
It’s possible that ZFS wouldn’t have worked with iOS due to its memory requirements. I’ve also read that this was solvable but don’t really know the details. And, of course, iPhones have a lot more RAM now.
Previously:
- Carbon Copy Cloner 7.1
- Error 702 Installing macOS on an External Drive
- What Happened to APFS Fast Directory Sizing?
- Time Machine Evolution and APFS
- APFS Enumeration Performance on Rotational Hard Drives
- Apple File System Reference
- APFS Native Normalization
- APFS to Add Case-Insensitive Variant for Mac
- APFS’s “Bag of Bytes” Filenames
- Testing Out Snapshots in APFS
- Apple File System (APFS)
- Apple Emerging Technology Job
- The Loss of ZFS
- ZFS
3 Comments RSS · Twitter · Mastodon
I use ZFS on some non-Macs. I trust it more than APFS because I think its data integrity (data checksums and self-healing) is a table-stakes feature.
APFS is a mobile-first file system. Users are expected to manage relatively little data and throw the rest in the cloud.
ZFS is a server-first file system. Users are expected to manage many terabytes across many drives and want redundancy.
Mac guys got the short end of the stick by not getting ZFS, but The iOS Company has their priorities.
ZFS's cache likes RAM. 1 GB + 1 GB per 1 TB is one rule of thumb, but this value can be tuned/restricted. FreeBSD (and derivatives like TrueNAS) let it ride -- ZFS will eagerly use all 'free' memory for caching, but release it as other processes request memory. I compare it to using NSCache or low-priority memory.
Maybe this played into Apple's decision-making? Apple still sold Macs with a pitiful 8 GB of RAM until this year and mobile has always had very little RAM.
IIRC the rumor at the time ZFS was canceled on macOS was that Apple wanted Sun to agree to an indemnity clause for licensing it and Sun (correctly) said no.
I view APFS as HFS++.
So sad and alarming that the two most widely used consumer operating systems on the planet use such weird hacky filesystems when there are better alternatives.
Seems money is the root cause of the issue in all cases. Apple identifying RAM as a place to save/make incredible profit margins, legal issues (money).
Microsoft continues to tinker with a more advanced filesystem, Apple flirted with ZFS. Apple made a really bold move foisting APFS on everyone with little documentation or warning. If they were going to make a bold filesystem choice, they should have gone bolder.
Agree completely with both above commenters. I wonder if Jobs' personal friendship with Ellison ever had anything to do with that.
