Tuesday, June 27, 2017 [Tweets] [Favorites]

APFS Native Normalization

The iOS transition to APFS seems to have gone very smoothly except for some Unicode normalization issues. Apple never really explained to developers how they could make their code work properly, most were not aware that there were issues at all, and the necessary app modifications were difficult to develop and fully test. In my view, pushing this responsibility onto apps was a recipe for endless obscure bugs and poor performance.

At WWDC 2017, Apple essentially admitted that they had made a mistake and told us how they are going to fix it. There is a short-term fix and also a long-term fix that will require another file system conversion. This is not yet documented in the APFS Guide, but here’s a summary of the different cases:

9 Comments

What does “APFS now uses normalization-insensitive hashes” mean?
How can hashes work without normalization?

@Anonymous It means that the hash is calculated in such a way that two filenames that differ only in their Unicode normalization will get the same hash. In other words, it’s as if they are normalized and then hashed.

Vivek Verma

To add additional details

- APFS stores the filename as it is received. No transformation is applied. As long as the filename is encoded correctly in utf8, the filename is accepted. The ondisk format is also utf8 ( HFS expects to get utf8 and stores ondisk in utf16 )

To achieve normalization insensitivity, Instead of storing normalized filenames on disk, APFS stores the filenames as passed to it and stores a hash of the normalized AND case folded version of the filename ( for case insensitive APFS ) and normalized version of the filename (for case sensitive APFS) alongwith the filename.

Vivek Verma

The documentation referenced in this blog post has now been updated for macOS High Sierra and iOS 11.

https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html

@Vivek Thanks. Is there any documentation for the new Fast Directory Sizing feature? The guide mentions that it exists, but not what the APIs are.

Vivek Verma

> Is there any documentation for the new Fast Directory Sizing feature

Unfortunately not yet. A more documentation project is in the works but at this time it is hard to say when it will be available. Radars for issues which are affecting you will help.

[…] If you do anything with file names in iOS or macOS, make sure you read APFS Native Normalization […]

[…] directory sizing is also undocumented. The APFS guide doesn’t even say what the filename limit is, although I was able to get an […]

Stay up-to-date by subscribing to the Comments RSS Feed for this post.

Leave a Comment