{"id":18242,"date":"2017-06-27T14:02:10","date_gmt":"2017-06-27T18:02:10","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=18242"},"modified":"2017-12-14T10:58:16","modified_gmt":"2017-12-14T15:58:16","slug":"apfs-native-normalization","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2017\/06\/27\/apfs-native-normalization\/","title":{"rendered":"APFS Native Normalization"},"content":{"rendered":"<p>The iOS transition to APFS seems to have gone <a href=\"https:\/\/twitter.com\/bob_burrough\/status\/878481851895042048\">very smoothly<\/a> except for some <a href=\"https:\/\/mjtsai.com\/blog\/2017\/03\/24\/apfss-bag-of-bytes-filenames\/\">Unicode normalization issues<\/a>. Apple never really explained to developers how they could make their code work properly, most were not aware that there were issues at all, and the necessary app modifications were difficult to develop and fully test. In my view, pushing this responsibility onto apps was a recipe for endless obscure bugs and poor performance.<\/p>\n<p>At <a href=\"https:\/\/developer.apple.com\/videos\/play\/wwdc2017\/715\/\">WWDC 2017<\/a>, Apple essentially admitted that they had made a mistake and told us how they are going to fix it. There is a short-term fix and also a long-term fix that will require another file system conversion. This is not yet documented in the <a href=\"https:\/\/developer.apple.com\/library\/content\/documentation\/FileManagement\/Conceptual\/APFS_Guide\/Introduction\/Introduction.html\">APFS Guide<\/a>, but here&rsquo;s a summary of the different cases:<\/p>\n<ul>\n<li><p>The default for macOS 10.13 will be <a href=\"https:\/\/mjtsai.com\/blog\/2017\/03\/31\/apfs-to-add-case-insensitive-variant-for-mac\/\">case-insensitive APFS<\/a>. It is normalization-preserving (unlike HFS+) but not normalization-sensitive. I expect this to be highly compatible with existing Mac apps. The main difference is that when you read filenames they are no longer necessarily in Form D, but you shouldn&rsquo;t have been relying on that, anyway.<\/p><\/li>\n<li><p>macOS 10.13 will also support case-sensitive APFS, which will use <em>native normalization<\/em>. This is new in the developer beta. The filenames are still stored in the same way as prior APFS (not normalized like with HFS+), but APFS now uses normalization-insensitive hashes so that it can quickly and transparently find files without knowing their normalizations. If your code worked with case-sensitive HFS+ and works with case-insensitive APFS, there&rsquo;s likely nothing new that you have to do for this case.<\/p><\/li>\n<li><p>iOS 10.3 through 10.3.2 use the problematic version of APFS that is case-sensitive, normalization-preserving, and normalization-sensitive. You can write a lot of <a href=\"https:\/\/mjtsai.com\/blog\/2017\/03\/24\/apfss-bag-of-bytes-filenames\/\">app code<\/a> to make everything work, but anyone who hasn&rsquo;t done this already probably won&rsquo;t.<\/p><\/li>\n<li><p>iOS 10.3.3 and iOS 11 will also be case-sensitive, normalization-preserving, and normalization-sensitive, but they will add <em>runtime normalization<\/em>. If you try to read a file but don&rsquo;t have the right normalization in your path, the file system APIs will transparently look for the file using other normalizations. This should give the correct behavior but at a performance cost.<\/p><\/li>\n<li><p>If you get a new device or erase and restore, iOS 11 will use case-sensitive APFS with native normalization. This is what Apple should have done from the start. It should have basically the same user experience as with HFS+ but with better performance.<\/p><\/li>\n<li><p>An unspecified future update will convert iOS devices using the &ldquo;bad&rdquo; APFS to case-sensitive with native normalization, thus completing the fix.<\/p><\/li>\n<\/ul>\n<p>Update (2017-12-14): Despite the native normalization, I&rsquo;m seeing problems with Git and accented filenames on macOS 10.13.2. If I edit a file with such a name, Git sees it as a new file, and therefore sees two files whose names differ only in normalization. It&rsquo;s somewhat <a href=\"https:\/\/stackoverflow.com\/questions\/23808611\/git-checkout-remove-file-with-special-characters\">tricky<\/a> to then remove the original entry.<\/p>","protected":false},"excerpt":{"rendered":"<p>The iOS transition to APFS seems to have gone very smoothly except for some Unicode normalization issues. Apple never really explained to developers how they could make their code work properly, most were not aware that there were issues at all, and the necessary app modifications were difficult to develop and fully test. In my [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"","apple_news_api_id":"","apple_news_api_modified_at":"","apple_news_api_revision":"","apple_news_api_share_url":"","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[1395,537,377,31,1380,1472,30,1529,71,258,1221],"class_list":["post-18242","post","type-post","status-publish","format-standard","hentry","category-technology","tag-apple-file-system-apfs","tag-filesystem","tag-git","tag-ios","tag-ios-10","tag-ios-11","tag-mac","tag-macos-10-13","tag-programming","tag-unicode","tag-wwdc"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/18242","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=18242"}],"version-history":[{"count":4,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/18242\/revisions"}],"predecessor-version":[{"id":19840,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/18242\/revisions\/19840"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=18242"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=18242"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=18242"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}