Archive for November 19, 2018

Monday, November 19, 2018 [Tweets] [Favorites]

APFS in 2018

Howard Oakley:

Before explaining how Mojave’s version 3 APFS handles Fusion Drives, let me remind you how they work in good old HFS+.

Jonas Plum:

We developed different approaches to identify and recover (deleted) files on an APFS file system and published a paper about the used methodologies. Additionally, we implemented the open source recovery tool afro which was released three months ago. By using afro, we evaluated and compared the different approaches amongst each other and identified the method that so far delivers the best results and compared it to photorec. This showed that AFRO outperforms photorec on the evaluated APFS dataset. In the presentations of this research we were often asked if other tools like Blackbags Blacklight do not already support this recovery process. So, we decided to compare the file recovery capabilities of BlackLight and afro. We wanted to compare afro to the sleuth kit as well, as at the DFRWS conference it was discussed about adding APFS Support to The Sleuthkit Framework, but no implementations are public yet.

Gregory Szorc (via Peter Steinberger):

I sample profiled all processes on the system when running the Mercurial test harness. Aggregate thread stacks revealed a common pattern: readdir() being in the stack.

[…]

While the source code for APFS is not available for me to confirm, the profiling results showing excessive time spent in lck_mtx_lock_grab_mutex() combined with the fact that execution time decreases when the parallel process count decreases leads me to the conclusion that APFS obtains a global kernel lock during read-only operations such as readdir(). In other words, APFS slows down when attempting to perform parallel read-only I/O.

[…]

It is apparent that macOS 10.14 Mojave has received performance work relative to macOS 10.13! Overall kernel CPU time when performing parallel directory walks has decreased substantially - to ~50% of original on some invocations! Stacks seem to reveal new code for lock acquisition, so this might indicate generic improvements to the kernel’s locking mechanism rather than APFS specific changes. Changes to file metadata caching could also be responsible for performance changes. Although it is difficult to tell without access to the APFS source code. Despite those improvements, APFS is still spending a lot of CPU time in the kernel. And the kernel CPU time is still comparatively very high compared to Linux/EXT4, even for single process operation.

Jonathan Levin (PDF, via Objective-See):

APFS has become the de facto file system for MacOS and iOS as for 10.13/10.3- but what do we really know about it? Apple has promised the spec would be released “later this year” … over two years ago!

Reversing the complex filesystem structures, container blocks, snapshots and trees is a lousy job, but someone had to do it. Jonathan will present the unofficial APFS specification as it appears in Volume II of the “*OS Internals” trilogy, and present a free tool for inspecting and traversing APFS partitions and disk images for MacOS, iOS - and Linux.

Previously: macOS 10.14 Mojave Released, Apple File System Reference.

Update (2018-12-04): Monkeybread Software:

This is a blog article to remind everyone writing application in C to no longer use FSCreateFileUnicode function because it is very slow on APFS in our tests.

Update (2019-02-07): Howard Oakley:

I turned then to the latest and much fuller APFS documentation, where there is no mention of Fast Directory Sizing at all.

[…]

Here it all gets more difficult, because macOS doesn’t appear to provide high level languages such as Swift or Objective-C with ready access to the j_inode_flags which can tell whether it is enabled, nor does it provide any single call to return the size of a folder which might take advantage of this new feature of APFS.

[…]

It’s time for FileManager to get a function to return a folder’s total size (and total number of items) making best use of APFS Fast Directory Sizing when it’s available.

Six iPad Gestures You Should Know

Matthew Cassinelli (via TJ Luoma):

If you’re trying to use the new iPad Pro as your full-time computer, you need to learn these gestures to move quickly and efficiently.

Plus, I cover how I use them alongside the Shortcuts app.

Update (2018-11-26): Jason Snell:

Yes, it’s hard to make interface features on touchscreens obvious. (I wonder a bit about why Apple isn’t more aggressive in using the built-in Tips app to teach people how to use features of its devices.) But it’s a learning opportunity, too! And with that, I present nine other simple iPhone features that maybe escaped your attention.

Category Theory for Programmers

Bartosz Milewski (via Igal Tabachnik):

Category Theory is one of the most abstract branches of mathematics. It is usually taught to graduate students after they have mastered several other branches of mathematics, like algebra, topology, and group theory. It might therefore come as a shock that the basic concepts of category theory can be explained in relatively simple terms to anybody with some experience in programming. That’s because, just like programming, category theory is about structure. Mathematicians discover structure in mathematical theories, programmers discover structure in computer programs. Well structured programs are easier to understand and maintain, and are less likely to contain bugs. Category theory provides the language to talk about structure, and learning it will make you a better programmer.

There is also a free PDF version.

See also: Monad: From Category Theory to Swift.

Hardened Runtime and Sandboxing

Jeff Johnson:

The relationship between the hardened runtime and sandboxing can be confusing to Mac developers, both because the hardened runtime is new and because it’s not well documented by Apple. I’ll attempt to explain the relationship here. App sandboxing was introduced in Mac OS X 10.7 Lion and eventually became a requirement for all Mac App Store apps, though developers can also choose to sandbox apps distributed outside the Mac App Store. The hardened runtime was introduced in macOS 10.14 Mojave and is currently optional for all apps, though it is required in order to notarize your app. Apple has announced that at some point in the future, all apps distributed outside the Mac App Store will need to be notarized, which means they will need to be "hardened" too. I suspect that Apple will eventually require Mac App Store apps to hardened as well. This may be surprising to developers, who associate sandboxing with the App Store and the hardened runtime with Developer ID, but the two technologies are independent of the distribution method and independent of each other, which means that a single app can be sandboxed and hardened.

[…]

Some protections of the hardened runtime such as debugging and Address Book are indeed enforced by SIP. However, it turns out that the Apple Events protection is not enforced by SIP but rather applies to hardened apps regardless of whether SIP is enabled.

Previously: AEDeterminePermissionToAutomateTarget Added, But AEpocalyse Still Looms.