In March 2013, I chronicled the problems developers were having with Core Data iCloud syncing. Many concluded that the basic NSPersistentStoreUbiquitousContent design was flawed. Bugs could be fixed, but it would never be able to do what people wanted. Apple didn’t give the impression that it thought there were major problems. It kept reporting minor improvements and bug fixes. When CloudKit was announced in 2014, it had no official Core Data story. As far as I know, there is still isn’t one. Meanwhile, Drew McCormack built the Ensembles framework, which fixes just about all the problems with Apple’s design and works with both CloudKit and Dropbox.
Fast forward to WWDC 2016. When installing the Xcode 8 beta, I noticed that all of the symbols related to iCloud Core Data were marked as deprecated in macOS 10.12 and iOS 10, with the comment “Please see the release notes and Core Data documentation.” Strangely, the Core Data release notes and What’s New in macOS 10.12 documents make no mention of this. What’s New in iOS 10 simply says that “Several NSPersistentStoreCoordinator symbols related to ubiquitous content” have been deprecated.
I asked on Twitter what was up, and no one seemed to know. No one had anything nice to say about iCloud Core Data, either, though no doubt many developers still rely on it. I hoped that all would be explained in today’s WWDC Session 242: What’s New in Core Data. Alas, reports are that absolutely nothing was said about iCloud in the session. A key technology that underpins apps is (apparently) to be removed, and Apple didn’t tell anyone at its own developer conference. In my view, this is shameful. The issue is more acute than with a normal deprecation because the APIs rely on cloud services, which could conceivably stop working even for older OS versions where they are not deprecated.
It seems clear that the way forward is Ensembles, unless you want to write your own sync engine or wait for Apple to announce one. I hope that the deprecation will spur more of the community to coalesce around Ensembles.
On a more positive note, it looks like there are a lot of great developments in Core Data this year, including the integration of a mogenerator-like tool into Xcode and, apparently, the end of NSObjectInaccessibleException, which doesn’t play well with Swift.
Update (2016-06-18): Drew McCormack:
If you move to Ensembles now, you not only get the usual support, documentation, and source code, you also get it all at half the standard price. The sale will continue until macOS Sierra is released later this year.
Update (2016-06-19): See the comments on Hacker News.
Update (2016-06-22): Apple’s Ben Trumbull:
Apps will continue to work for the foreseeable future. Aside from the deprecation warnings, there are no changes to or removal of the functionality in iOS 10. We don’t have any specific time line to announce now, but traditionally deprecated symbols on our platform remain functional for a considerable period of time before removal.
Only the client Core Data iCloud API symbols are deprecated. Core Data with iCloud is built on top of the iCloud Drive service. The service pieces are not effected in any way and will continue to work. If and when the deprecated APIs are disabled in some future OS version, applications running on iOS 9 or 10 will continue to work.
However, App Store Review Guidelines say that apps cannot use deprecated technologies.
Update (2016-06-27): Daniel Pasco:
Last time I dove in to iCloud I lost a year of schedule and an evangelist eventually called me a masochist for continuing to try.
Core Data iCloud iOS iOS 9 Mac macOS 10.12 Sierra mogenerator Programming Swift Programming Language Syncing Top Posts Xcode
Apple File System Guide (Hacker News):
Apple File System is a new, modern file system for iOS, OS X, tvOS and watchOS. It is optimized for Flash/SSD storage and features strong encryption, copy-on-write metadata, space sharing, cloning for files and directories, snapshots, fast directory sizing, atomic safe-save primitives, and improved file system fundamentals.
[…]
A Developer Preview of Apple File System is included in OS X 10.12. The APFS on-disk volume format is pre-release and subject to change. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017.
Lee Hutchinson (Slashdot):
APFS also adds a copy-on-write metadata scheme that Apple calls “Crash Protection,” which aims to ensure that file system commits and writes to the file system journal stay in sync even if something happens during the write—like if the system loses power.
[…]
Snapshots and clones both are going to be available in APFS. Snapshots let you throw off a read-only instance of a file system at any given point in time; as the file system’s state diverges away from the snapshot, the changed blocks are saved as part of the snapshot. This is similar in concept to Microsoft’s shadow copies, and it’s an incredibly handy feature. There are obviously huge implications here in how Time Machine works—a true file system set of snapshots could totally replace the kludgy and aging mechanism of hard links that Time Machine builds and maintains.
Clones differ from snapshots in that clones are writable instead of read-only. According to the documentation, APFS can create file or directory clones—and like a proper next-generation file system, it does so instantly, rather than having to wait for data to be copied. A cloned file or directory stores the changes made between it and the original objects, giving you a writable, editable, point-in-time copy of a file or a directory. As the documentation points out, this is an easy way to create document revisions or do versioning of anything you might want to track.
Also interesting is the concept of “space sharing,” where multiple volumes can be created out of the same chunk of underlying physical space. This sounds on first glance a lot like enterprise-style thin provisioning, where you can do things like create four 1TB volumes on a single 1TB disk, and each volume grows as space is added to it. You can add physical storage to keep up with the volume’s growth without having to resize the logical volume.
Russ Bishop:
Although APFS does checksum metadata blocks it does not do anything to provide resilience for data blocks. That is a huge omission in a modern filesystem, a point I tried to politely but forcefully make in the File System Lab directly to a responsible engineer. I got the feeling that the APFS team is divided on the necessity of this feature and some people on the team would appreciate some ammo to help win the argument internally. I would encourage anyone who agrees to file radars ASAP requesting this feature.
[…]
I did get confirmation that Apple intends to provide an open specification in 2017 when the design is finalized. They also emphasized that the data structures are designed to be extended in backward and forward compatible ways so things like parity checks should be easily implemented even if they don’t make the first release.
Mark Dalrymple:
I love the sense of humor the low-level engineers have, as demonstrated by the -IHaveBeenWarnedThatAPFSIsPreReleaseAndThatIMayLoseData option.
Aymeric Barthe (via Benjamin Encz):
When it was first released, HFS+ did not support hard links, journaling, extended attributes, hot files, and online defragmentation. These features were gradually added with subsequent releases of Mac OS X. But they are basically hacked to death, which leads to a complicated, slow and not so reliable implementation.
[…]
Finally there is Bit Rot. Over time data stored on spinning hard disks or SSDs degrade and become incorrect. […] HFS+ lost a total of 28 files over the course of 6 years.
Adam Leventhal (via Hacker News):
By the next WWDC it seemed that Sun had been forgiven. ZFS was featured in the keynotes, it was on the developer disc handed out to attendees, and it was even mentioned on the Mac OS X Server website. Apple had been working on their port since 2006 and now it was functional enough to be put on full display. I took it for a spin myself; it was really real. The feature that everyone wanted (but most couldn’t say why) was coming!
[…]
I’m told by folks in Apple at the time that certain leads and managers preferred to build their own rather adopting external technology—even technology that was best of breed. They pitched their own project, an Apple project, that would bring modern filesystem technologies to Mac OS X. The design center for ZFS was servers, not laptops—and certainly not phones, tablets, and watches—his argument was likely that it would be better to start from scratch than adapt ZFS. Combined with the uncertainty above and, I’m told, no shortage of political savvy their arguments carried the day. Licensing FUD was thrown into the mix; even today folks at Apple see the ZFS license as nefarious and toxic in some way whereas the DTrace license works just fine for them. Note that both use the same license with the same grants and same restrictions. Maybe the technical arguments really were overwhelming (note however that ZFS was working internally on the iPhone), and maybe the risks really were insurmountable. I obviously have my own opinions, and think this was a great missed opportunity for the industry, but I never had the burden of weighing the totality of the facts and deciding. Nevertheless Apple put an end to its ZFS work; Apple’s from-scratch filesystem efforts were underway.
[…]
In the 7 years since ZFS development halted at Apple, they’ve worked on a variety of improvements in HFS and Core Storage, and hacked at at least two replacements for HFS that didn’t make it out the door. This week Apple announced their new filesystem, APFS, after 2 years in development. It’s not done; some features are still in development, and they’ve announced the ambitious goal of rolling it out to laptop, phone, watch, and tv within the next 18 months. At Sun we started ZFS in 2001. It shipped in 2005 and that was really the starting line, not the finish line.
Previously: ZFS, The Loss of ZFS.
See also: WWDC Session 701: Introducing Apple File System, ZFS data integrity explained, Trust, But Verify, Belt and Suspenders, Computational Skeuomorphism.
Update (2016-06-20): Adam Leventhal has written a series of posts based on his talks with Apple engineers Dominic Giampaolo and Eric Tamura at WWDC. Overview (Hacker News):
APFS, the Apple File System, was itself started in 2014 with Dominic as its lead engineer. It’s a stand-alone, from-scratch implementation (an earlier version of this post noted a dependency on Core Storage, but Dominic set me straight). I asked him about looking for inspiration in other modern file systems […] he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself.
Encryption, Snapshots, and Backup:
ZFS includes snapshots and serialization mechanisms that make it efficient to backup file systems or transfer file systems to a remote location. Will APFS work like that? Probably not[…]
[…]
Space sharing is more like an operational detail than a game changing feature. You can think of it like special folders with snapshot and encryption controls…
Space Efficiency and Clones:
Clones open the door for potential confusion. While copying a file may take up no space, so too deleting a file may free no space. Imagine trying to free space on your system, and needing to hunt down the last clone of a large file to actually get your space back.
Aaron Meurer:
Cloning fixes an important problem with hard links. The issue with hard links is that some applications will write “into” hard links, effectively changing both “copies” of the file, and other applications will “break” the link (copy on write).
[…]
It’s also worth pointing out that hard links typically share the same metadata (such as file permissions). APFS clones assumedly would be less strict about this (e.g., allow two clones of the same data have different file permissions).
Performance:
Apple controls the full stack including the SSD, FTL, and file system; they could have built something differentiated, optimizing this components to work together. What APFS does, however, is simply write in patterns known to be more easily handled by NAND.
[…]
APFS also focuses on latency; Apple’s number one goal is to avoid the beachball of doom. APFS addresses this with I/O QoS (quality of service) to prioritize accesses that are immediately visible to the user over background activity that doesn’t have the same time-constraints.
Data Integrity:
APFS removes the most common way of a user achieving local data redundancy: copying files. A copied file in APFS actually creates a lightweight clone with no duplicated data. Corruption of the underlying device would mean that both “copies” were damaged whereas with full copies localized data corruption would affect just one.
[…]
APFS checksums its own metadata but not user data. The justification for checksumming metadata is strong: there’s relatively not much of it (so the checksums don’t consume much storage) and losing metadata can cast a potentially huge shadow of data loss. If, for example, metadata for a top level directory is corrupted then potentially all data on the disk could be rendered inaccessible. ZFS duplicates metadata (and triple duplicates top-level metadata) for exactly this reason.
Explicitly not checksumming user data is a little more interesting. […] Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices, but if your software can’t detect errors then you have no idea how your devices really perform in the field. ZFS has found data corruption on multi-million dollar storage arrays; I would be surprised if it didn’t find errors coming from TLC (i.e. the cheapest) NAND chips in some of Apple’s devices.
This is disappointing. EagleFiler has detected lots of bit rot for me over the years, including on SSDs, and I’ve seen lots of unviewable old JPEG files.
Conclusions:
Those are great goals that will benefit all Apple users, and based on the WWDC demos APFS seems to be on track (though the macOS Sierra beta isn’t quite as far along).
Update (2016-06-27): See also: TidBITS, comments on ArsTechnica, comments on Hacker News, Accidental Tech Podcast.
Update (2016-07-03): See also: Accidental Tech Podcast.
Apple File System (APFS) Extended Attributes File System Hard Links History Mac macOS 10.12 Sierra Solid-State Drive (SSD) Sun Microsystems Time Machine ZFS
Git (via Hacker News):
If you’ve used Git, you’ve probably looked at a lot of diffs. Most of the time the diff shows exactly what you did: changed some lines, added some new ones, or took some away. But because Git generates the diff only from seeing the “before” and “after” states of your files, sometimes the way it shows the changes can be pretty confusing.
[…]
The author probably didn’t write the loop as two out-of-order halves. But because the outer lines of the old and new loops are the same, it’s equally valid to attribute the lines either way (it’s tempting to say that the problem can be solved by always starting the added lines at the top, but that only fixes this case; you’d have the opposite problem if the new loop were added after the old).
In 2.9, Git’s diff engine learned a new heuristic: it tries to keep hunk boundaries at blank lines, shifting the hunk “up” whenever the bottom of the hunk matches the bottom of the preceding context, until we hit a blank line.
It’s experimental, so you have to enable diff.compactionHeuristic with git config. Xcode 7.3.1 comes with Git 2.7.4, and the Xcode 8 beta comes with 2.8.1.
Git Version Control Xcode
Zennaware:
The removal of color from the user interface played a significant role in meeting these goals. Color is now exclusively reserved for the indication of status, both for user interface elements, where blue is used consistently to include that something is on/selected, and for the version control status of user content (e.g. file status, text differences).
[…]
Cornerstone 3 implements sandboxing in order that we can continue to offer Cornerstone to Mac App Store customers. […] The implementation of sandboxing in Cornerstone was extremely challenging. While many other developers have either given up or taken the opportunity to remove their apps from the Mac App Store citing the difficulty of implementing sandboxing, we have invested heavily into re-engineering significant parts of Cornerstone to not just work within the confines of sandboxing, but to embrace it.
[…]
External compare tools can no longer be run directly from Cornerstone. Instead, the compare tool plugin for your tool of choice must be installed to Cornerstone’s application script folder using the new Compare Tools window (see the Compare Tools… command in the Cornerstone menu).
Looks like a huge update. They also sell direct, which is the only way to get the upgrade discount.
Design Mac Mac App Mac App Store Sandboxing Subversion Version Control
Kirk McElhearn:
There were rumors that Connect was going to be deleted, but apparently Apple is going to try to make it work. It’s is a good idea for Connect to be in the For You section, instead of having its own tab.
[…]
Switch to Browse to see what’s new, and what’s not.
Again, there are text tabs in the window, rather than using a menu. New Music shows what’s new on Apple Music, Curated Playlists is what it says, then you see Top Charts, if you want to know what’s popular, and Genres takes you to a rather boring page of text links from which you can access a number of genres. The options within each genre haven’t changed, but the fonts have.
[…]
Here are a few screenshot of Apple Music on iOS; as you can see, they essentially reproduce the same elements as iTunes, just in a vertical layout. One problem I see is the size of the artwork for playlists and albums; it should be smaller, so you can see a bit more content.
Apple Music iOS iOS 10 iTunes Mac macOS 10.12 Sierra
