Archive for August 29, 2019

Thursday, August 29, 2019

The Curious Case of the Core Data Crash

Sean Heber:

iOS routinely terminates apps for all sorts of reasons, including using too much memory or taking too long to do background processing. I don’t know why this information isn’t included in Xcode’s Organizer, but it’s a critical piece of a debugging puzzle.


But one result let to Tech Note 2151, with the code listed at the end under Other Exception Types:

The exception code 0xdead10cc indicates that an application has been terminated by the OS because it held on to a file lock or sqlite database lock during suspension.


I discovered that Twitterrific was sometimes closed while it was doing a network download and iOS left it running in the background long enough for the network request to finish. But not long enough for the database update to finish.

Sean Heber:

Converting to NSURLSession wasn’t very hard and we did it a long time ago. However something had been bothering me and I could never quite pin it down. Specifically, it really felt like network tasks that started while foregrounded and ended while backgrounded were less reliable.


I eventually determined that sometimes a request would fail to start and the error code returned by the NSURLSessionDataTask would be something odd like NSURLErrorNetworkConnectionLost or the ever-helpful NSURLErrorUnknown.


I scoured the NSURLSessions docs looking for a clue and came across something I had missed all the times before - a property called shouldUseExtendedBackgroundIdleMode “In addition to requesting that the connection be kept open … when the app moves to the background.”

Waitaminute! What is this? Does this mean there’s now an assumption that the connection will NOT be kept open when moving into the background?! Holy buckets on a wagon.

It appears that they REALLY want you to use the special background support in NSURLSession. In fact they seem to say to go ahead and use it all the time - don’t try to make it conditional.


And just like that, the flakiness seemed to be gone. For years I made the assumption that an open NSURLSession connection would be allowed to finish as long as you had a background task active like NSURLConnection did - but NOPE it doesn’t do that by default!

Spotlight Excludes Mail Folder on macOS 10.15


Spotlight search for emails neither works with MDQueryRef nor with mdfind in the even if the user has granted full disc access for the app in the security settings. The application logic of our app works for High Sierra bot not for Catalina any more. Is this a bug or a feature? Hopefully it’s not a feature. Other content like documents, calendar events and contacts can be retrieved by MDQueryRef.

If I search with the default Spotlight interface (command-space), I can find emails. But even if I select “Show in Finder” in the result list, the Finder window is empty.

This seems to be due to Mail using Core Spotlight. This newer API makes it possible to index items that don’t have a one-to-one correspondence with files on disk. However, though the Spotlight user interface integrates results from both the old file-based Spotlight and Core Spotlight, it seems that the APIs for apps to do their own global querying only work with the former. Mail has been shifting towards using Core Spotlight for several releases, but until now the actual files in ~/Library/Mail were still searchable. The message files are still there in Catalina, and there’s still a Spotlight importer plug-in that can read them, but you can’t actually search them.

And this extends to system apps that use those APIs:

We realized, that even Automator has a problem, if one use Spotlight in a workflow. Alternatively one can use Find Email Messages, but this takes minutes before Automator shows a result.


In prior versions of macOS, Spotlight searches allowed programatic access to mail messages. Mail metadata was readily available and well documented as properties on MDItem. The removal of this OS feature breaks what in essence was public API.


As a user I am worried to see more and more application data confined to closed “silos”. Previous versions of macOS / OS X have removed Safari bookmarks and history, Apple notes, etc. from indexing and thus from access by third party applications. This reduces the extensibility, scriptability, flexibility and thus usefulness of these core applications and ultimately the platform as a whole.


There’s an obvious conflict between the original Spotlight architecture (every app has access to everything) and the Core Spotlight architecture (every app is siloed), and it’s also obvious that the latter is better aligned with Apple’s ongoing privacy efforts.


I don’t think the siloing of Core Spotlight is part of Apple’s privacy effort or that it actually aligns with this. The privacy effort is focussed on user consent. Once consent is given the data should be readily available. This allows for application integration, automation, platform extension and avoids duplicated effort. To me it seems we are actually looking at an incomplete implementation. API to search Core Spotlight is missing. API to access mail messages is missing.

For example: once access to photos is granted, photos can be accessed via the file system, via scripting, via PHPhotoLibrary, and via MLMediaLibrary frameworks. All sorts of things become (resp. remain) possible. All hinges on user consent.

The current siloing and move to Core Spotlight has two problems:

  • Much information is no longer available as siloing proceeds faster than API evolution. E.g. there is no API to access not notes or email messages. This limits integration and automation opportunities. In some cases, third-party developers can resort to duplicating effort. E.g. by direct access to IMAP servers
  • Where public API to silos exist (PHPhotoLibrary, Contacts, …) the API lack the unifying nature of Spotlight / NSMetadataQuery. A public API to Core Spotlight should solve that.

Update (2019-11-27): Pierre Bernard:

Spotlight was the de-facto API for accessing Mail messages. It gave access to messages, their subject, sender and recipient names, as well as a wealth of other well-documented metadata. Spotlight also provided notifications when new mail was downloaded.

This allowed applications and scripts to work with mail without duplicating the effort of connecting to mail servers. Automation tools could set up actions to run upon receiving email messages. E.g. a mail to self to “turn on screen sharing”.


This may appear to be a cautious approach that favors security and privacy over application interoperability and productivity. In truth, the new situation is likely to undo privacy benefits provided by the “Full Disk Access” protection introduced with macOS Mojave.

Power users and third-party applications are likely to create their own search indexes. These additional copies of the private data contained in mail messages will not benefit from SIP / “Full Disk Access” protection.


Since there is no way for third-party applications to search Core Spotlight, no third-party can offer a full-featured alternative to the Spotlight window.

What Happened to PencilKit?

Geoff Pado:

The culprit here is lines 9–13; the ones that reference the PrivateFrameworks path. These lines are telling the linker to link against the private version of PencilKit… if the deployment target is less than iOS 13.0. Which Black Highlighter is. So we’re out of luck, right? No way to use PencilKit without dropping support for iOS 12? 😭

Geoff Pado:

Apple is now saying that your app’s deployment target has to be 13.1 to link PencilKit at all? Even if you cordon it off with #available checks? Will there be a public 13.0 at release? Will this mean that developers can’t support that release, either?


Update (2019-08-30): Steve Troughton-Smith:

New Xcode beta explicitly excludes PencilKit classes from the Catalyst SDK, so the app I’ve been working on all summer, which worked great up to the current betas, is no longer compilable or functional. In the last beta they removed input so I could render but not draw

The release notes have said since WWDC that PencilKit apps on macOS are ‘view-only’, but that was untrue up to the latest betas where they ripped out the (till-then, working) functionality and forcibly made it so. I don’t understand this decision at all

I think every app I’ve worked on this summer used PencilKit in some form, even on the desktop. Having that ripped out at the last minute, with no guarantee that drawing/input will be coming back to macOS this year at all, is preeetty frustrating

Missing Family Support

Michael Potuck:

As you’ve probably heard by now, Apple isn’t offering the option of getting an additional Apple Card for a second user. If you’re in a relationship, that means both you and your significant other have to fill out separate applications for Apple Card on iPhone or iPad which if approved, will be tied to your Apple ID.

And then, since there’s no Web interface, I guess you need to borrow their phone every time you want to update the shared family financial records.

Matt Birchler:

Apple Photos would also benefit from a lot of family-oriented updates. it’s ridiculous that I can easily access any app my wife has bought on the App Store but there’s no way for me to get a photo she took without her manually sharing it with me (via album or more often iMessage)

Goodbye, Safari Extensions Gallery

Apple (via Jeff Johnson):

As a reminder, the Safari Extensions Gallery will no longer be available in September 2019. Legacy Safari Extensions (.safariextz files) are not supported in Safari 13 on macOS Catalina, macOS Mojave, or macOS High Sierra. Users on macOS High Sierra or later can easily find extensions on the Mac App Store by choosing Safari Extensions from the Safari menu.

To make sure users can take advantage of your extensions on the latest versions of Safari on macOS High Sierra or later, we recommend that you convert your Legacy Safari Extensions (.safariextz files) built with Safari Extension Builder to the Safari App Extensions format and either submit them to the Mac App Store or notarize them for distribution outside the Mac App Store.