Archive for November 13, 2020

Friday, November 13, 2020

Apple Server Outage Makes Mac Apps Hang on Launch

Jeff Johnson:

WTF somehow my TCC seems fucked up on Mojave suddenly, for no apparent reason, no software updates. But only when my internet is connected?

Apps are hanging on launch! Reboot didn’t help.

Jonathan Deutsch:

I’m hitting the exact same thing on 10.15.7 starting ~30 min ago… lots of random hangs only when connected to wifi.

Skylar Lewis:

All of my non-Apple apps became really slow to open as well.


😅 Looks like, when apps are launched, Gatekeeper is unable to check their validity over the internet, due to overwhelmed Apple servers.

Jeff Johnson:

I figured out the problem using Little Snitch.

It’s trustd connecting to

Denying that connection fixes it, because OCSP is a soft failure.

(Disconnect internet also fixes.)

Make sure you deny it for both system and user. I ended up having to make 2 rules.

Patrick Wardle:

On Big Sur, trustd is in Apple’s “ContentFilterExclusionList”….meaning firewalls can’t block it! 😭

Welcome to the future? 😱

Jeff Johnson:

If you don’t have @littlesnitch then try /etc/hosts to fix Mac app launching port 80 is the problem

Nathan H. Leung shows how to do this with vi.

Jeff Johnson:

Don’t confuse Developer ID certificate status (/usr/libexec/trustd to with notarization (/usr/libexec/syspolicyd to

Notarization check only occurs on first launch. Online Certificate Status Protocol can occur on any launch.


I thought it was just Catalina being Catalina. I woke my computer from sleep and it couldn’t detect the fucking keyboard or trackpad.

Adam Engst:

It’s quite troubling that an Apple server being down could cause this. My iMac is sludge right now.

Guilherme Rambo, on the System Status page:

🔥 This is fine 🔥

Josh Centers:

It’s very simple: a screwed up server on the other end of the country shouldn’t render your computer unusable.

Łukasz Langa:

I am currently unable to work because macOS sends hashes of every opened executable to some server of theirs and when trustd and syspolicyd are unable to do so, the entire operating system grinds to a halt.

I’m typing this from my phone since the Mac is effectively frozen.

Nilay Patel:

I had three different Macs go sideways today because of a server issue I had no idea was happening. Many thoughts about how much we actually own our computers :(

Jeff Johnson:

Good news, Mac users! Our long international nightmare is over.

People are saying that is back online, and that seems to be true.

Yan Zhu:

don’t block forever because apple uses it to check for revoked notarizations

Jeffrey Paul (via David Heinemeier Hansson, Reddit):

It’s here. It happened. Did you notice?

I’m speaking, of course, of the world that Richard Stallman predicted in 1997. The one Cory Doctorow also warned us about.

On modern versions of macOS, you simply can’t power on your computer, launch a text editor or eBook reader, and write or read, without a log of your activity being transmitted and stored.

See also: Hacker News, 9to5Mac (Hacker News), ArsTechnica, MacRumors, The Verge, Philipp Defner, Nick Heer.


Update (2020-11-16): Jeff Benson (via Nick Heer).

This brings with it several privacy concerns. First, because your computer has to send your IP to communicate with Apple, it means Apple can see your IP address and the application you’re trying to use. Second, OCSP uses unencrypted HTTP communications so “any entity with visibility to your macOS-based computer could also observe and/or log these facts.”

Jeff Johnson (tweet, Hacker News):

When you launch a Mac app, macOS may check with Apple’s Developer ID OCSP to see whether the app developer’s code signing certificate is revoked. […] Unfortunately, if there’s an internet connection problem involving the Developer ID OCSP, that can also prevent Mac apps from launching.


This actually wasn’t the only Developer ID disaster recently. A few weeks ago I wrote another blog post after Apple temporarily revoked HP’s Developer ID cert, which caused a widespread failure of HP printer software.


The reason I mention the cache period is that it appears Apple has greatly increased it, from 5 minutes to half a day, likely in order to mitigate the problems caused by Thursday’s outage.


The notarization status is cached permanently and has no expiration, unlike OCSP. Thus, notarization only affects your ability to install new apps, it doesn’t affect your ability to launch already installed apps.

Dave Wood:

I would really like to see a response from @apple on this. They need to acknowledge the problem & what they’re doing to ensure it doesn’t happen again. Bonus points if they explain how they’re not tracking everything we do.

Jeff Johnson:

One bad side effect of blocking is that it can break the Mac App Store[…] because they’re running more than one service on that domain!

Howard Oakley:

We did have an alternative in macOS, which used to maintain a local database of revoked certificates, or so we suspect, until over a year ago. At the height of its use, that database was updated every couple of weeks. So if Apple revoked a certificate being used to sign malicious software, it could take another two weeks or more before that revocation had trickled down to all active Macs. One of the advantages of the newer OCSP approach is that your Mac can block software within minutes of Apple revoking its certificate, something we saw only too well with the recent accidental revocation of some old HP printer software.


There are fallbacks. If your Mac doesn’t have an internet connection at all, or the route to Apple’s OCSP service is blocked, your apps still open, with their certificates unchecked. It’s when that service isn’t inaccessible, but has failed, that the biggest problems arise. This is a well-known engineering problem, fail-safe design.

As Apple so devastatingly demonstrated last Thursday to millions of Mac users around the world, its design of the trustd signing certificate check doesn’t fail safe in those circumstances.

John Gruber:

Just an embarrassing bug for Apple on a high-profile launch day.

John Gruber:

Apple should publish information about this system in the excellent — but alas, not comprehensive — Apple Platform Security report[…]

Jacopo Jannone:

The problem is that Apple’s responder didn’t go down; it was reachable but became extremely slow, and this prevented the soft failure from triggering and giving up the check.


To make things worse, it is common for OCSP to use HTTP - I’m talking about good old plaintext HTTP on port 80, none of that HTTPS rubbish. There is usually a good reason for this, that becomes especially clear when the OCSP service is used for web browsers: preventing loops. If you used HTTPS for checking a certificate with OCSP then you would need to also check the certificate for the HTTPS connection using OCSP. That would imply opening another HTTPS connection and so on.

There’s got to be a way to do better than this for Gatekeeper given that Apple controls both ends of the connection.

It is clear that the trustd service on macOS doesn’t send out a hash of the apps you launch. […] macOS does actually send out some opaque information about the developer certificate of those apps, and that’s quite an important difference on a privacy perspective.

For privacy purposes, I think it’s a distinction without much difference. Rather than your Mac broadcasting that you launched a particular version of the Signal app, it broadcasts that you launched an app from Signal Messenger, LLC.

David Heinemeier Hansson:

I don’t see how this makes anything better? Sending a global unique hash of the developer certificate in the clear still allows both Apple to keep a log and anyone the power to snoop. This is fundamentally busted. Apple should send ban lists to the user.

Apple (Hacker News):

Gatekeeper performs online checks to verify if an app contains known malware and whether the developer’s signing certificate is revoked. We have never combined data from these checks with information about Apple users or their devices. We do not use data from these checks to learn what individual users are launching or running on their devices.

Notarization checks if the app contains known malware using an encrypted connection that is resilient to server failures.

These security checks have never included the user’s Apple ID or the identity of their device. To further protect privacy, we have stopped logging IP addresses associated with Developer ID certificate checks, and we will ensure that any collected IP addresses are removed from logs.

In addition, over the the next year we will introduce several changes to our security checks:

  • A new encrypted protocol for Developer ID certificate revocation checks
  • Strong protections against server failure
  • A new preference for users to opt out of these security protections

Nick Heer:

The prior version is available on the Internet Archive.

So they were logging the IPs. And they don’t deny using aggregate information about what users are launching, e.g. to get competitive data. In typical Apple fashion, the only acknowledgement that there was a problem is via a quote given to a third-party site (also: MacRumors, Hacker News):

What caused the OCSP server problem? Apple says it was due to a server-side misconfiguration that specifically interfered with macOS being able to cache OCSP responses for Developer ID. This configuration error, along with an unrelated content delivery network (CDN) misconfiguration, is what caused the slow performance for apps to launch.

The people who discovered and publicized the issue don’t get to break this news.

David Heinemeier Hansson:

This is a very welcome admission by Apple that the current system is deeply flawed, and the changes promised are solid improvements. But why does shit like this always have to be let out to back door with an obscure update to an Apple help site article?

It’s not clear whether the new preference will be for OCSP’s successor, notarization, or both.

Paul Haddad:

I know lots will make fun of “over the the next year” being fast, but I’m impressed that in just a few days Apple acknowledged a problem and promised a fix. That’s fast for them, its not an incident report, but its progress?

Howard Oakley:

What I attempt in this article is a coherent account of how macOS checks executable code before it’s loaded and run, in macOS 10.15 and 11.0.

Phil Vachon (via Hacker News):

Mayhem ensued, and after the issues were cleaned up, many questions remained about the implications of this failure. But first, let’s take a look at the mechanisms involved in authenticating an application package, at the most fundamental level.


Perhaps more transparency would help ease peoples’ concerns around misuse of their data. Having an auditable third-party run the OCSP responders for app certificate checks would assuage peoples’ concerns that Apple is misusing this data.

Update (2020-11-25): Adam Engst:

It’s hard to overstate the effect this problem had on the Mac world. Although Josh and I were able to get our iMacs working properly again reasonably quickly, the rest of our afternoon disappeared into trying to figure out what was happening. In the MacAdmins Slack, IT admins and consultants were doing the same, not just because of their personal Macs but also because they were being deluged with calls, email messages, and trouble tickets from their users and clients. Developers received bug reports demanding fixes, and the problem disrupted many online presentations, meetings, and conferences taking place during that time. A Hacker News thread about the problem garnered over 1150 comments, including some from Mac users who, like Josh, wasted significant time with troubleshooting, worried that their Macs had suffered a hardware failure.

Apple may not have actually taken every Mac in the world offline, but this network failure wasted several hours of time for what must have been millions of Mac users. (I suspect that people who weren’t attempting to launch apps during this time might not have noticed.) Nothing will give us that time back, but an acknowledgment and apology would be welcome.

This debacle also threw a spotlight on what seems like a weak point in macOS. It’s clear that Apple designed trustd to fail silently and gracefully when a Mac is offline, but why is there such a long timeout in the event of a network failure? Are there other components of macOS that make similar checks in everyday usage that could hurt the user experience in error conditions?

Update (2020-11-27): Howard Oakley (Hacker News):

Until 2018-19, it appears that macOS stored information about certificate revocations locally, in the ‘Gatekeeper’ database at /private/var/db/gkopaque.bundle, which at one time Apple updated every couple of weeks. But those Macs which have kept pace with the latest release of macOS stopped accessing that database in September 2019, with the release of macOS 10.15 Catalina. Apple hasn’t released an update to it since 26 August 2019, and anyone with a fresh installation of Big Sur will have a truly ancient version installed. As I pointed out here, that ‘Gatekeeper’ database is now disused.

Instead, Catalina and Big Sur now check all executable code on loading, and, when that code is signed with a developer certificate, perform an online check with Apple’s OCSP service, which has suddenly become so controversial.

Since the introduction of Gatekeeper in 2012, Apple has apparently revoked many compromised developer certificates. We see the tip of the iceberg of malicious software which is signed, detected by Apple, and quickly has its certificate revoked.


So Apple only seems to have been performing such extensive checks over the last 16, and no more than 23, months, although they have been applied to quarantined apps for around six years.

Update (2021-03-15): CryptoHack:

Overall, the incident this week was a good time to reflect on the trust model that has been promoted by organisations like Apple and Microsoft. Malware has grown in sophistication and most people aren’t in a position to judge whether it’s safe to run particular binaries. Code signing seems like a neat way to leverage cryptography to determine whether or not to trust applications, and to at least associate apps with known developers. And revocation is a necessary part of maintaining that trust.

However, by adding several mundane failure modes to the verification process, OCSP spoils any cryptographic elegance the code signing and verifying process has. While OCSP is also widely used for TLS certificates on the internet, the large number of PKI certificate authorities and relaxed attitude of browsers means that failures are less catastrophic. Moreover, people are accustomed to seeing websites become unavailable from time to time, but they don’t expect the same from apps on their own devices. macOS users were alarmed at how their apps could become collateral damage for an infrastructure issue at Apple. Yet this was an inevitable outcome arising from the fact that certificate verification depends on external infrastructure, and no infrastructure is 100% reliable.

Scott Helme also has concerns about the power that Certificate Authorities gain when certification revocation actually works effectively. Even if you aren’t bothered about the potential for censorship, there will be occasional mistakes and these must be weighed against the security benefits.

Update (2021-07-28): Howard Oakley:

In November 2020, Apple’s use of online OCSP checks came under fire, driving it to take immediate steps to protect privacy, and to state that certificate revocation checks will change in the following year to feature:

  • “a new encrypted protocol”;
  • “strong protections against” [OCSP] “server failure”;
  • “a new preference for users to opt out of these security protections”, which presumably means both hash lookup and certificate revocation checks.

As far as I’m aware, none of those three changes has yet been implemented, although there are only four months left before that year elapses.


I’m the reason they “came under fire”, and they had been transmitting the app launches unencrypted back to Apple for two years already at that point. It of course continues today. It looks like it will continue in 12.x all next year, too.

Update (2021-08-13): Howard Oakley:

With less than three months to go to the end of that year, I can’t discover any further announcements from Apple that anything has changed, and by the end of November last year the trail runs cold. Apple revised that support article on 30 March 2021, but doesn’t appear to have altered anything of substance concerning its OCSP checks.

Of Apple’s four promises, removal of IP addresses from the OCSP servers should have happened immediately, and there appear to have been no further server outages, making it plausible that the service is now more robust.

Apple has made no announcement regarding the more difficult problem of introducing an encrypted protocol to protect revocation checks.


I think it’s time for Apple to provide an update on its progress in implementing the changes which it so publicly announced on 16 November 2020.

Update (2021-11-12): Howard Oakley:

In the normal run of macOS updates, we wouldn’t expect Monterey 12.1 for a month yet. Although its current beta-release apparently brings SharePlay, slightly delayed from the initial release, there’s no mention of a key feature which Apple promised us almost a year ago: the option to disable signing certificate checks with Apple’s OCSP servers. While this may not be at the top of everyone’s priorities, for many Mac users around the world it’s essential protection from prying state security services, and not a promise that Apple can renege on.


It’s the fourth promise which should be most obvious. I can see no change in Monterey 12.0.1 which provides a means for users to opt out of OCSP revocation checks. Perhaps Apple intends to introduce this in 12.1, but there’s no mention of it in the release notes. It’s also an important issue for those still using Big Sur. Given that Apple’s promise isn’t confined to any future release of macOS, the new user preference should surely also be implemented retrospectively in Big Sur as well, perhaps in its forthcoming 11.6.2 update.

Given the tens of thousands of engineers employed by Apple, and the apparent simplicity of this task, has Apple forgotten the promises it spells out so clearly in that support article, or has it no intention of doing what it still says it will? Perhaps you’d like to ask Apple whether it’s ever going to honour those promises.

Waiting to Update to Big Sur

Howard Oakley:

Many experienced Mac users like to leave it a while before committing their main, production Mac to a new version of macOS. This article looks at some of the issues involved, with particular reference to Big Sur. If you still want to be an early adopter, then this article gives practical advice on what you should do to prepare for the upgrade.


As with Catalina, upgrading to Big Sur involves commitment. Should it prove a disaster, the road back isn’t quick or easy: you’d need to reformat your boot disk and install a fresh copy of the previous version of macOS. It’s also worth noting that, however alluring it might be that Big Sur can make Time Machine backups to APFS volumes, those are incompatible with previous versions of Time Machine, and converting old backups for use with Big Sur is also likely to be a one-way trip.

Dave Nanian:

It’s never a good idea to update to a just-released major OS version unless you have to. Nobody knows how reliable Big Sur is going to be for regular users. Let someone else find out before you take the jump.

On our end, SuperDuper! will not be compatible with Big Sur on day of release.

Rich Trouton:

Not yet ready for macOS Big Sur in your environment, but you’ve trained your folks to look at the Software Update preference pane to see if there’s available updates?


You can block it from appearing using the softwareupdate --ignore command, but for macOS Catalina, Mojave and High Sierra, that command now requires one of the following enrollments as a pre-requisite:

  • Apple Business Manager enrollment
  • Apple School Manager enrollment
  • Enrollment in a user-approved MDM


Update (2020-11-16): See also: Slashdot.

Dave Nanian:

As if it wasn’t bad enough, Big Sur’s Disk Utility makes it frustratingly hard to wipe a drive when people want to roll back from Big Sur to Catalina, Mojave, etc.

Do we really have to pay so little attention to these things? I know everything new is perfect, but really.

Native Instruments (via Hacker News):

Using a TRAKTOR KONTROL S4 MK3 on macOS 11 (Big Sur) can cause malfunction and potentially damage your controller! We are working together with Apple to find a solution to this problem.

The rare software problem that can cause a hardware problem.

Update (2020-11-17): macmule:

As forewarned in my prior post, here’s a post detailing methods to block tof macOS Big Sur.

In truth, the majority of this post will be rehashing items mentioned in previous post titled: Blocking macOS Catalina with Jamf Pro.

Update (2020-11-20): Adam Engst (forum):

Unfortunately, there’s no Apple-provided way to make that System Preferences badge go away, so it constantly reminds the user that an update is waiting. That’s problematic because it teaches users to ignore the badge, which could prevent them from installing a critical security update in the future. It’s also a visual distraction. The macOS interface shouldn’t be cluttered with information that the user has deemed unnecessary.


With macOS 11 Big Sur, Apple seems to have taken the upgrade nags a step further. In the Updates screen of the App Store app, most Mac users will be offered an update to GarageBand 10.4.1. However, if you haven’t yet upgraded to Big Sur, trying to update GarageBand will result in an admonishment that the update isn’t compatible with previous versions of macOS.


This is shoddy behavior on Apple’s part. That softwareupdate -ignore command should be given back to everyday users. The App Store app should reliably tell you when there are updates available for your Mac. Advertising an update that a Mac can’t install is at best unnecessary.

Update (2021-03-11): Adam Engst:

It’s a hard question to answer because everyone’s situation is different—I can’t know if you might rely on an app that doesn’t work perfectly in Big Sur. Worse, emotions often run high when it comes to macOS upgrades, with some people viewing “different” as “bad” on principle, and Big Sur’s visual redesign is quite different. So I won’t tell you that you should upgrade to Big Sur—if you choose not to, that’s entirely your prerogative. But I will say that I have upgraded with no real problems, and if you wish to upgrade, it’s generally safe to do so.


I will say that I think Big Sur has proven itself more solid than 10.15 Catalina. I never officially recommended an upgrade to Catalina because it never felt entirely baked, even after Apple announced Big Sur.


I don’t know if this is related to Big Sur, Apple’s update servers, or my Internet connection, but I’ve had trouble installing Big Sur updates. I must have tried to install macOS 11.2.3 at least 30 times between my 2020 iMac and M1-based MacBook Air, each attempt being met with a Download Failed dialog at varying points in the download process.

This has happened to me with nearly every Big Sur update and beta.

Update (2021-08-18): Jeff Johnson (tweet, Hacker News):

Ultimately, though, I couldn’t wait any longer, because I need to run the latest Xcode beta in order to develop Safari extensions for iOS, and the latest Xcode beta requires Big Sur, so I finally decided to update to Big Sur on Saturday. This was a disaster.


Some friends told me that they had experienced something similar before, but they were able to get the update to finish by forcing their Mac to power off, and then rebooting. So I did this, maybe 4 hours after I started the update. When I rebooted, the Mac showed the lock screen for FileVault, and I successfully unlocked it with my password. But then it immediately went back to the black install screen, with the progress bar once again stuck at exactly the same point as before!


A Mac with no admin account is not very usable. There are a bunch of things you can’t do, and many features didn’t work right. So the operating system was effectively hosed. At this point, I decided that there was nothing more I could do with the update. Yet going back to Mojave was not an option, because Xcode’s OS requirements still weighed on me. My only truly viable option was to erase the disk, do a “clean” install of Big Sur, and manually migrate my data from backup. This was tedious but seems to have been successful, and there was no getting stuck during the install.

Due to the annual release schedule, Monterey will be out before even the serious bugs have been fixed in Big Sur.


Update (2021-09-08): Howard Oakley:

As Apple prepares to release macOS 12 Monterey, if you’re still running Catalina, you might be tempted to remain one major release behind, and take this as the time to upgrade to Big Sur. If you do, I think you’re making a serious mistake: anyone ready to go beyond Mojave should now be preparing for Monterey, not Catalina or Big Sur. It’s a matter of balancing the costs and benefits.

Usually, running the last release of the last major version of macOS provides a relatively stable platform, disturbed only by the series of Security Updates, and spares you from wrestling with all the new bugs which come bundled with the new major version. I don’t think that’s the prospect in store for those using Big Sur over the coming year.

macOS 11.0 Big Sur Released

Apple (Hacker News, Slashdot):

macOS Big Sur, the latest version of the world’s most advanced desktop operating system, is now available to Mac users as a free software update. Big Sur introduces a beautiful redesign and is packed with new enhancements for key apps including Safari, Messages, and Maps, as well as new privacy features. And Big Sur has been engineered, down to its core, to take full advantage of all the power of the M1 chip to make the macOS experience even better for the new 13-inch MacBook Pro, MacBook Air, and Mac mini. The combination of Big Sur and M1 truly takes the Mac to a whole new level with incredible capabilities, efficiency, and more apps than ever before, while maintaining everything users love about macOS.

See also:


Evernote Goes Electron


We’re excited to share the all-new Evernote app for Mac. The new app provides a more modern appearance and streamlined Evernote experience.

Ilja A. Iwas:

If you have been relying on AppleScript to export your data from @evernote, you might be in for a surprise after the upgrade to version 10. Sidenote: Built with Electron Framework. 😭🤦‍♂️

Previously, you could import from Evernote to EagleFiler just by selecting the notes and pressing EagleFiler’s capture hotkey. But that feature relies on AppleScript, which was removed in the switch to Electron.

Evernote 10 also no longer supports local notebooks or exporting to standard formats like HTML. However, you can still export to ENEX format and use EagleFiler to convert the ENEX to RTF files. If the notes have embedded images, you’ll instead get RTFD files.