Archive for November 13, 2020

Friday, November 13, 2020 [Tweets] [Favorites]

Apple Server Outage Makes Mac Apps Hang on Launch

Jeff Johnson:

WTF somehow my TCC seems fucked up on Mojave suddenly, for no apparent reason, no software updates. But only when my internet is connected?

Apps are hanging on launch! Reboot didn’t help.

Jonathan Deutsch:

I’m hitting the exact same thing on 10.15.7 starting ~30 min ago… lots of random hangs only when connected to wifi.

Skylar Lewis:

All of my non-Apple apps became really slow to open as well.

Panic:

😅 Looks like, when apps are launched, Gatekeeper is unable to check their validity over the internet, due to overwhelmed Apple servers.

Jeff Johnson:

I figured out the problem using Little Snitch.

It’s trustd connecting to ocsp.apple.com

Denying that connection fixes it, because OCSP is a soft failure.

(Disconnect internet also fixes.)

Make sure you deny it for both system and user. I ended up having to make 2 rules.

Patrick Wardle:

On Big Sur, trustd is in Apple’s “ContentFilterExclusionList”….meaning firewalls can’t block it! 😭

Welcome to the future? 😱

Jeff Johnson:

If you don’t have @littlesnitch then try /etc/hosts to fix Mac app launching

ocsp.apple.com port 80 is the problem

Nathan H. Leung shows how to do this with vi.

Jeff Johnson:

Don’t confuse Developer ID certificate status (/usr/libexec/trustd to ocsp.apple.com) with notarization (/usr/libexec/syspolicyd to api.apple-cloudkit.com).

Notarization check only occurs on first launch. Online Certificate Status Protocol can occur on any launch.

nut_bunnies:

I thought it was just Catalina being Catalina. I woke my computer from sleep and it couldn’t detect the fucking keyboard or trackpad.

Adam Engst:

It’s quite troubling that an Apple server being down could cause this. My iMac is sludge right now.

Guilherme Rambo, on the System Status page:

🔥 This is fine 🔥

Josh Centers:

It’s very simple: a screwed up server on the other end of the country shouldn’t render your computer unusable.

Łukasz Langa:

I am currently unable to work because macOS sends hashes of every opened executable to some server of theirs and when trustd and syspolicyd are unable to do so, the entire operating system grinds to a halt.

I’m typing this from my phone since the Mac is effectively frozen.

Nilay Patel:

I had three different Macs go sideways today because of a server issue I had no idea was happening. Many thoughts about how much we actually own our computers :(

Jeff Johnson:

Good news, Mac users! Our long international nightmare is over.

People are saying that ocsp.apple.com is back online, and that seems to be true.

Yan Zhu:

don’t block ocsp.apple.com forever because apple uses it to check for revoked notarizations

Jeffrey Paul (via David Heinemeier Hansson, Reddit):

It’s here. It happened. Did you notice?

I’m speaking, of course, of the world that Richard Stallman predicted in 1997. The one Cory Doctorow also warned us about.

On modern versions of macOS, you simply can’t power on your computer, launch a text editor or eBook reader, and write or read, without a log of your activity being transmitted and stored.

See also: Hacker News, 9to5Mac (Hacker News), ArsTechnica, MacRumors, The Verge, Philipp Defner, Nick Heer.

Previously:

Update (2020-11-16): Jeff Benson (via Nick Heer).

This brings with it several privacy concerns. First, because your computer has to send your IP to communicate with Apple, it means Apple can see your IP address and the application you’re trying to use. Second, OCSP uses unencrypted HTTP communications so “any entity with visibility to your macOS-based computer could also observe and/or log these facts.”

Jeff Johnson (tweet, Hacker News):

When you launch a Mac app, macOS may check with Apple’s Developer ID OCSP to see whether the app developer’s code signing certificate is revoked. […] Unfortunately, if there’s an internet connection problem involving the Developer ID OCSP, that can also prevent Mac apps from launching.

[…]

This actually wasn’t the only Developer ID disaster recently. A few weeks ago I wrote another blog post after Apple temporarily revoked HP’s Developer ID cert, which caused a widespread failure of HP printer software.

[…]

The reason I mention the cache period is that it appears Apple has greatly increased it, from 5 minutes to half a day, likely in order to mitigate the problems caused by Thursday’s outage.

[…]

The notarization status is cached permanently and has no expiration, unlike OCSP. Thus, notarization only affects your ability to install new apps, it doesn’t affect your ability to launch already installed apps.

Dave Wood:

I would really like to see a response from @apple on this. They need to acknowledge the problem & what they’re doing to ensure it doesn’t happen again. Bonus points if they explain how they’re not tracking everything we do.

Jeff Johnson:

One bad side effect of blocking ocsp.apple.com is that it can break the Mac App Store[…] because they’re running more than one service on that domain!

Howard Oakley:

We did have an alternative in macOS, which used to maintain a local database of revoked certificates, or so we suspect, until over a year ago. At the height of its use, that database was updated every couple of weeks. So if Apple revoked a certificate being used to sign malicious software, it could take another two weeks or more before that revocation had trickled down to all active Macs. One of the advantages of the newer OCSP approach is that your Mac can block software within minutes of Apple revoking its certificate, something we saw only too well with the recent accidental revocation of some old HP printer software.

[…]

There are fallbacks. If your Mac doesn’t have an internet connection at all, or the route to Apple’s OCSP service is blocked, your apps still open, with their certificates unchecked. It’s when that service isn’t inaccessible, but has failed, that the biggest problems arise. This is a well-known engineering problem, fail-safe design.

As Apple so devastatingly demonstrated last Thursday to millions of Mac users around the world, its design of the trustd signing certificate check doesn’t fail safe in those circumstances.

John Gruber:

Just an embarrassing bug for Apple on a high-profile launch day.

John Gruber:

Apple should publish information about this system in the excellent — but alas, not comprehensive — Apple Platform Security report[…]

Jacopo Jannone:

The problem is that Apple’s responder didn’t go down; it was reachable but became extremely slow, and this prevented the soft failure from triggering and giving up the check.

[…]

To make things worse, it is common for OCSP to use HTTP - I’m talking about good old plaintext HTTP on port 80, none of that HTTPS rubbish. There is usually a good reason for this, that becomes especially clear when the OCSP service is used for web browsers: preventing loops. If you used HTTPS for checking a certificate with OCSP then you would need to also check the certificate for the HTTPS connection using OCSP. That would imply opening another HTTPS connection and so on.

There’s got to be a way to do better than this for Gatekeeper given that Apple controls both ends of the connection.

It is clear that the trustd service on macOS doesn’t send out a hash of the apps you launch. […] macOS does actually send out some opaque information about the developer certificate of those apps, and that’s quite an important difference on a privacy perspective.

For privacy purposes, I think it’s a distinction without much difference. Rather than your Mac broadcasting that you launched a particular version of the Signal app, it broadcasts that you launched an app from Signal Messenger, LLC.

David Heinemeier Hansson:

I don’t see how this makes anything better? Sending a global unique hash of the developer certificate in the clear still allows both Apple to keep a log and anyone the power to snoop. This is fundamentally busted. Apple should send ban lists to the user.

Apple (Hacker News):

Gatekeeper performs online checks to verify if an app contains known malware and whether the developer’s signing certificate is revoked. We have never combined data from these checks with information about Apple users or their devices. We do not use data from these checks to learn what individual users are launching or running on their devices.

Notarization checks if the app contains known malware using an encrypted connection that is resilient to server failures.

These security checks have never included the user’s Apple ID or the identity of their device. To further protect privacy, we have stopped logging IP addresses associated with Developer ID certificate checks, and we will ensure that any collected IP addresses are removed from logs.

In addition, over the the next year we will introduce several changes to our security checks:

  • A new encrypted protocol for Developer ID certificate revocation checks
  • Strong protections against server failure
  • A new preference for users to opt out of these security protections

Nick Heer:

The prior version is available on the Internet Archive.

So they were logging the IPs. And they don’t deny using aggregate information about what users are launching, e.g. to get competitive data. In typical Apple fashion, the only acknowledgement that there was a problem is via a quote given to a third-party site (also: MacRumors, Hacker News):

What caused the OCSP server problem? Apple says it was due to a server-side misconfiguration that specifically interfered with macOS being able to cache OCSP responses for Developer ID. This configuration error, along with an unrelated content delivery network (CDN) misconfiguration, is what caused the slow performance for apps to launch.

The people who discovered and publicized the issue don’t get to break this news.

David Heinemeier Hansson:

This is a very welcome admission by Apple that the current system is deeply flawed, and the changes promised are solid improvements. But why does shit like this always have to be let out to back door with an obscure update to an Apple help site article?

It’s not clear whether the new preference will be for OCSP’s successor, notarization, or both.

Paul Haddad:

I know lots will make fun of “over the the next year” being fast, but I’m impressed that in just a few days Apple acknowledged a problem and promised a fix. That’s fast for them, its not an incident report, but its progress?

Howard Oakley:

What I attempt in this article is a coherent account of how macOS checks executable code before it’s loaded and run, in macOS 10.15 and 11.0.

Phil Vachon (via Hacker News):

Mayhem ensued, and after the issues were cleaned up, many questions remained about the implications of this failure. But first, let’s take a look at the mechanisms involved in authenticating an application package, at the most fundamental level.

[…]

Perhaps more transparency would help ease peoples’ concerns around misuse of their data. Having an auditable third-party run the OCSP responders for app certificate checks would assuage peoples’ concerns that Apple is misusing this data.

Update (2020-11-25): Adam Engst:

It’s hard to overstate the effect this problem had on the Mac world. Although Josh and I were able to get our iMacs working properly again reasonably quickly, the rest of our afternoon disappeared into trying to figure out what was happening. In the MacAdmins Slack, IT admins and consultants were doing the same, not just because of their personal Macs but also because they were being deluged with calls, email messages, and trouble tickets from their users and clients. Developers received bug reports demanding fixes, and the problem disrupted many online presentations, meetings, and conferences taking place during that time. A Hacker News thread about the problem garnered over 1150 comments, including some from Mac users who, like Josh, wasted significant time with troubleshooting, worried that their Macs had suffered a hardware failure.

Apple may not have actually taken every Mac in the world offline, but this network failure wasted several hours of time for what must have been millions of Mac users. (I suspect that people who weren’t attempting to launch apps during this time might not have noticed.) Nothing will give us that time back, but an acknowledgment and apology would be welcome.

This debacle also threw a spotlight on what seems like a weak point in macOS. It’s clear that Apple designed trustd to fail silently and gracefully when a Mac is offline, but why is there such a long timeout in the event of a network failure? Are there other components of macOS that make similar checks in everyday usage that could hurt the user experience in error conditions?

Update (2020-11-27): Howard Oakley (Hacker News):

Until 2018-19, it appears that macOS stored information about certificate revocations locally, in the ‘Gatekeeper’ database at /private/var/db/gkopaque.bundle, which at one time Apple updated every couple of weeks. But those Macs which have kept pace with the latest release of macOS stopped accessing that database in September 2019, with the release of macOS 10.15 Catalina. Apple hasn’t released an update to it since 26 August 2019, and anyone with a fresh installation of Big Sur will have a truly ancient version installed. As I pointed out here, that ‘Gatekeeper’ database is now disused.

Instead, Catalina and Big Sur now check all executable code on loading, and, when that code is signed with a developer certificate, perform an online check with Apple’s OCSP service, which has suddenly become so controversial.

Since the introduction of Gatekeeper in 2012, Apple has apparently revoked many compromised developer certificates. We see the tip of the iceberg of malicious software which is signed, detected by Apple, and quickly has its certificate revoked.

[…]

So Apple only seems to have been performing such extensive checks over the last 16, and no more than 23, months, although they have been applied to quarantined apps for around six years.

Waiting to Update to Big Sur

Howard Oakley:

Many experienced Mac users like to leave it a while before committing their main, production Mac to a new version of macOS. This article looks at some of the issues involved, with particular reference to Big Sur. If you still want to be an early adopter, then this article gives practical advice on what you should do to prepare for the upgrade.

[…]

As with Catalina, upgrading to Big Sur involves commitment. Should it prove a disaster, the road back isn’t quick or easy: you’d need to reformat your boot disk and install a fresh copy of the previous version of macOS. It’s also worth noting that, however alluring it might be that Big Sur can make Time Machine backups to APFS volumes, those are incompatible with previous versions of Time Machine, and converting old backups for use with Big Sur is also likely to be a one-way trip.

Dave Nanian:

It’s never a good idea to update to a just-released major OS version unless you have to. Nobody knows how reliable Big Sur is going to be for regular users. Let someone else find out before you take the jump.

On our end, SuperDuper! will not be compatible with Big Sur on day of release.

Rich Trouton:

Not yet ready for macOS Big Sur in your environment, but you’ve trained your folks to look at the Software Update preference pane to see if there’s available updates?

[…]

You can block it from appearing using the softwareupdate --ignore command, but for macOS Catalina, Mojave and High Sierra, that command now requires one of the following enrollments as a pre-requisite:

  • Apple Business Manager enrollment
  • Apple School Manager enrollment
  • Enrollment in a user-approved MDM

Previously:

Update (2020-11-16): See also: Slashdot.

Dave Nanian:

As if it wasn’t bad enough, Big Sur’s Disk Utility makes it frustratingly hard to wipe a drive when people want to roll back from Big Sur to Catalina, Mojave, etc.

Do we really have to pay so little attention to these things? I know everything new is perfect, but really.

Native Instruments (via Hacker News):

Using a TRAKTOR KONTROL S4 MK3 on macOS 11 (Big Sur) can cause malfunction and potentially damage your controller! We are working together with Apple to find a solution to this problem.

The rare software problem that can cause a hardware problem.

Update (2020-11-17): macmule:

As forewarned in my prior post, here’s a post detailing methods to block tof macOS Big Sur.

In truth, the majority of this post will be rehashing items mentioned in previous post titled: Blocking macOS Catalina with Jamf Pro.

Update (2020-11-20): Adam Engst (forum):

Unfortunately, there’s no Apple-provided way to make that System Preferences badge go away, so it constantly reminds the user that an update is waiting. That’s problematic because it teaches users to ignore the badge, which could prevent them from installing a critical security update in the future. It’s also a visual distraction. The macOS interface shouldn’t be cluttered with information that the user has deemed unnecessary.

[…]

With macOS 11 Big Sur, Apple seems to have taken the upgrade nags a step further. In the Updates screen of the App Store app, most Mac users will be offered an update to GarageBand 10.4.1. However, if you haven’t yet upgraded to Big Sur, trying to update GarageBand will result in an admonishment that the update isn’t compatible with previous versions of macOS.

[…]

This is shoddy behavior on Apple’s part. That softwareupdate -ignore command should be given back to everyday users. The App Store app should reliably tell you when there are updates available for your Mac. Advertising an update that a Mac can’t install is at best unnecessary.

macOS 11.0 Big Sur Released

Apple (Hacker News, Slashdot):

macOS Big Sur, the latest version of the world’s most advanced desktop operating system, is now available to Mac users as a free software update. Big Sur introduces a beautiful redesign and is packed with new enhancements for key apps including Safari, Messages, and Maps, as well as new privacy features. And Big Sur has been engineered, down to its core, to take full advantage of all the power of the M1 chip to make the macOS experience even better for the new 13-inch MacBook Pro, MacBook Air, and Mac mini. The combination of Big Sur and M1 truly takes the Mac to a whole new level with incredible capabilities, efficiency, and more apps than ever before, while maintaining everything users love about macOS.

See also:

Previously:

Evernote Goes Electron

Evernote:

We’re excited to share the all-new Evernote app for Mac. The new app provides a more modern appearance and streamlined Evernote experience.

Ilja A. Iwas:

If you have been relying on AppleScript to export your data from @evernote, you might be in for a surprise after the upgrade to version 10. Sidenote: Built with Electron Framework. 😭🤦‍♂️

Previously, you could import from Evernote to EagleFiler just by selecting the notes and pressing EagleFiler’s capture hotkey. But that feature relies on AppleScript, which was removed in the switch to Electron.

Evernote 10 also no longer supports local notebooks or exporting to standard formats like HTML. However, you can still export to ENEX format and use EagleFiler to convert the ENEX to RTF files. If the notes have embedded images, you’ll instead get RTFD files.

Previously: