Apple Remote-Kills Long-time Developer’s Apps
William Gallagher (also: Charlie Monroe):
As Apple continues to face controversy over its App Store policies and fees, software developer Charlie Monroe has told AppleInsider that the company has killed all his apps with no warning. Each of his ten macOS apps, and two that are also iOS, remain available to buy in the App Store, but Apple has stopped them launching.
[…]
“Looking into it, I found that Apple revoked my distribution certificates, which generally kills the apps remotely.”
“When I sign in to my developer account, it asks me to enroll to the Apple developer program and I don’t seem to be in the Apple developer program anymore,” he continued, “even though the apps that I have on the App Store are still available.”
Every Apple platform developer’s worst nightmare. It’s bad enough that a seemingly innocuous developer has been effectively banned from development, his apps rendered non-functional, but ... no explanation? That is just cold.
In the morning no one got back to me. They did now, but only said on the phone they have no idea what’s wrong and are passing the issue to internal team... 🤔
macOS displays a message that the app “will damage your computer” just because the certificate was revoked, which IMHO is bordering with slander. Damages your name and brand. Aside from users unable to use your apps, of course.
The wording for the dialog and intent behind signed code is to protect from malware.
If this action isn’t based on that, Apple is the one that’s damaging their name and brand.
And if it is? At least give the developer a chance to rectify the situation.
Can’t run the long-installed app any more. Can’t open the downloaded installer, not even with right-click + option key, even on High Sierra.
This means for us Mac devs that Apple not only has the power to make it near-impossible (at least for the layman) to run your publically available app, but they actually assume the right to do so as they please. They’re judge, jury and executioner. Doesn’t that scare you?
Apple’s dev account was originally meant to be necessary only to sign your app, to ensure it can be checked against malicious modification. But now, it’s become the stick by which Apple alone controls which apps can run on a Mac.
I wish I could be a fly on the wall when decisions like this get made. Was this a snap judgement made in response to some automated alert, or an executive decision? Did someone suggest contacting the developer but get overruled? Or did nobody even consider it? So many questions.
Previously:
- New App Store Review Processes
- Guilherme Rambo Locked Out of Apple Developer Account
- The True and False Security Benefits of Mac App Notarization
- HandBrake Proton Trojan
- Apple Removed Dash From the Mac App Store
- KeRanger Transmission Ransomware
Update (2020-08-05): See also: Hacker News.
THIS is what sucks about Apple’s iron gatekeeper approach. One of my favorite apps suddenly fails to even launch, via a “Binary is improperly signed” error, apparently because Apple pulled the developer’s account, and apparently without a word of explanation. EXPLAIN, Apple.
Why was there no human review or due process?
After more investigation, I found out that the distribution certificates were revoked – evidently by Apple as no one else has access to them and I was sound asleep when all this happened. Each macOS app these days needs to be codesigned using an Apple-issued certificate so that the app will flawlessly work on all computers. When Apple revokes the certificate, it’s generally a remove kill-switch for the apps.
[…]
This is the message macOS shows to all users who try to launch my app. That it will damage their computer with a checkbox to report malware enabled. Average user immediately goes nuts.
[…]
Fortunately, possibly thanks to the traction the story got and all the support from everyone I got (for which I am infinitely grateful), after almost 24 hours after 10PM, I got my account re-instated.
Apple has called and apologized for the complications. The issue was caused by my account being erroneously flagged by automated processes as malicious and was put on hold.
I want to believe you, and I do believe you, but @Apple absolutely needs to say publicly and explicitly that they were wrong when they said it would damage the computer.
Apple really needs to provide emergency telephone contact details to people whose accounts are put into this state. Ideally it’d proactively get in touch to explain.
Sounds like @Apple needs to look into their process for this. Make sure there are checks in place to prevent this happening to anyone else.
Ben Lovejoy (tweet):
It seems incredible that all this could happen without human intervention. Apple does, of course, have to act swiftly when there is a chance of malware in the Mac App Store, but you would have thought it would have pinged a human being to verify the situation before inconveniencing significant number of Mac users, and potentially doing permanent damage to a developer’s reputation. Most app users will never know the story behind this, only that they bought an app, Apple told them it was malware, and they deleted it as instructed.
This was a big goof on Apple’s part. I’m glad it only lasted a day, but it should not have happened in the first place.
False positives happen with automated systems. Apple needs a faster way to detect and reverse them. A lost day of revenue can be A LOT of money to an indie.
The lost revenue is not that big of a deal IMHO. One can deal with one day of revenue falling out. As I note in the blog post, the more damaging is the alert notifying a user that the app will damage their computer. I’ve worked hard to earn some reputation and this damages it.
Apple might like to disingenuously compare itself to a brick and mortar store, but is there really an analogous case where something like this happens overnight to an independent supplier, with little ability for recourse?
Update (2020-08-10): Howard Oakley (tweet):
There’s also the curious question as to why Apple revoked the certificate, rather than pulled one or more of Charlie’s notarizations. When it introduced notarization, one of Apple’s justifications was that it would provide finer control, rather than the huge and heavy-handed kill switch of revoking a certificate and blocking everything signed with that. Perhaps Apple didn’t really mean that after all, but just wanted another level of control over your Mac?
Apple has since apologised to Charlie Monroe for its error. It hasn’t released any statement to reassure other developers that it’s changing anything which might prevent such as catastrophe from happening again, nor has it explained to the billions who run third-party software on Apple products how it’s going to prevent a recurrence – which could readily prevent any Apple user from using their software on their computer or device.
[…]
Apple will no doubt try to ride this one out in silence, as it usually does in matters of security. For developers and users, that doesn’t answer these fundamental questions.
I’m not convinced that notarization-based blocking would work in case like this (but with actual malware), so it’s not clear what Apple was referring to when it said that notarization “provides a much better experience” than revoking the certificate.
There were so many failures here:
- Faulty automated system
- Lack of human review
- Blocking all the apps instead of just the particular binaries that were under suspicion
- Lack of communication with developer
- Alert saying that the apps would damage your Mac, when there was no evidence of that
- Lack of communication with users, afterwards, that the software was not in fact dangerous
- Lack of public acknowledgement of the problem with an explanation for how Apple will prevent this from happening again
The crazy thing about the Charlie Monroe situation is that not only is there no phone # to call Apple to find out why your Developer ID cert is revoked, there’s no # to call to report your cert was compromised! You can’t even revoke it yourself, unlike your Mac App Store cert.
I can’t help but think not only should Apple turn off its automated execution of such bans but they should also move to a more nuclear-launch type system where at least two people need to turn their key. This is an incredibly destructive event for the third-party vendor like Charlie. It’s unprofessional of Apple to have this connected to an automated system.
Additionally, if Gatekeeper is truly about protecting the users, I don’t see why we can’t have a transparency report listing the identifiers that have been disabled and why. A lot of people keep saying Apple does not abuse this power, but there is no proof to this; it is a closed system. We only know of Charlie’s situation because he posted it on Twitter. Considering it wasn’t too long ago when the App Store Guidelines down right threatened you about going public I don’t know if we can give Apple the benefit of the doubt here.
This happened months ago with the game League of legends as well
Can’t wait for Apple to accidentally revoke BMW’s certificate so no one can open or start the cars.
Update (2020-08-12): Charlie Monroe (tweet):
Here is a quote from Apple:
We appreciate your patience while we continued our investigation into why your Developer ID certificate was erroneously revoked and to examine ways in which we could assist you. We determined that your app Downie 4 was erroneously identified as malicious due to invalid logic in our malware detection system. This triggered the revocation of your certificate under Section 5.4 of the Developer Program License Agreement. This should not have happened and teams across Apple have been working diligently to figure out a solution.
Earlier today, we successfully un-revoked your Developer ID certificate. Users who were affected by the initial revocation will have app functionality restored when their OCSP cache refreshes (typically within 2 hours).
See also: Core Intuition.
Update (2020-08-24): Nick Heer:
Apple said in an apology email to Monroe that it is “taking action to make sure this doesn’t happen in the future”, but what does that mean? Why isn’t this being communicated more broadly to developers who might reasonably be spooked by this incident?
25 Comments RSS · Twitter
You'd think Apple would have thought through scenarios like this, and would have an immediate solution available to fix any consequences of introducing and then breaking distribution certificates. But no. It seems they didn't. The whole thing was clearly very well thought out (sarcasm).
Charlie Monroe is a quality dev.
Screw notarization and screw all the Apple execs. This.is. FRAUD.
We knew this was coming, abuse of “security controls.”
Big More of the Same? “Notarization?”
Hard pass, y’all at AAPL with your “subjective judgement” burned burned a good dev and a bunch of happy endusers.
This is total BS on Apple's part having control over apps this way. This is also a completely hypocritical approach by Apple whom opposed such "Big Brother" ideologies with their 1984 advertisement. Seems to me Apple was always Big Bro just hiding in the closet waiting to be set loose. I hope Charlie Monroe is back on track soon and Apple will compensate him for their stupid mistake.
Absolutely shameful behavior from Apple. Permute and Downie are wonderful apps that embody the best of Mac software design — far more so than a lot of the Catalyst crap Apple has shoveled out the door as of late.
This is bad and maybe I’m behind in my reading on the subject, but it sounds like it could be an honest mistake rather than a purposeful action.
It still shows how dangerous for developers this whole system can be.
It could have been an accident, but it is still my worst Apple fear realized. Ever since Notarization was introduced I have waited for this to happen. Once you notarize you’re app you give full control to Apple. In the last few years Apple has given itself more and more control in the name of security. There is always a balance, but for me they have gone too far.
This is terrible!
Tried with clean 10.14.6 in VM:
1) Restore snapshot, disable internet -> app is working fine.
2) Just enable internet and wait a minute -> app doesn't run. No clear message to user (or developer) what's the cause and how to resolve problem.
From crash report: "EXC_CRASH (Code Signature Invalid)".
It is possible to re-sign app by hand:
codesign --force --deep --sign - /Applications/
App executes now, but this is not good solution for average user.
One of Charlie's apps is accounting software. In our country, there are substantial penalties if a company can't send electronic info from cash register (EET) or prepare tax reports in time. If it will not be resolved ASAP, Apple effectively kills all serious software on the platform. Apple becomes as stupid as Windows "antivirus" companies with their countless false positives.
The more I think about it, the more I think that Notarization is really a step too far, veering into real anti-trust territory. They really need to be investigated *seriously* by the Feds. Probably won't happen though.
Via Notarization, Apple basically owns, in the malicious hacker sense, but also the very real sense, all our machines. We will only be able to conduct business using apps that they approve or can revoke at any time. This is more egregious than whatever MSFT did back in the day.
That $6000 shiny maxed out MBP? You don't own it. Apple does. And by the time these shinies become "Apple Silicon" you'll be owning a glorified iPad with keys and a mouse, and Apple will decide what Apps you run on it, not Paddle, not SetApp, and certainly not you, Mr. lowest EndUser.
I wish I could time travel back to 1984 and tell my younger self "Ignore 1984, Ignore Think Different. It's a trap.." Shoulda gone Windoze. Wouldn't have this grief. Might have other grief, but not being totally "owned" by Apple via Notarization.
>I wish I could be a fly on the wall when decisions like this get made
I see two reasonably likely options:
1. This wasn't an actual decision, somebody typed some kind of ID into a text field and mistyped a character, thus killing a random dev's apps.
2. Google told Apple that the YouTube downloading app was violating its TOS (or something along those lines)
Option 1 seems less likely. If it's a random act of violence, it's a pretty astonishing coincidence that this particular dev got hit, so either it wasn't random, or this kind of thing happens very often, but we haven't heard about it yet because no prominent dev was hit.
So this leaves Option 2. You can't run any apps on your Mac that big multinational corporations don't like.
>Shoulda gone Windoze
You still can. Come over to the dark side. We have cookies and you can run any app you want.
UPDATE: Developer Charlie Monroe has told AppleInsider that the issue has been resolved. "Apple just called and apologized for the complications," he said. "The issue was caused by my account being erroneously flagged by automated processes as malicious and was put on hold."
...but still not working. Probably author will need update all apps with new certificate...
@Lukas
Turns out it was more likely Option 1. At least Apple claims that they made a simple mistake, or their automated malware detection system did. And apparently Apple has no safeguard that have someone (who actually understands malware) verify it first before damaging someone's business and reputation.
Now Charlie has to re-sign every app he has released, including older versions he's still offering to his customers.
And no relief offered by Apple for the damages the caused.
I wouldn't be surprised if the agreement we're all forced to sign states that you can't make claims to Apple for negligence on their part.
While I'm making my money with macOS software I moved some time ago to linux for my personal computing. It's not as polished as macOS but at least no one's gonna remote wipe my software.
I was actually waiting to see if this was possibly a legitimate revocation, but that update confirms the worst case scenario. "Resolved" my ass, this is how it begins.
I wonder if this has already happened to other devs, and we just didn't hear about it until now because they weren't as longstanding/well-known.
>Turns out it was more likely Option 1.
TBH, I didn't even consider the possibility that Apple could have automated processes that just completely end a developer, apparently either without human intervention, or without competent people clicking the "yes, really" checkbox.
As much as Apple likes to insinuate that they're the company who cares, in situations where it matters they sure do show a complete lack of humanity.
I’m going to pattern match here: I’m betting Apple has exactly ZERO developer relations people in the EU in Krystof’s timezone.
I know this due to the time lag and due to the Kapeli incident wherein a monolingual Apple PR person talked over, and kept repeating what “Apple wanted from him.” No cultural, linguistic or any other savvy was shown to that dev.
Both of these devs are based in the EU, or in Eastern Europe. The Pulltube dev as well.
If Apple really cared they would hire key developer relations “locals” or savvy expats to “work from home” to salve these issues as they crop up in the developer’s timezone , and not rely on “staff on PST back at the donut.”
You would think that with all the outsourcing and offshoring that Apple does that they would be cognizant of this, and it would extend to developer relations. You would think.
If Apple feels the need to police every app inside AND outside the AppStore, they’d better have “rapid response” staff all over the world before one of their “accidental flaggings” shuts down an end user, a bank or even a hospital “for 24 hours.”
Or (as I suspect) maybe they shouldn’t have “let go” of the one key person who would’ve said NO NO NO you fools to Notarization as well as this “automated malware flagging system” that is, at best, a really bad idea on par with Boeing MCAS.
@Thomas and @Lukas, and Michael, who knows more about Notarization from me. You guys are going to love this:
Option 3: Apple is/was testing a new, much more invasive version of Malware/Notarization/ for “Apple Silicon” and BigMore
OfTheSame. Krrystof has a fanboy inside Apple so they used his data, because live data is better.
If you’ve ever done e-commerce testing or dev YOU KNOW what happened next: 2 environments, “test” and “release”.(sometimes called production)
Apple oompa-loompah somehow Runs his “test” on the “release” environment and the rest is history. Dear Apple, this better be a “future product” that has no future.
> Or (as I suspect) maybe they shouldn’t have “let go” of the one key person who would’ve said NO NO NO you fools to Notarization
Who would that be? If you're going to say Schiller, what's your reasoning? He always seemed like the worst type of executive to me -- someone who has been there long enough to know better, yet still makes stupid decisions that are hostile to long-time Mac users and developers. He's the guy who made everything require dongles. "Courage", remember?
@Ben Oh no, Schiller is NOT who I was thinking about at all, from what I’ve heard he was part of the problem. Personally I’m glad he’s gone.
it would’ve been some nameless upper middle manager maybe a lower badge number lifer GenX more like a Woz or a Forstall, maybe one of the reasons he was ousted.
Apple let go of a bunch of senior “expensive” employees like Sal Soghoian and it shows. There is no brakes or common sense on their “young and cheap” crazy train ATP.
Play games with lower salary, win stupid prizes like unseasoned workers with little to no initiative— that’s silly valley as I left it, not just Apple. Twitter, FB, “youth and diversity” (read: cheap ‘at will’ hires or even contractors) are more valued.
I follow and know some ex-Apple folks, they were NOT happy with the “kill-switch thing or with Schiller’s “management” at all.
[…] their 30% or 15% cut or isn’t Netflix or Spotify is on thin ice. In an Orwellian move, they remote-killed Charlie Monroe’s apps and told users that they couldn’t run apps they’d paid for, because the apps would […]
[…] of this incident in light of recent App Store matters: first it was Hey, and then WordPress, Charlie Monroe’s account suspension, and — of course — Epic Games’ lawsuit joined the mix of confusing App Store policing. In the […]
