No One Minding the Store
I woke up to an inbox full of e-mails from customers reporting that my apps wouldn’t launch. This included new customers who had just purchased from the Mac App Store as well as people who had purchased long ago, hadn’t made any changes, and expected that things would just keep working.
On my own Mac, 1Password and Dash wouldn’t launch until I entered the Apple ID password for my App Store account. For some customers, the fix is more complicated: restarting the Mac or deleting and redownloading the app. I was in the middle of using ReadKit, when it suddenly quit, then wouldn’t launch, with the OS reporting that it was damaged. However, redownloading the app didn’t work; I had to restart the Mac to get it running. Then I got the password dialog for Tweetbot. In some cases, there seems to be no way to get the App Store version working, so I’ve pointed customers to the direct sale versions of the apps and issued them temporary serial numbers. Fortunately, my apps don’t require iCloud, Map Kit, or other system services that are withheld from non–App Store apps.
The Mac App Store is supposed to make things easier, but it’s also a single point of failure. Not only is it neglected, but sometimes even the existing functionality stops working. Mac OS X 10.9 introduced a code signing bug that prevented me from submitting updates for several months. In June 2015, there was a month-long iTunes Connect bug that prevented my uploaded build from entering the review queue. And I currently have a bug fix update that Apple has been reviewing for 33 days (with 8 days of waiting before that). When I inquired about the status, Apple told me that everything was normal and that I should just keep waiting. In short, the system is broken on multiple levels, and there is no evidence to suggest that things will get better.
Paul Haddad shows the expired certificate that seems to be the source of the problem.
Dan Counsell shows a flurry of “App is damaged” dialogs.
Every single app I have downloaded from the Mac app store is failing to launch, with a variety of errors. Every one.
Um. Launching Photoshop because MAS Acorn isn’t opening due to MASpocolypse.
The “damaged” screen seems to be a GateKeeper glitch (fixed by reboot). Then, some apps don’t check expiring receipt certs; most do.
Turns out that the App Store is just another DRM scheme with all the nonsense and dysfunction that implies. Who’d’a thunk it.
Whoa, serious Mac App Store problem: It is delivering a binary to users that is still waiting for review; crashing on receipt validation.
Had to pull the app from the store, because otherwise all my customers will upgrade and be left with a non-functioning app.
Catch-22. (Also, no, Apple. It wasn’t. I bought this app on this computer, and just yesterday, it worked fine.)
Seriously, what a bunch of noobs sometimes…
Update (2015-11-12): Craig Hockenberry:
Just verified that you don’t need to reboot to work around the Mac App Store certificate problem. Instead:
$ killall -KILL storeaccountd
When that dialog says “YourApp” is damaged, who’s the customer going to contact? You or Apple?
Worse, there’s no way for us to be proactive about this situation because we have no fricken’ idea who’s affected.
This is because only Apple has the customers’ contact information.
Restart your computer. (This is a necessary step, because the App Store’s code signing certificate has expired, and restarting will clear the local certificate cache.)
Necessary, but alas not always sufficient.
Mac App Store meltdown: the less a developer heeded Apple’s own advice for validating receipts, the better they look to customers today.
I can’t get MAS Fetch to launch on any OS.
This is some MobileMe-level brand tarnishing.
Apple did not respond to request for comment.
Update (2015-11-13): John Gruber:
Inexcusable for a service that is absolutely essential to users and developers.
Harsh words, but I don’t see how anyone could disagree.
So many of their products feel this way. They’re just stretched too thin. And for what? Apple Watch? They’ve lost focus.
Every aspect of this MAS cert thing is completely infuriating to me.
I spent a lot of years being sarcastic but optimistic about the Mac App Store. I guess my patience, like so many others’, has worn thin.
More than anything else, sandboxing and my assumption that the future was in the Mac App Store, has shaped my priorities the last 5 years.
Paul Haddad shows a 1-star review from a customer whose app stopped launching.
Andrew Wickliffe shows a reply from Apple Support encouraging him to post a review in the Mac App Store in the hopes of the developer contacting him. This is ironic because Apple does not let developers contact customers who post reviews.
A customer e-mailed me to say that AppleCare told him that “actually the app store certificates come from the developer of the app, not Apple. Apple only approves the certificates. […] So their current position is that it’s the responsibility of the app developer to fix it!” I think this is incorrect and that Apple itself signs the apps that the store distributes. My own certificates are for submitting to the Mac App Store and have not expired. Furthermore, if AppleCare’s explanation were correct, the workarounds (entering your password, redownloading the app, restarting the Mac to clear the caches) wouldn’t work for anyone.
Wishing all my favorite MAS developers the best after Apple dropped the cert and then blamed devs. Sad situation. Everyone take a month off.
I woke up in the middle of the night thinking about how egregious Apple’s behavior this week has been toward devs.
Between Apple nuking sideloading for f.lux and the Mac App Store issues, I’m really feeling ecosystem angst today.
This is not just unacceptable: this is a fundamental violation of the trust that both app developers and customers have placed in Apple, namely that bought, installed and compatible apps would keep working (short of any dramatic action taken for consumer protection so that they would not, such as revoking the certificate of a malicious developer).
[…]
So, in turn, how am I supposed to trust iCloud or Apple Maps, if I am not sure I can run any app that can access it? As if these services did not already have a reputation…
But even more troubling are the implications for long-term usage and preservation of software and it data.
Before it expired, Apple issued a new certificate, but one using SHA-2 (secure hash algorithm 2). This was supposed to be transparent, but once the old certificate expired, some people began experiencing problems.
First, outdated certificate information was stuck in cache, which required some people to reboot or re-authenticate in order to clear it out.
Second, some apps are apparently using an old version of OpenSSL for receipt validation, and—you guessed it!—it doesn’t support SHA-2, and hence isn’t compatible with the new certificate.
This makes sense, although I suspect there are also other factors involved because it doesn’t explain all the cases that I’ve heard about.
Grabbed a new Mac App Store receipt. They are back to using SHA1 and it now has an expiration date in 2023.
A security certificate Apple installed to protect users from malware had expired on Nov. 11, 21:58:01 GMT—precisely five years after its original creation—and nobody at Apple had thought to renew it.
The company fixed the problem—pushing through a new certificate that expires in 2035—but not before breaking untold numbers of Mac apps and confusing and inconveniencing countless Mac owners.
Matt Stevens says that developers need to be careful to validate App Store receipts using the receipt’s creation date rather than the current time. The creation date field was not initially documented, and Apple’s sample code uses the current time.
What we know, so far, is the receipts embedded in most, if not all, Mac App Store apps became invalid yesterday. This happened without any advance warning from the mothership. How apps reacted to this varied. Our apps are among those affected, and in the worst way. […] In the meantime, we’re giving away our apps at our online store.
As of November 13, 2015, it appears that Apple has fixed this issue. If your copy of Fetch from the Mac App Store does not open, drag it to the trash, empty the trash, and download a fresh copy from the App Store.
Today’s ongoing certificate expiration issue is yet another reminder that Apple needs to commit more talent and resources to the Mac App Store, or get rid of it.
Graeme Devine posts another response from Apple Support blaming the developer.
Update (2015-11-14): Shawn King:
This is a huge embarrassment to Apple (and one they haven’t explained or apologized for) as well as being a giant pain point for developers. After all, when your app stops working, who do you contact? The developer or Apple?
Daniel returns from Amsterdam to find Mac App Store issues abound. Manton buys an iPad Pro but has to wait for the Pencil. The two discuss the Mac App Store’s 6-year failure to evolve substantially, and dig into the emotional highs and lows of enjoying and surviving Apple’s platform constraints.
When a certificate fails—whether through an accidental expiration or due to tampering—it’s a reasonable precaution for software to act as if the sky is falling, because there’s no good reason it should fail unless an attack or compromise is underway.
[…]
And yet because Apple’s infrastructure is seemingly so brittle, not only did it happen, it inconvenienced an unknown number of Mac App Store software purchasers, while offloading the frustration and customer-service load to developers.
There are actually several different unfortunate problems here. First, the “damaged” dialog seems to be caused by some sort of cache or memory corruption in the system processes that coordinate to implement GateKeeper and the app store updates; some reports say killing the “storeagentd” process solves this problem without rebooting. (My system doesn’t seem to run this, FWIW.) What not everyone knows is that this dialog appears before the app it allowed to run; that is, it’s not affected by any checking done inside the app itself!
Second, asking for a new AppleID password. This is caused by the app itself checking the store receipt; something strongly recommended by Apple, since otherwise, it’s easy to copy a downloaded app to another computer and having it run there; I remember some early games not doing this and being widely pirated.
[…]
When and if you get a new version of the app, all certs will probably be new ones. So there’s no “allowing” a leaf cert to expire — they do so naturally.
[…]
Apple “pushed” a new certificate that expires in 2035. This is probably just looking in the wrong place — not knowing which certificate had expired, someone glanced at the root certificate and noticed the “new” 2035 date. Nothing new to see, of course; that cert was created in 2006!
Update (2015-11-18): Benjamin Mayo:
Apple has emailed developers about the recent damaged apps bug affecting a sizeable proportion of the OS X user base with some getting repeated errors on app launch. Whilst a reboot should be enough to invalidate and reload the certificate cache for most people, there are some weird edge cases. Apple says that a permanent fix for the caching issue will be included in a future OS X software update.
Rainer Brockerhoff notes that Apple’s e-mail linked to the wrong documentation page and neglected to mention the important receipt creation date issue.
Pierre Lebeaupin:Conceptually, there are two “security” services the Mac App Store provides: DRM, to protect the developer against unlicensed use of the app or the app being pilfered, modified, and passed off as being the modifier’s creation; and code signing, to protect the user against an attacker tampering the app between the moment the app was signed by someone the user (supposedly) trusts and the moment he runs it.
Code signing, by its nature, relies on digital certificates, and these certificates expire, for what I hear are good security reasons. The archivist does not particularly care about code signing: even if the app was tampered with by an attacker, the archivist has a pristine copy of the data, and the machine is off the network and nothing will ever exit it. Since code signing is put for the user’s benefit he should have as a last resort the ability to disengage it, otherwise this is not done for the user’s benefit and is not just code signing, is it?
Update (2015-11-20): Gus Mueller:
Maspocalypse. The gift that keeps on giving. Now I get to support family members who bought things years ago, that just stopped working.
Other users continue to find apps that aren’t working after rebooting.
Update (2015-11-24): See also Accidental Tech Podcast.
But given that the Mac is doing tremendously well, setting sales records—even if not approaching the sales volume of iOS devices—and given that Apple takes a 30-percent cut of both iOS and Mac app sales, regardless of the disparate support for the two app stores, it might behoove the company to spend a little time bringing the Mac App Store up to snuff.
Still finding new “app is damaged” errors, over a week later.
Put aside the argument about whether a fiasco like this should have ever happened in the first place. Why did it take six days for Apple to publicly respond and explain what happened?
And since Apple only contacted developers and select Mac press, not the people who bought the apps, most users probably never heard anything about it.
Update (2015-11-29): Rob Griffiths:
However, with a few simple changes—and one not-so-simple changes—the Mac App Store really could be the place to shop for Mac software, instead of a place where you only find apps that meet Apple’s narrow definition of what an app should be.
42 Comments RSS · Twitter
For me, it's a catch-22 because the app that won't open unless I enter the Mac App Store password is... wait for it... 1Password. Which contains my Mac App Store password.
I guess I have to admit to not fully understanding how the *App Stores work. I had always thought that once you downloaded and installed an app, it would run forever. Now it seems that Apple has a true "kill switch" on any app and software you have licensed.
@Lukas, I recently learned about "1PasswordAnywhere". If you use Dropbox to synch and your vault is in the agilekeychain format you can use 1Password without the app. See https://support.1password.com/guides/mac/1passwordanywhere.html.
@John In this case, it’s more like a life extending drug that you have to keep taking than a kill switch. It looks like the OS is always validating app signatures. Apps can be “killed” even if your Mac is not connected to the network because the signing certificate will eventually expire. Hopefully you have Internet access and remember your App Store password when this happens.
@Joel 1PasswordAnywhere is a great feature, although (due to browser changes) it no longer works locally, only when the files are hosted on a server. If you put them on Dropbox, you need to remember your Dropbox password. If you host them publicly, you open them up to attacks.
Fortunately, I also have 1Password running on my Android phone, which allowed me to retrieve my App Store password.
As for the App Store problem, just today, I had a similar problem with a non-App-Store app. I have a ton of Lineform files; for a while, Lineform was my preferred vector illustration app. I own a license for the app. Today, I had to open an old Lineform file, make a change to it, and export it as a PDF. Unfortunately, I can no longer use my Lineform license, since activating the app requires it to connect to a server. Which no longer exists. Because Lineform's publisher, Freeverse, no longer exists. I can open Lineform in demo mode, and make changes to my documents, but when I export them, they're watermarked.
Sigh...
@Michael Yes. It all makes sense. I'm sure I knew that once but never really paid much attention. I'm an Apple user so I don't have to deal with such details. Having internet access is key. After any update, I have to make sure I have good, fast, uncapped internet and then run through every piece of Apple software I have to "prime the pump" and make sure it all works before leaving the house. I guess we get to go through all this again when the current certificate expires on Oct 23 19:09:31 2017 GMT.
Lukas has a point: if the developer's licensing scheme requires phoning home for activation, when home is no longer responding then it's a worse problem for the user. But that's a problem with any DRM scheme that depends on connectivity: usage of what you purchased depends on the vendor still being there and still supporting this scheme.
I wonder, will OS X 10.11 still be able to talk to the MAS server in 20 years? Or in 40 years? How will you install and run any app on your vintage iMac still running 10.11? Today's MAS problem is a temporary annoyance in comparison to what has to come eventually.
Pointing out the obvious, but Apple has little interest in the Mac. Upgrade cycles are longer and it's simply not as popular as iOS, which is an amateur consumption oriented experience but clearly where the money is, and thus the focus on pushing annual iPhone upgrades. iPhone, iPhone, iPhone. Keeping those shareholders happy.
>Today's MAS problem is a temporary annoyance in comparison to what has to come eventually.
Yep. I can still open my ClarisWorks documents from 25 years ago, and my HyperCard Stacks from almost 30 years ago, because I can get these apps to run on old systems, or in an emulator.
I'm not betting on being able to open most of today's stuff 30 years from now.
I can answer Michel's comment: that apocalypse has already happened today on older Macs. I have one stuck on 10.7 and another on 10.6. All the software bought from the Mac App Store has stopped working and says I need to redownload - but the Mac App Store doesn't allow you to download old versions, and the newer versions of apps like Tweetbot don't run on 10.7. Running killall -KILL storeaccountd doesn't work on 10.6, for instance. My copy of Byword is permanently broken, especially since they don't offer a non-MAS version. The loss of Tweetbot is hitting hard as well.
This shows a big downside of App store applications are (as opposed to direct purchase & download) that they will eventually expire and you'll be high and dry. Note to self...try to get things direct. Microsoft desperately wants to move all Windows Application purchases through a similar fee generating App Store with v10.
It shouldn't be called purchasing, that gives an incorrect connotation (that the application will run from then onward), Apple should call it the App Rental Store.
@Kohan This is another area where the Mac App Store has been neglected. The iOS App Store has let devices with older OSes download old app versions for more than two years now.
[…] Mac App Store is supposed to make things easier, but it’s also a single point of failure,” says app developer Michael Tsai, who woke up Thursday to an inbox full of e-mails from customers […]
“[…] hadn’t made any changes, and expected that things would just keep working.”
(tongue -> cheek) This right here. Can. You. Believe! The *gall* of these users? If they start getting the expectation that their purchased software is to keep working, what will they expect next? Their music and movies to keep working?!
Anyway, I expressed my thoughts at http://wanderingcoder.net/2015/11/13/mac-app-store-preservation/
@Sasparilla You're not purchasing most software anyway. You're licensing the use of the software. Most EULAs point this out clearly.
Thanks for taking the time and effort to gather all these developers' comments about the *major* app store disruption from this expired certificate. Apparently, Apple *is* asleep at the wheel. (:-( What does Ron Okamoto have to say about all this?
John: Yep, Apple's App Stores absolutely have a kill switch: http://www.macworld.com/article/1134930/iphone_killswitch.html
They've used it at least once, IIRC.
"Their music and movies to keep working?!"
Was really happy when I saw Jobs' "Thoughts on Music" or whatever it was... I don't see un-DRM'd AAC dying in the future the way my Steam games (and my iBooks too) probably will. Still mostly buying movies on disc those rare times I buy them, though.
But then, I'm a dinosaur. Subscription or tar pits it is... Claiming the end of ownership is only a slight exaggeration.
Apple does tend to ride the crest of *today's* money, so to speak. I fault them as a customer for the app stores' mvp nature, but as a stockholder, I don't react as vehemently. Yet.
About Apple's resources being stretched thin. Financially, they might be the wealthiest company in the world. They are spending huge amounts on new campuses. They can afford to hire people to take care of everything.
The Unix underpinnings in OSX are sadly out of date. Bash is version 3.2.57. When I have a scripting question and do an internet search, I can only find tips for bash version 4.x. which don't apply to the OSX version. They certainly could get some interns each summer to compile the unix binaries for the fall rollout of the new OSX. And more experiences people for more important things.
"About Apple's resources being stretched thin. Financially, they might be the wealthiest company in the world. They are spending huge amounts on new campuses. They can afford to hire people to take care of everything."
But all that's so boring compared to constructing a headquarters equipped to fly up into the sky like a spaceship.
Bash? Who cares? MAS? Who cares? UX? Who cares? Users want spaceship campuses rather than usability for their products. Keep your priorities in order, please. That money flowing in like a river is designated for shareholders, executives, and bling.
@Michel & Kohan: Alas, your absolute undying love (and equally undying machines) is not what pays Apple Inc's bills.
If you stop thinking like a fanboy and start thinking like a successful businessman, it makes total sense: the fact that you're still running 10.6 in 2015 (or 10.11 in 2025) makes you a rubbish source of revenue, so why expect Apple's love or attention when you're not giving them anything they want in return?
The product is clearly defined; you can take it or leave it as you will. Complaining about it though is pointless, on account of the massive lack of greenbacks attached.
Thanks for this info Michael. This discussion has pointed out a few problems with the App Store model that I hadn't thought about. Just like in our old ATPM days you've shown a knack for getting to the heart of the matter.
[…] Tsai, writing on the latest issue that hit the Mac App Store, preventing users to launch apps they previously […]
[…] selhání. Nejen, že je opuštěný, ale někdy i dosavadní funkcionalita přestane fungovat,“ napsal v široce linkovaném blogpostu vývojář Michael Tsai, který má na svědomí například […]
@Ramsey, Apple will drop bash before they upgrade it to version 4. This is not them asleep at the wheel, they refuse to ship a line of GPLv3 code (do to some of its more stupid clauses, imho). Why do you think they switched their investment from GCC to LLVM, or dropped shipping Samba?
[…] zorg voor verbeterde zoekfunctie in de App Stores (iOS en OSX), maar… hun toewijding aan een betere App Store blijft helaas beperkt, met het recente ‘damaged app’-probleem als […]
[…] Flucht aus dem Mac App Store. Mit Sketch, einem Designtool für den Mac, zieht eine weitere App aus dem Mac App Store aus und setzt stattdessen auf Direktvermarktung. Der Grund: Zu wenig Weiterenwicklung, kaum Reputation, keine Test-Downloads möglich. Eine Stellungnahme dazu gibt es hier. […]
Resurrecting a somewhat aged topic, but wanted to comment on what Oliver said above.
@Oliver:
> Pointing out the obvious, but Apple has little interest in the Mac. Upgrade cycles are longer and it's simply not as popular as iOS, which is an amateur consumption oriented experience but clearly where the money is, and thus the focus on pushing annual iPhone upgrades.
So true. To me this smells like Microsoft in the 90s. Upgrade cycles on Windows started to get longer, it wasn't as much a cash cow as MS Office was, and Microsoft started treating developers like dirt. The death knell was when someone finally convinced Ballmer--way waaaaay too late I might add--and he bellowed his infamous, "Developers, developers, developers!" ... I fear Apple is heading the same way.
I really think--in addition to great product development sense--that one of the reasons Apple has been so successful is because they moved to OS X with it's Unix-based OS, pleasing developers to no end--and when you have the 'influencers' happy, that trickles all the way down to all the consumers (mom, grandad, cousins, etc.) influenced by them.
> iPhone, iPhone, iPhone. Keeping those shareholders happy.
Long story short: I think APPL shareholders (I'm one) will not be happy in 2-3 years, when Apple has abandoned developers' interests to the extent that they no longer have a real computing platform to use any longer. This will lead to consumers being advised away from their products, I predict, assuming they don't fix things pronto.
[…] side of software. On the other hand, Schiller was already in charge of App Review, which is a disaster in terms of speed, and he has a history of making misleading statements when justifying […]
[…] nice to have some notice this time, but it’s not that much notice. In the event that there’s a problem with 10.6, testing […]
[…] personally, although I have enjoyed quicker review times than in the past, I continue to encounter problems simply updating my apps. My last two updates were held up […]
[…] Mac App Store has received significant criticism, but Apple’s efforts to improve it apparently have paid off, as Adobe […]
