Archive for January 6, 2015

Tuesday, January 6, 2015

The 2014 Panic Report

Cabel Sasser:

My gut says no, that full price every single time is rough, but then we’re setting the precedent that maybe not all of our major upgrades are paid upgrades, which we’ve been pretty consistent about in the past. If we could offer traditional discounted upgrades via the App Store, this paragraph wouldn’t exist. This is one area where the App Store feels like one of those novelty peanut cans with the snake inside.


To be honest, I was pretty nervous to be pulling Coda from the Mac App Store. But when we finally did it, I felt an incredible, almost indescribable sense of relief — mostly because as we began to wrap up bug fix releases, we were able to immediately post them to our customers within minutes of qualifying them. My god. That’s how it should be. There’s just no other way to put it — that’s how you treat your customers well, by reacting quickly and having total control over your destiny. To not be beholden to someone else to do our job feels just fantastic. (Also to not pay someone 30% in exchange for frequent stress is a fine deal.)


The results were interesting. We sold a couple hundred fewer units of Coda post-App Store removal, but revenue from it went up by about 44%.


I can’t comfortably say “the system worked”. It’s still an awful and nerve-wracking feeling to know that, at any minute, we could get thrown into a quagmire of e-mails, phone calls, code removal, and sadness, just by trying to ship something cool.

There’s a little more history here than I’m letting on. We had a very long, very torturous situation with Status Board almost being pulled that we’ve never written up out of sensitivity to our relationship with Apple. I only mention it here because it proves that it is possible to fix these awkward rejection situations without Apple suffering negative PR in the public eye — we did that “offline”. But it took an absolutely massive amount of mental energy and time to work through — positively Sisyphean. I would never want to do it again — I’ve run out of patience, I guess. I can say for certain that the “bad PR” version of the app dispute process is monumentally more effective. Which is a shame.


This is the biggest problem we’ve been grappling with all year: we simply don’t make enough money from our iOS apps. We’re building apps that are, if I may say so, world-class and desktop-quality. They are packed with features, they look stunning, we offer excellent support for them, and development is constant. I’m deeply proud of our iOS apps. But… they’re hard to justify working on.

First, what a great piece of writing this post is. Second, Panic’s low iOS revenue is a terrible indicator for the health of the platform. If top-quality apps, a solid customer base and brand, and marketing support from Apple aren’t enough, what does one need to succeed? Apple clearly likes Panic-style apps, so you would think it would try to encourage their development. Instead, Apple has designed the App Store to do pretty much the opposite. Why? This strategy seems to make no long-term sense unless one believes that quality apps don’t matter.

Milen Dzhumerov:

It escapes my mind why a premium hardware company would want to run a software store that’s equivalent to a dollar shop (or worse). Yes, people will buy phones because there’s tons of free apps but we’re slowly reaching a point (or already there) where the quality of those apps is so bad, that all other platforms have them as well. The number of apps is no longer a competitive advantage like it used to be in 2008. In the long term, it wouldn’t matter what phone you buy, if all the software is available on all mobile OSes.

Due to the un-sustainability of the App Store policies, the premium software that will keep people in the iOS ecosystem just won’t get built - that’s a recipe for disaster. Just look at the Panic 2014 report (and multitude other reports as well). It’s precisely those developers that have built the ecosystem, have been innovating and creating superb 3rd party software and have been advocating Apple products for decades. And now Apple have turned their backs on them.

Update (2015-01-08): Wil Shipley has some suggestions for the App Store.

Apple’s Software Quality, Continued

Marco Arment (comments, follow-up) continues on the theme of my post from October:

Apple’s hardware today is amazing — it has never been better. But the software quality has fallen so much in the last few years that I’m deeply concerned for its future. I’m typing this on a computer whose existence I didn’t even think would be possible yet, but it runs an OS with embarrassing bugs and fundamental regressions. Just a few years ago, we would have relentlessly made fun of Windows users for these same bugs on their inferior OS, but we can’t talk anymore.


I suspect the rapid decline of Apple’s software is a sign that marketing is too high a priority at Apple today: having major new releases every year is clearly impossible for the engineering teams to keep up with while maintaining quality. Maybe it’s an engineering problem, but I suspect not — I doubt that any cohesive engineering team could keep up with these demands and maintain significantly higher quality.

A former Apple engineer:

With Bertrand, we would move in giant monolithic releases where every group would just dump in whatever they had ready and the whole thing would get released with nightly builds. With SnowLeopard in particular, I remember three dozen releases in a row where Xcode was unusable due to obj-c garbage collection issues. Random stuff you didn’t expect like CoreGraphics would have showstopper issues and then we’d report it and it would get fixed by the next week.

This resulted in extremely late releases that had a ton of bugs that we piled patches onto as time went on.

Craig moved the organization onto a sprint system, where we would develop new features for 2 weeks and then spend a week fixing bugs. After 10 or 12 or 16 of these cycles, we would deem it ready and ship it out.


What has changed is that releases and features happen more often. Tiger and Leopard had a good 2 years to mature and get patches while their delayed successors missed target dates. […] They felt stable because they were just old, sort of like Debian stable. Meanwhile, the development versions of Leopard and Snow Leopard (the two I spent most of my career at Apple developing) were downright horrible and unreleasable. […] It’s just that you remember them better because they had a longer history as a stable legacy OS than the modern versions.

[Update (2015-01-12): There are more comments about this change in development styles here.]

A current Apple engineer:

the sprint (milestone) development system is still in place... it’s not the problem though, it’s the problem is the focus on new useless [imo] features at the expense of core functionality and quality

Another former Apple engineer:

The good thing about the nightly builds was you didn’t have to use them, and people could respond quickly to showstoppers rather than wait for a sprint. There was a quicklook team to catch bugs which stopped nightly builds from release to general dev, and unless you really needed that build ( to test a new API) you didn’t install until quality was restored. The xCode bug was therefore unusual, as they could have fixed it the next day ( and with enough heat they would have).

With fortnightly builds which are only then released to Engineering, if that is now what is happening, there will be massive instability every 2 weeks, until the final round of bug fixing cycle stops all features being added.

Which can’t finish in time because the OS has to be released at an Apple event.

Geoff Wozniak (tweet, comments):

There are lots of other little things that irk me: mds being a hog, distnoted being a hog, lack of virtualization, other system services mysteriously Wring up, bogging the system down. It doesn’t help that the Macbook Pro I have is one of those lemons that overheats easily, thus kicking the fans into “rocket taking oX” mode. At this point, my default position on Apple software in OS X has moved from “probably good” to “probably not OK”. They seem more interested in pumping out quantity by way of more upgrades. It’s death by a thousand cuts, but it’s death nonetheless.

Daniel Jalkut:

The current state of Apple’s software does not particularly concern me. Are there embarrassing blemishes? Yes. Does the annual schedule for major OS updates seem rushed? Of course. Are there Apple employees in positions of power who do not share Marco’s and my enthusiasm for software that “just works?” I regret to surmise that, indeed, there are.


Over the years I have never been at a loss for identifying problems big and small with Apple’s products, or with the way it conducts its business. I’m sure I had plenty of complaints starting in 2002, but I didn’t start blogging in earnest until 2005. Here are some highlights to remind you that things have never been fine with Apple[…]

John Gruber:

Apple hasn’t (yet) lost any ground in the market, but they’ve created an opportunity for that to happen, because they’ve squandered a lot of trust with their users. It’s not that Apple has lost the “it just works” crown to a competitor, but rather that they’ve seeded a perception that Apple’s stuff doesn’t work, either.


Overall, Apple’s legendary stability and reliability have suffered some major blows.

For the time being, Mac OS X and iOS are effectively feature complete. The one thing we’ve repeatedly heard from users is a cry for stability. We’d like to see OS X 10.11 and iOS 9 be “Snow Leopard” updates that — just as 10.6 Snow Leopard did for 10.5 Leopard — remove cruft, clean up problems, and polish existing features so that we have a stable base going forward.

Dr. Drang channels Tim O’Reilly:

We are Apple’s best customers. By “we,” I don’t mean fanbloggers in particular, I mean power users in general. Those of us who learn the deep details of the software we use, or who write scripts and Automator actions to speed up our work. We’re not Apple’s best customers because we buy lots of Apple products (although some of us do). We’re its best customers because of our leverage.


Are you as enthusiastic about demonstrating recent versions of OS X as you were about Leopard? Have you avoided family members who keep asking you why their iPhones don’t have enough free space to install iOS 8? Do you think it might be better if your friends stick with Android because then you won’t feel responsible if some of their data doesn’t sync?

Casey Liss:

It was Apple that chose to establish this cadence, and chose to stick with it. Now, for better or worse, they may have backed themselves into a corner. When you’re that ox of a man standing in the boxing ring, the last thing you want to do is show weakness. To give the plucky little guy hope.

Lukas Mathis:

I get the same impression: Apple doesn’t see what’s happening.

It seems to me that the media covering Apple is partly to blame for this. There seem to be two main factions covering Apple: people who dislike Apple, and whose opinions can thus be disregarded. And people who like Apple, but would rather talk about how wrong the first faction is, and how badly Samsung and Google are doing, than discuss the problems Apple’s own products have.

Kirk McElhearn:

I no longer want to be the first to install an update to iOS or OS X, because I simply don’t trust that Apple will get it right.


Apple is losing its trust among long-term users. The company may be gaining plenty of new users, who, for now, are willing to accept this kind of problem, since they’re used to platforms where things may be even worse. But if Apple loses the loyalty of their oldest users, the company’s reputation will change from the company that we trusted, to just another computer and device manufacturer.


Neither I nor the many others who echoed his feelings did so because of any desire to trash Apple; it was rather because we are genuinely concerned that this company with which we have a long relationship is showing signs of decreasing quality in its software.

Guy English:

What I’d like to call out is this particular paragraph I’ve quoted. We don’t. We don’t. We need.

Marco may be passed off as a developer here and dismissed as expressing developer thoughts. The truth is, at least the truth I’ve known from supporting and dealing with people who aren’t technical who use these devices every single day — “we don’t” isn’t about developers.

Update (2015-01-06): Craig Hockenberry (tweet):

Our concerns come from seeing the start of something pernicious: our beloved platform is becoming harder to use because of a lot of small software failures.

It’s literally a death by a thousand tiny little cuts.

Apple may not be aware of the scope of these issues because many of these annoyances go unreported. I’m guilty of this when I open a Finder window on a network share. While the spinner in the window wastes my time, I think about writing a Radar, but a minute later it’s forgotten. Until the next time.


But I have a pretty simple metric for the current state of Apple’s software: prior to the release of Yosemite, I could go months between restarts (usually only to install updates.) Lately, I feel lucky to go a week without some kind of problem that requires a complete reset.


Every holiday season, my wife and I make sure that everyone’s computer is up-to-date and running smoothly. This year, for the first time ever, we didn’t install the latest version of OS X. The problems with Screen Sharing are especially problematic: it’s how we do tech support throughout the year.

Garrett Murray:

This, a million times over[…]

Jason Snell:

This is a complicated issue, and one difficult to assess without knowing the facts about what’s happening inside of Apple. An avalanche of bug complaints and misty water-colored memories about the stability of Snow Leopard aside, I do think that there’s a problem here.

Federico Viticci:

When I started MacStories in 2009, two pillars sustained the narrative around Apple: its “attention to detail” and the “just works” aspect of its software. Since iOS 7, it feels like those pillars have begun eroding at a quicker pace.


What it comes down to, really, is balance. I believe that Apple used to be more disciplined at balancing its desire for new features and commitment to refinements.

My problem with most commentary to Marco’s piece is the binary interpretation of Apple’s software releases: that they should either do new stuff or fix bugs. That’s too simplistic and shortsighted. Software is never bug-free, but there’s a threshold where it’s good enough to be shipped. I want to see Apple get better at releasing updates like iOS 8 and Yosemite with a better balance between novelty and stability. They shouldn’t be mutually exclusive.

Joe Cieplinski:

I bring this up not because I disagree with Marco Arment’s post from last night about the recent decline in Apple’s software quality, which is undeniable. I just think it helps to remember that mass market success and decline in build quality pretty much go hand in hand. And that we’ve been here with Apple before. Many times.


The only question now is how does Apple balance the speed of innovation against the need to maintain quality moving forward? As the Apple Watch starts shipping later this year, and the critics of Cook finally quiet down about Apple’s inability to have a hit new product, will Apple shift gears a little? Will the organization realize that it’s out of whack and start to feel the need for a Snow Leopard moment? I think it probably will.

Victor Agreda Jr:

Apple’s longtime brand promise of “it just works” applies to fewer and fewer products the company makes.

David Chartier:

Conversations I overhear in public have gone from “hey Apple’s new thing looks pretty great” to “meh, just wait until they work out all the problems.”

Shawn King:

I hate agreeing with Arment but sometimes, he’s bang on. I believe in this case he is. From embarrassing software updates to apps that simply don’t work properly or well – Apple’s poor quality and functionality of the being just one of many examples – the assessment that “We don’t need major OS releases every year” is something many of us hope Apple listens and pays attention to.

Nick Heer:

For every dumb bug or feature regression, I also find something that works far better than it ever has, and often far better than its competition. Perhaps the big thing Apple needs to do in 2015 is reassert its unique skill in creating unique, easy-to-use software that — hyperbolically — “just works”. Not necessarily with new features, but by making the features that already exist truly great.

Lloyd Chambers:

MPG has been writing about Apple Core Rot for a year now, and longer before making it explicit. Lately, there are so many dozens of specific issues that could be documented in OS X Yosemite that weeks could be spent documenting them. While adding the numerous examples to Apple Core Rot would strengthen the piece tremendously, MPG has useful work to do.

BTW, my Apple Mail VIP list has been deleted about fifty times now. Uncle.

Graham Lee:

So there are plenty of alternatives, many of which are good in some ways and bad in others, and all I know is that I don’t want things to be like this, without being able to say I want one of those instead. I don’t even think there will be one of those, at least not in the sense of a competitor to Apple on laptop operating systems.


In fact I don’t even think that Apple’s systems are bad, they’ve just lost the “it just works” sheen. It’s just that when you combine that with the lack of credible alternative, you realise the problem is probably in expecting some corporation to put loads of resources into something that’s not going to have a great value, and merely needs to be “good enough” to avoid having any strategic penalty.

Update (2015-01-10): Ashley Nelson-Hornstein:

I know a ton of smart people work at the mothership, so I’m betting these incredibly intelligent people have already noticed the problem and moved to make the proper adjustments. I think what we likely can’t see from the outside are the projects that have already been cancelled to allocate resources for bugs.

Update (2015-01-11): Riccardo Mori:

When a new OS X version introduces issues that were absent in the previous one, that doesn’t go unnoticed, especially when such issues — like Wi-Fi reliability — are taking two minor OS X releases to be fixed. When a new OS X version makes your Mac feels more sluggish than it was in the previous version, that perception clouds whatever new exciting features the new OS X version brings to the table.

John Gruber:

But in avoiding the problems of stagnation and hubris, it feels like Apple has run into a different problem: nothing ever feels settled and stable. If the pattern Apple has established the last two years holds, by the time the loose screws get tightened in iOS 8 and OS X 10.10, we’ll be getting developer betas of iOS 9 and OS X 10.11 at WWDC. And as Guy English has keenly remarked numerous times, the annual schedule means that by now — that is, January — a lot of engineering talent in Cupertino is being directed to next year’s OS releases, leaving less talent on the task of tightening the remaining loose screws in last year’s.

Bogdan Popescu:

  • Last year I spent at least 30% of my development time fixing Apple bugs.
  • I spent 4 days just to get the lovely new search field in Yosemite to work properly. In the end, I just found a hacky way of reverting it to how it was before Yosemite.
  • I always check if an Apple framework or class is at least 2 years old before using it. If it’s not, I assume it’s full of bugs and I try to use something else.

Update (2015-01-12): Glenn Fleishman:

Many of us have been grumbling quite publicly since iOS 7 and Mavericks shipped that the fit and finish we expect either on release or shortly afterwards for Mac OS X and iOS has slipped. That we spent a lot of time dealing with bugs or, if we write about Apple, teaching people how to avoid them or work around them. That software and OS problems, once they occur, are rarely fixed in part or full; features we need are removed rather than matured; and new features are added that aren’t fully baked.


Part of what makes these sorts of statements reasonable, though, is to enumerate the problems, whether they’re long-running or unique to Yosemite or iOS 8 (or to the last two releases of each system). Here’s a list of regularly recurring issues or fundamental problems I’ve seen supplemented by those provided by others.

Chuq von Rospach:

This isn’t an “Apple is doomed” scenario, but to me, the trendlines are negative — there is no cliff, but the beginning of the same kind of worry for a trip into the future that ends up looking like SGI (remember them? No? Ask your dad). A couple of people put up the claim that Apple was nowhere near as bad as Microsoft, and I agree, but to me, that’s irrelevant. What we’re comparing is not Apple compared to any other company, but to the Apple that should be, and the existing Apple and the possible Apple are starting to diverge.


My evidence for the prosecution is the quality of iCloud, the absolute disaster that is today’s iTunes (a tool that’s three or four years overdue for a complete overhaul) and the woeful quality of many of the Apps and how those have been changed significantly in non-compatible ways without any real recourse for existing users and no real warning to let them prepare for the update. That’s just not understanding or caring for the end user, and to me unacceptable.

Update (2015-01-14): Nick Heer:

So many of these reactions are simultaneously true. Yes, there are extremely stupid bugs and regressions littered throughout Apple’s software products. Yes, there’s the impression of a downward slide in quality assurance. And, yes, there have previously been really stupid bugs and regressions. I think Apple is cognizant of the fact that their software quality needs to improve faster than they gain new users; if it’s slower, it feels significantly worse than it really is.

Update (2015-01-17): John Gruber found a 2004 article, “Apple to Slow Down the Pace on Mac OS X.”

John Gruber and Marco Arment discuss Apple’s software quality on The Talk Show.

Update (2015-01-18): Jean-Louis Gassée:

For the past six months or so, I’ve become increasingly concerned about the quality of Apple software. From the painful gestation of OS X 10.10 (Yosemite) with its damaged iWork apps, to the chaotic iOS 8 launch, iCloud glitches, and the trouble with Continuity, I’ve gotten a bad feeling about Apple’s software quality management. “It Just Works”, the company’s pleasant-sounding motto, became an easy target, giving rise to jibes of “it just needs more work”.


The other view is that the quality lapses we observe are the beginning of a slide into satisfied mediocrity, into organizations and projects that “run themselves”, that are allowed to continue for political reasons without regard for the joy of customers.

I know what I hope for. I don’t expect perfection, I’ve lived inside several sausage factories and remember the smell. If Apple were to spend a year concentrating on solid fixes rather than releasing software that’s pushed out to fit a hardware schedule, that would show an ascent rather than a slide.

Update (2015-01-22): Rene Ritchie:

It’s arguable whether or not it’s any more pain than last year, the year before, the year before that, the year before that, and so on. But it’s inarguable that there’s been pain. People at Apple know that. They and their families and friends use the same hardware and software we do. Whether or not the right people were paying attention to the right measures, recent events have at the very least made even those who might not have realized the sentiment aware of it now.

Marco Arment, Casey Liss, and John Siracusa discuss Apple’s software quality on the Accidental Tech Podcast.

Dave Heinzel:

But for the past year or so, I have noticed so many little things that drive me absolutely insane, that I would actually be happy to jump ship, assuming there was another ship to jump to (there’s not, yet).

Last November, I started keeping a list of all the things that bug me. Some of them hinder productivity. Some are lost opportunities. They cover the full range of Apple products in my life: Mac, iPhone, iPad, Apple TV.

Lloyd Chambers:

Virtually all of these issues persist months after OS X Yosemite was released. And no doubt will never be fixed, or perhaps will be replaced by new bugs as Apple arbitrarily breaks things and rips out good useful features by increasingly disrespectful-to-users judgment. A few of these issues are very serious (security), and inexcusable.

Gus Mueller:

And that is probably the biggest difference between Apple and Microsoft. Apple knows when it’s time to show a new product. Apple knows when something is ready for real world use, and Apple won’t rush something out the door because of market pressures.


And I think that is why we’re seeing so many people reacting to Apple’s software quality lately. You expect Microsoft not to deliver. But we expect Apple to. And lately, it really hasn’t felt like they’ve been doing it.

Update (2015-01-23): Guy English:

If you’d like to know how the sausage is made, how people who have been in positions to make these quality control calls think, and get a sense of the camaraderie tune your compu-radios to Debug 60.

Update (2015-02-05): Nitin Ganatra hypothesized that the Apple bugs follow the Pareto principle, i.e. that there are probably about 8 P1 bugs that, if fixed, would address 80% of the problems people are seeing. I’m more inclined to Marco Arment’s viewpoint that the problem is more like 6,000 P2 bugs that have been building up because (as Ganatra and Don Melton described) there is never time in the schedule to fix them.

Update (2015-02-06): NitinGanatra:

Just listened to @atpfm 102. Good stuff! One thing I obviously forgot to mention. Perennial internal discussion: how many P2s = 1 P1

Overall quality is tied to state of P2s, and 6000 P2s is definitely felt by users. Agree with host who said that on @atpfm

John Siracusa:

@nitinganatra You didn’t say “X P1 bugs,” you just said “X bugs.”