How Apple Plans to Root Out Bugs
Mark Gurman (tweet, MacRumors, Reddit, Hacker News, ArsTechnica):
Instead of keeping engineers on a relentless annual schedule and cramming features into a single update, Apple will start focusing on the next two years of updates for its iPhone and iPad operating system, according to people familiar with the change. The company will continue to update its software annually, but internally engineers will have more discretion to push back features that aren’t as polished to the following year.
[…]
The shift is an admission of what many customers have already come to notice: Some Apple software has become prone to bugs and underdeveloped features. […] Apple has also recently released features later than it expected, as the rush to meet the annual deadline overtaxed engineers and created last-minute delays.
I’m not so sure the above is a new strategy so much as a tacit admission of what’s actually been going on the last few years.
Then why should we expect any improvement?
The idea of postponing features a year until they’re “ready” misses the whole point. It’s very difficult to find all the bugs in a major change until after you ship it. To get to a stable operating system, you need to spend at least a year just fixing bugs after a major release.
You can’t just consider the internal costs of annual updates. There are major external costs. Third party developers play an essential role in QA. If we never get the thing until June, and you ship every fall, never enough time to fix bugs.
A lot of people are pointing to Steven Sinofsky’s comments (Reddit). He makes some good points about the “broader context,” but I think he’s completely wrong about Apple’s software quality:
In any absolute sense the quality of Mac/iOS + h/w are at quality levels our industry has just not seen before. […] On any absolute scale number of bugs—non-working, data losing, hanging mistakes—in iOS/Mac is far far less today than ever before.
I don’t see how that can be taken seriously. He doesn’t have access to Apple’s bug database, so how would he know? I really doubt that the number of open bugs is lower than in the past, and even if it were there’s no reason to assume that Radar is representative of the actual number of bugs. He later says that the list of bugs is “infinitely long,” so this whole line of argument seems nonsensical. In what way is today’s Mac/iOS quality better in “any absolute sense” than in, say, 2010? He doesn’t say, except that more people are using it:
What is different is that at scale a bug that happens to 0.01% of people is a lot of people. A stadium full or more. […] No one ever anywhere has delivered a general purpose piece of S/W+H/W at scale of 1B delivering such a broad, robust, consistent experience. We don’t have a measure for what it means to be “high quality”.
Well, we can look at how many problems an individual user runs into. Is it higher or lower than before? This measure is independent of Apple’s scale. So is the circle of people I hear complaining. Apple’s customer base has doubled many times over, but the number of family members, friends, and customers that I communicate with has not. Now you could argue that maybe we have become exceptionally unlucky and are running into more than our share of issues, but I don’t find that very convincing.
He wants to discount the actual experiences of “many super smart/skilled people” because “the more a product is used the more hyper-sensitive people get to how it works.” What does that even mean? The number of hours in a day hasn’t increased; I don’t think my Mac/iPhone usage has increased much, if at all. Hardly anyone complains to me about the “slightest changes”; I hear about things that flat out don’t work. That’s not being hyper-sensitive.
Previously: Apple Delays Features to Focus on Reliability, Performance.
Update (2018-02-13): jarjoura:
As someone who used to work on iOS at Apple, what that company honestly needs is a culture not beholden to the whims of their EPMs (project managers). They used to help organize and work with engineering to schedule things across the company’s waterfall style development. However, by the time I left, they essentially took power over engineering. Radar became the driver for the entire company and instead of thinking about a holistic product, everything became a priority number. P0 meant, emergency fix immediately, P4 meant nice to have. You get the idea.
Nothing could be worked on if it wasn’t in Radar with a priority number attached and signed off by the teams’ EPM. No room for a side project or time away from your daily duties because there were always P1s to fix. If you didn’t personally have any left for the day, you’d take one from another engineer who was likely swamped with their own list of P1s.
[…]
This is how you get bugs in shipping software. EPMs driven to schedule things and over manage engineers would decide on a whim that something was a P2. That was basically always shelved to a follow-up .1 release.
Ultimately, engineers lost the freedom to decide when a feature was ready to ship.
This point about bug prioritization came up two years ago.
This is absolutely, 100% true, and jibes with my experience.
There was (don’t know if there still is), another really whacky problem with iOS work prioritization back then. Radar has P1, P2, P3, etc priorities. Milestones were arranged such that “No P3’s” happened (the point at which P3’s were no longer allowed to be worked on)… followed by “No P2’s” then finally “No P1’s.” At first glance that arrangement makes sense because it means the only bugs getting fixed late in the game are the really high priority ones. However, what it meant in practice was, if there was a P2 bug an engineer wanted to fix… They would scramble to make sure it gets fixed before the “No P2’s” milestone occurs…in effect, causing P2 bugs to be worked on before P1’s.
I’m a former iOS EPM (not speaking for Apple, obviously, since I don’t work there anymore), and although the Reddit commenter got the atmosphere of constant crisis right, he/she is misplacing the blame and misunderstanding the power dynamic. EPMs at Apple essentially have zero power over engineers’ workload. They take the list of stuff the engineering managers said they want to get done this year and say “You guys are crazy, you’ll never be able to do this without 3x the hours/manpower.” Then they proceed to drive the team as hard as necessary to make sure that they actually deliver what they said they were going to deliver. That’s it. The idea that there is this cabal of mighty EPMs twirling their mustaches and loading developers down with work is pretty far from reality.
It’s true that you shouldn’t be working on anything not in Radar (the bug tracker) but this is true anywhere you’ll work. Project managers however do not sign developers up for all those radars--on the contrary--we’re usually trying desperately to help you get rid of scope and get the task list down to what’s actually do-able!
One of the great things that IMHO sets Apple apart is how engineering-driven they are. I’ve never worked anywhere else where engineers had so much freedom to decide what they’re working on. The fact that they always decide to work on 3x what they can actually achieve is kind of on them. But that drive to try to do so much is part of what keeps innovation strong at Apple.
It sure looks like this is a case of the feedback loop working. The Apple community complains about software quality, the executive team reviews procedures and makes structural changes.
[…]
As an outsider, I think it’s hard to really assess whether these changes are meaningful rather than empty, ambitious, words. However, I’m glad the way it is portrayed in the Bloomberg report indicates it is a deeper shift of philosophy rather than a one-time focus for iOS 12 followed by a return to the status quo.
If the changes are as modest as this report makes them out to be, how much of an improvement can we realistically expect in software quality?
First Apple shareholder question is about software quality, which he says is “very unsatisfactory”. “We are getting plenty of changes but not many improvements... My solution has been to stop upgrading because I no longer trust Apple.” Apple is “losing touch with working people”
Update (2018-02-14): Riccardo Mori:
While I’m certain there are still underlying issues left unsolved in both Tiger and Leopard, in day-to-day general use, nothing prominent shows up on my radar. I turn on this PowerBook, it boots into Mac OS X 10.5.8, I open whatever apps I need for this session, and I feel I’m working in a stable, predictable environment. The only unfortunate thing I notice is that in places the hardware shows its age, or that certain features or services are too new to support this platform, but neither this particular vintage Mac nor its Mac OS X version are at fault. And it’s pretty amazing I’m still being productive with a 14-year old machine.
[…]
I’m just an outside observer, with perhaps the vantage point of having been using Apple hardware for almost 30 years. I can’t say with certainty that today both Mac OS and iOS have more bugs and issues than before. I’m also not saying that everything was 100% perfect before and now it’s all rubbish, because it’s not true. But from having extensively used (almost) each version of Mac OS and iOS, what I do notice is that behind the scenes there was a different approach to their development before a certain point in Mac OS X’s timeline, and that something changed (for the worse) after that point.
Update (2018-02-16): See also: Download.
Update (2018-02-21): See also: The Talk Show.
17 Comments RSS · Twitter
If you want to determine whether there are more bugs than there used to be, a simple question is: do all the things that worked before still work now?
The answer, for me at least, is no.
macOS has tons of examples of this. Look at the Calculator widget: it used to always remember the last value that was entered or calculated, but now when you close the Today panel for 10 seconds it forgets and consistently reverts back to whatever value happened to be in the widget when the user upgraded from Sierra to High Sierra (I kid you not). High Sierra has been released for months now and this simple known bug, duped in radar, still isn't fixed, and a lot of people use the Calculator widget.
Mail will randomly not allow you to Minimize it, with the option being greyed out until you make it full screen and then normal again. This never happened before, either.
The Touch Bar in Safari often stops taking up the full width of the Touch Bar if there are fewer than four Control Strip buttons, or starts showing doubled-up incorrect selection outlines around tabs that no longer exist. You'd think they'd at least focus on how the OS works the latest hardware features.
MacBook Pro user with a discrete GPU? You've had a fun time since High Sierra... (external display connectivity/sleep issues that are ongoing, 75% performance hit to Metal/OpenGL after letting your MBP go to sleep, graphical glitching and corruption, poor interface/scrolling performance despite this supposedly being something that would get better in High Sierra, etc)
If I (outside the US) ask Siri "when is thanksgiving", it reports that thanksgiving is today, always. It used to report the correct date, and this radar has also existed for months.
Meanwhile, on other devices...
Apple Watch Activity Achievements used to work, but now sometimes don't get awarded at all, or sometimes get awarded before you've achieved them. And don't get me started on the new Monthly Challenges - they've been broken since day one for me. Every month it's incorrectly calculating the total progress for the month by looking at last month's data, meaning it is either impossible to complete the challenge (because the totals never increase), or it'll incorrectly award it on the 1st of the month saying you've completed thousands of minutes or exercise already. This is a very widespread issue. Should it count as "more bugs" because it's a new feature? Because it breaks existing achievements, I'd say yes. The Ring in the New Year achievement was similarly incorrectly awarded early this year. These Monthly Activity Challenges were a headline feature of watchOS 4.
The Weather app on Apple Watch periodically shows a totally blank forecast display and you have to go to the top level cities view and back to fix it. This used to work, but now I hit this issue at least once a day.
The Control Center media control touch targets on iPhone SE-sized devices are offset horizontally so that when you tap one button it often taps the one next to it instead.
And the list goes on and on and on. In summary: even the basic features don't work reliably right now. So are there more bugs? Absolutely there are.
Why is all the quality improvement articles does not mention Swift 5.0? There is an on going rewrite with different library or components in Swift at Apple. And sometimes a rewrite doesn't necessarily guarantee it is better.
Swift 5.0 will finally have ABI stability, not sure how it plays into iOS 12 or macOS quality release.
"I’m not so sure the above is a new strategy so much as a tacit admission of what’s actually been going on the last few years."
I was about to say... allowing engineers to delay the shipping of features (but not OS releases) is less of a 'renewed focus on quality' and more of a 'we can no longer even pretend to keep up with our own cadence'.
When the annual release cycle started, we got buggy releases and new features. Today, we still have buggy releases, but now we have new features that don't ship until halfway through the release cycle. Again, this doesn't seem like a shift in focus so much as a description/confirmation of the decline we've been seeing...
I manage 20,000 Macs for a public school district. Ranging from 10.6 to 10.13. This site's author, and its commentators, have the worst case of the sky-is-falling that I've seen in a while.
Rose colored glasses and all I guess. Folks seem to forget all about the bugs in 10.6.3 or 10.4.8 or or or or.
I'm sorry that everything is so shitty for you. But for every day people doing every day things, well... in my experience, they're just not seeing things the way you all do. Their Macs work. They get things done and all is good.
I mean an entire paragraph on the calculator widget? I don't get it. /shrug
@Bryan I do appreciate hearing from people like you who are not running into problems. I find that more persuasive than what Sinofsky wrote.
"But for every day people doing every day things, well... in my experience, they're just not seeing things the way you all do. Their Macs work. They get things done and all is good."
Well, this does not explain why I always need to do sys admin tasks on the iMacs and iPhones in my family in order to deal with bugs in macOS and iOS.
My dad needed help with his Macs just last night....he's doing complex reinstalls to fix mysterious "bugs". Still trying to figure everything out. He's a professional developer. These are his work tools. It doesn't just work.
However, I will never claim Windows or Linux just works either....I've used all three; frankly, they all kind of suck in their own way.
[…] Commenting on a recent Bloomberg article by Mark Gurman, How Apple Plans to Root Out Bugs, Revamp iPhone Software, Michael Tsai references an otherwise insightful tweetstorm by Steven Sinofsky (a former President of the Windows Division at Microsoft), and shares a few critical observations: […]
When it comes to Apple software quality it's not just the bugs. There's been a trend where Apple takes a piece of software (Pages, Numbers, iPhoto/Photos, AirPort Utility, Disk Utility and more) that mostly works fine but maybe is showing it's age and they rip it apart and build it up from scratch again. Only, Apple never really finish the job and get the software back to the level where it used to be. So in the end there's an application that maybe looks prettier but lacks useful features that people were actually using. Or maybe the feature do come back eventually, but it takes unreasonable amount to time.
So, in addition to the question "Are users experiencing more bugs in their day-to-day usage of Macs now than before?" I'd like to add "Can users do everything they used to do with the current software?"
(This also ties in with the discussion of sandboxing and Mac App Store restrictions, because that has also caused useful features to disappear from third party apps.)
@Bryan "I mean an entire paragraph on the calculator widget? I don't get it. /shrug"
Yeah, I don't get it either. In fact it's totally absurd that I should have to say anything about it at all, because the calculator widget should simply work, and shouldn't keep reverting to the value 757.12 every time I close the Today panel! But it does! That's really annoying when you've been using it for years as a handy swipe-across place to add things up, swiping open Today, adding something to it, switching apps or tabs, then swiping open Today again to paste or enter the next value in. Except now that simple task, which has worked fine for years, is broken, because the value you entered just seconds ago is gone and replaced by a value you calculated four months ago! It's such a simple thing, but it's broken in High Sierra.
And this is a very simple "every day people doing every day things" example on the latest hardware, much like the trying to minimize Mail, or switching tabs on the Touch Bar in Safari, or asking Siri a question about the date of a major holiday, or checking the weather or your activity because you wear an Apple Watch, or wanting to pause or play music in the control center on an iPhone SE. These are not complicated tasks, they aren't "pro" use cases, they aren't niche functions.
I didn't bring up the more complicated bugs, or bugs with previous years' hardware, because I wanted to make it clear that even simple tasks on the latest products don't work properly in the current releases. But if you care to listen to the experiences of someone trying to do more complicated things, well, I could be here forever. God help you if you if you do anything involving professional graphics, web development or video editing on a Mac - you'll run into problems daily in High Sierra. Bugs in Metal acceleration and complex-shader performance regressions, bugs getting external monitors to stay connected (or having them work at all if they're 4K), bugs with monitors and GPU performance when resuming from sleep (Radeon GPUs), the "black screen" bug requiring restarts, problems with drivers causing memory/VRAM leaks requiring restarts (Nvidia GPUs), to name a few. Then there's the bugs in High Sierra that cause problems in third party apps, like Coda no longer having noncontiguous text layout due to a High Sierra bug (which means a huge performance hit and regular beachballing), various PDF rendering issues in many apps, or Adobe Illustrator no longer being able to use GPU acceleration because of High Sierra GPU driver bugs that cause all sorts of rendering problems (Adobe are currently recommending users turn off GPU acceleration and - it'd be funny if it weren't sad - sending Apple feedback about the bug), shader compilation hitching in OpenGL, etc.
I'm just a simple person trying to get work done. I can only tell you my experience and the experiences of those around me, and they haven't been good in the past six months. The sky isn't falling, but it sure is darker than it was.
Yep, lots of little everyday things are frustrating. Here are just a few things off the top of my head:
- Lately, Safari has decided to take 2 seconds to close a tab. Sometimes, not always.
- Holding down the Mission Control button while selecting a different window has been broken for years (in 10.13 it seems Apple just gave up and no longer highlights windows when the button is held down)
- Using the 'Application Windows' Exposé mode regularly causes random moves to completely different Spaces, sometimes activating a completely different app
- Clicking an iOS app link in Safari will cause iTunes to appear (auto-shifting to a different Space if necessary), only to tell me the app can only be downloaded on my iOS device
- The Mac App Store continues to have 5+ second latency when showing progress for app updates, and even then, it displays incorrect information and button labels.
- The Upgrade All button on the Mac App Store has a ~70% chance of not actually updating all the apps in the list, but nevertheless removes the entries for the apps it missed, requiring switching tabs to (hopefully) have them appear again
- The Mac App Store regularly shows an incorrect number of available updates, takes multiples quit/relaunch cycles to show OS updates, and then after upgrading, will sometimes prompt for the installation of a *previous* OS version for several minutes until it realizes what OS version is on the system
- Renaming a file via the title bar only works about 50% of the time, and if it doesn't work, it will continue not working so long as that window is open
- When deleting files on my iPhone via Image Capture, there's a 20% chance it'll crash with an index-out-of-bounds exception on an array. I am then unable to access my camera roll in Image Capture or on my phone until I unplug/replug it.
- When using scanners with Image Capture, retention of filenames/settings between captures is completely inconsistent. Sometimes certain scanning options will just be missing from the drop-down menu.
- Final Cut / Motion GPU processing is bottlenecked on modern Nvidia GPUs... maybe a 10th of the GPU is actually getting utilized for exports
- Final Cut does not restore windows to their original Spaces... it just plops them in the active Space
- Quitting and relaunching an app does not restore windows to their original Spaces... they just get plopped in the active Space
- Mail.app lost track of all my local mailboxes a release or so ago... never again
- Mouse cursor doesn't properly reflect current input type
- Grabbing the corner of a window to resize it, with the resize cursor showing, will move the window instead
- When copying photos off my phone via Image Capture, sometimes a photo or two will get written as a blank 0-byte file. This is especially insidious because the thumbnail looks fine in Finder, but once you open it, oops, turns out nothing's there.
- Signing into iCloud while upgrading to 10.13 deleted all my local calendars, contacts, and bookmarks
Those last two are bona fide data-loss bugs. All of the above bugs disrupt flow and get in the way of 'just getting work done'. Also, it's worth mentioning it only took me a few minutes to come up with the items on this list.
Combine all this with Apple not updating "important" Mac product lines for years on end (while charging the same price for them the whole time), and yeah, it's no wonder I'm questioning the viability of the Mac platform.
I just got a new MacBook Pro from work, one of the touch bar devices. I've had this for two weeks.
- The battery indicator worked correctly exactly once; now my MacBook consistently just shuts down at 20% battery; it doesn't sleep, it just shuts down completely.
- Rearranging Spaces no longer works. I drag a space to some other place; five minutes later, I scroll through my spaces, and it has moved back.
- I also now have a problem where dragging a file on a folder in the Finder doesn't work consistently, and I have to point at a specific location of the folder to make it highlight.
- When I drag an image form Safari to the Finder, the cursor often doesn't change to the "add image" cursor, and the file is only created half the time.
That's just off the top of my head. This is a fresh install of OS X on a device that's two weeks old.
@Lukas "Rearranging Spaces no longer works. I drag a space to some other place; five minutes later, I scroll through my spaces, and it has moved back."
Regarding this, I ran into the same issue and put up with it for a long time before I went searching and found there's a setting in System Preferences > Mission Control, "Automatically rearrange Spaces based on most recent use", which is on by default. I must have turned this off years ago and it was only when I bought a new MacBook Pro and started from scratch did I therefore get the default setting back. Having Spaces automatically rearrange by default seems like a terrible idea, because it destroys the spatial memory association that otherwise naturally makes it fast to use, but I can also understand why it would be set that way.
If that setting isn't your issue, well... it's just another day on High Sierra I guess.
> I went searching and found there's a setting in System Preferences > Mission Control,
> "Automatically rearrange Spaces based on most recent use", which is on by default
Argh! Yes, that's it! I must have turned it off years ago on my old Mac, and forgotten that it even exists. Yeah, auto-rearranging spaces is weird to begin with, because the visual metaphor is that they're sitting side-by-side, and you're moving horizontally through them, so it doesn't make sense that they would magically appear somewhere else than you left them, but it seems doubly absurd to me that Apple allows you to manually rearrange spaces even when that "feature" is turned on, and any manual changes are pretty much immediately discarded.
At any rate, thanks for pointing this out! Not a bug, then, just a UX failure, I guess.
Just a few weeks ago I kept getting confused when I was swiping between full-screen apps. Then I went digging and found that System Preferences setting . Maybe the rationale is that the cmd+tab app switcher is in most-recently-used order, so full-screen apps rearrange the same way? In any case, it feels like the wrong default behavior to me.
