{"id":24566,"date":"2019-03-11T16:33:20","date_gmt":"2019-03-11T20:33:20","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=24566"},"modified":"2021-09-06T10:18:32","modified_gmt":"2021-09-06T14:18:32","slug":"the-sad-state-of-logging-bugs-for-apple","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2019\/03\/11\/the-sad-state-of-logging-bugs-for-apple\/","title":{"rendered":"The Sad State of Logging Bugs for Apple"},"content":{"rendered":"

Corbin Dunn<\/a> (tweet<\/a>, Hacker News<\/a>):<\/p>\n

\n

This is where things get screwy depending on the component your bug lands in, since bug management is group dependent. Many groups will have only one or two QA people to do the initial screening of those large drop areas for bugs. QA engineers are sometimes instructed to screen bugs with a priority and “fix period” before passing them off to the engineer responsible for the code. This is terrible because many engineers will not look at bugs with a low priority. It is much better for the engineer who “owns the code” to look at a bug and determine the priority. The QA engineers will frequently get a huge back log of bugs to screen, and it can take weeks, or even months, for some bugs to get screened. Sometimes this leads to a mass screening of bugs, marking them all with a low priority. Bug originators have to notice this, and complain about it for the priority to get increased. Worse yet, some groups mass close bugs older than a year or so, and ask the originator to re-open the bug if the issue still exists. A lot of people don’t pay attention to bugs that need verification, and they simply become lost.<\/p>\n

[…]<\/p>\n

Engineers also dislike screening bugs because sometimes they have to add them to their queue for the current release. This increases their required workload for that release, which is something people don’t like doing. So, instead, many bugs stay unscreened.<\/p>\n

[…]<\/p>\n

Sometimes QA screen bugs with a low priority and holds onto them. They never get moved to the appropriate code engineers, and effectively become lost in the system. Sadly, I had seen this happen way too often.<\/p>\n

[…]<\/p>\n

When a bug is sent back as fixed, the internal developer who originated the bug is supposed to verify the problem is resolved. They can send it back if the problem isn’t resolved. However, internal developers don’t really have an incentive to verify bugs. Management doesn’t keep track of bugs that need verification or really require developers to verify them. Most engineers do verify bugs; they like to make sure problems are resolved. But external developers are left in a more sad state. The bug becomes closed for them, and is dead.<\/p>\n

[…]<\/p>\n

Internal engineers need to take more responsibility in promptly screening bugs. Management needs to allow engineers to have more time to do this, which is at the expense of working on features or fixing already screened bugs. Engineers should always be expected to have a very low unscreened bug count.<\/p><\/blockquote>\n\n

This matches what we felt like must be going on when filing bugs, as well as the way the smaller bugs seem to hang around forever, with new ones added each year. Even Mojave, which was supposed to be a refinement release, seems to have, on balance, increased the number of bugs. As a user, it sucks that things don’t work as well as they used to. As a developer, I spend too much time working around OS bugs and breakage<\/a>—in other words, preventing my apps from getting worse rather than actually making them better. I assume other developers are in the same boat, and this may be one reason there seems to be less excitement around apps these days. Everyone is wasting a lot of energy treading water.<\/p>\n\n

It’s as if the OS is rotting away before our eyes. The good news is that this should be fixable. Apple has tons of smart engineers who care. But the process does not seem to be set up to produce quality. Management talks a good game but clearly has other priorities. There are undoubtedly many policies that could be changed to improve the organizational incentives, and a core problem seems to be that Apple remains understaffed for its ambitions. The headcount can’t and shouldn’t be massively increased in a short period of time, but there is something Apple could do today to help stem the tide: go off the annual schedule.<\/p>\n\n

Peter Ammon<\/a>:<\/p>\n

\n

The single easiest and most effective thing Apple could do to improve its SWE org is to invest in Radar.<\/p>\n

Radar’s importance within Apple cannot be overstated. It subsumes what would be multiple tools in other orgs. As an Apple SWE you spend a massive amount of time in it. And yet Apple treats Radar as a cost center, developed by an outsourced offshore team. It’s slow to search, supports only plain text, is hard to script, and is missing obvious features, e.g. automatic duplicate finding.<\/p>\n

Hire five good SWEs, give them a mandate to serve the needs of the org, and you will massively increase the effectiveness of every other engineer.<\/p>\n<\/blockquote>\n\n

ThemalSpan<\/a>:<\/p>\n

\n

Anecdotally, I didn’t find the situation internally to be much better. Many bugs internally go unanswered because there is not enough time allocated to fixing core systems and designing better replacements. The truth is, I know personally of several teams that aren’t able to get through the queue of internally filed and scheduled bugs.<\/p>\n

To me, it feels like Apple hasn’t resourced core pieces of infrastructure and engineering teams in line with upper management’s plans for growth. While many teams are relatively sequestered, once you start talking to folks elsewhere in the company it becomes clear that many teams are struggling to stay above water. More still, everyone shrugs about it because it’s not clear exactly what is wrong. The best description I’ve heard is in many cases engineers are willing to offer hacks as a solution to meet management’s demands, and management is either willing to accept those hacks or doesn’t know better.<\/p>\n<\/blockquote>\n\n

satisfice<\/a>:<\/p>\n

We originally designed Radar so that bugs would be verified as closed by the person with most interest in seeing this happen: the tester assigned to that part of that project. Then management swooped in with an edict that bugs must be verified as closed by whomever originally reported them. This is a stupid idea, because it creates the perverse incentive that no one should report a problem if they are outside the team (because then you are committing to verify the fix, which just means more work for you that has nothing to do with any of your main responsibilities).<\/p>

When I pointed out that the system would now discourage people on different teams from helping each other, the sponsoring director said “that’s what pink slips are for.” Direct quote. Soon after that I resigned from the design team.<\/p>

Without reasonably skilled and principled leadership, you just don’t get quality software. And “quality is everyone’s job” is just an empty and childish slogan. Excellence is not transmitted through slogans and wishful thinking. You have to assign responsibility, provide resources and time (which means lowering velocity of new development), and follow-up.<\/p>

The fundamental reason why it doesn’t happen is the technology market is not efficient. Quality is, in fact, not as important as career testers wish it were. You can get away with doing terrible work and not lose your job. The fact that Apple pays no significant penalties for having buggy products insulates it from our slings and arrows.<\/p><\/blockquote>\n\n

Corbin Dunn<\/a>:<\/p>\n

Some obvious issue, like “this button should do X but it does Y” can be verified by almost anyone. But some issues need the attention of the original author to really verify the bug. Maybe what needs to be done is someone in QA needs to attempt to verify the bug, or “pre-verify”, and then it goes back to the originator for final verification, who can also verify it, or simply close it if they feel like QA did a good job.<\/p><\/blockquote>\n\n

drfindley<\/a>:<\/p>\n

\n

What’s even sadder is we used to be better at this when I started at Apple in 2008. Bugs often got screened and triaged and sometimes fixed within a week. I blame the yearly release schedule, where shipping features became a higher priority than overall quality<\/p>\n<\/blockquote>\n\n

Corbin Dunn<\/a>:<\/p>\n

\n

I feel the same way; people took more time on bugs back in those days. I also think the yearly schedule is to blame.<\/p>\n<\/blockquote>\n\n

akecheck<\/a>:<\/p>\n

\n

When a process is annoying and you do nothing, people eventually do<\/i> give up and leave. When it reaches that point, they’re not coming back<\/i> even if you finally wake up and fix what bugged them.<\/p>\n

Apple software quality is in serious danger precisely because of this type of community and infrastructure rot. They are not encouraging developers to help them, and a not-surprising number of serious issues have shown up in released products in recent years.<\/p>\n<\/blockquote>\n\n

jakobegger<\/a>:<\/p>\n

\n

By now, a significant fraction of bugs are bugs in Apple’s frameworks. We try to report them to Apple, but they are ignored, or simply closed because they are related to deprecated APIs.<\/p>\n

Of course, customers don’t complain that Apple frameworks are buggy -- they complain that our app crashes! So Apple has no incentive to fix it.<\/p>\n<\/blockquote>\n\n

Peter Kamb<\/a>:<\/p>\n

\n

The entire value of WWDC is going to the Labs, giving an Apple engineer your Radar number, and having them read you or paraphrase the internal-only notes attached to the ticket. Half the time the question\/bug will be clearly resolved internally or a workaround posted. But no updates are added to the public ticket, and it will remain open and unchanged for years.<\/p>\n<\/blockquote>\n\n

Gus Mueller<\/a>:<\/p>\n

\n

Corbin is a former AppKit engineer, and this is a must read for developers. It’ll make you angry, and it’s stuff you already figured was happening.<\/p>\n<\/blockquote>\n\n

Tanner Bennett<\/a>:<\/p>\n

\n

This confirms what we already knew. Almost no one at Apple takes bug reporting seriously. Reports will stack up indefinitely and eventually macOS will be a shell of its former self.<\/p>\n<\/blockquote>\n\n

Corbin Dunn<\/a>:<\/p>\n

\n

It is not just macOS, but iOS too.<\/p>\n<\/blockquote>\n\n

Paul Haddad<\/a>:<\/p>\n

\n

Interesting read. My view on bugs, work around them and move on. Even if it gets assigned it’s not getting fixed for at least a year.<\/p>\n<\/blockquote>\n\n

Jeff Johnson<\/a>:<\/p>\n

\n

IMO Radar screening issues are merely a symptom. The root problem is that Apple produces a completely unmanageable volume of bugs. Even if they screened all Radars quickly, then what? Bugs still get written much much faster than they get fixed. That’s unsustainable.<\/p>\n

I suspect that Radar screening is allowed to be lax precisely because everybody knows that a huge volume of bugs will never get fixed anyway. It’s like bailing out the Titanic.<\/p>\n

There’s also a tolerance for shipping bad bugs. If heads rolled at the company for shipping bad bugs, then Radars would get screened.<\/p>\n<\/blockquote>\n\n

Adam Savage<\/a>:<\/p>\n

\n

Just getting my music downloaded to my phone is a recurring nightmare I relive every time I upgrade. Having my music ON my device should be a simple choice, & you’ve made it Byzantine. How is it that I have to visit a support forum to learn how to download SONGS to my PHONE?!<\/p>\n

The language of permissions is still fascinatingly and infuriatingly opaque, to the degree that when using iTunes, I’m regularly convinced it has an agenda antithetical to mine. Searching Suport forums is also nightmarish as helpful buttons from one version disappear in others.<\/p>\n<\/blockquote>\n\n

Previously:<\/p>\n