{"id":27335,"date":"2019-11-21T11:15:16","date_gmt":"2019-11-21T16:15:16","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=27335"},"modified":"2019-12-20T13:45:26","modified_gmt":"2019-12-20T18:45:26","slug":"full-steam-ahead-but-with-feature-flags","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2019\/11\/21\/full-steam-ahead-but-with-feature-flags\/","title":{"rendered":"Full Steam Ahead, But With Feature Flags"},"content":{"rendered":"<p><a href=\"https:\/\/www.bloomberg.com\/news\/articles\/2019-11-21\/apple-ios-14-features-changes-testing-after-ios-13-bugs\">Mark Gurman<\/a> (<a href=\"https:\/\/twitter.com\/markgurman\/status\/1197515472851226625\">tweet<\/a>, <a href=\"https:\/\/news.ycombinator.com\/item?id=21593307\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/www.bloomberg.com\/news\/articles\/2019-11-21\/apple-ios-14-features-changes-testing-after-ios-13-bugs\">\n<p>Apple Inc. is overhauling how it tests software after a swarm of bugs marred the latest iPhone and iPad operating systems, according to people familiar with the shift.<\/p>\n<p>Software chief Craig Federighi and lieutenants including Stacey Lysol announced the changes at a recent internal &ldquo;kickoff&rdquo; meeting with the company&rsquo;s software developers. The new approach calls for Apple&rsquo;s development teams to ensure that test versions, known as &ldquo;daily builds,&rdquo; of future software updates disable unfinished or buggy features by default. Testers will then have the option to selectively enable those features, via a new internal process and settings menu dubbed Flags, allowing them to isolate the impact of each individual addition on the system.<\/p>\n<p>When the company&rsquo;s iOS 13 was released alongside the iPhone 11 in September, iPhone owners and app developers were confronted with a litany of software glitches. [&#8230;] This amounted to one of the most troubled and unpolished operating system updates in Apple&rsquo;s history.<\/p>\n<p>[&#8230;]<\/p>\n<p>Test software got so crammed with changes at different stages of development that the devices often became difficult to use. Because of this, some &ldquo;testers would go days without a livable build, so they wouldn&rsquo;t really have a handle on what&rsquo;s working and not working,&rdquo; the person said.<\/p>\n<p>[&#8230;]<\/p>\n<p>Still, iOS 14 is expected to rival iOS 13 in the breadth of its new capabilities, the people familiar with Apple&rsquo;s plans said.<\/p>\n<\/blockquote>\n\n<p>It sounds like they are still in the denial. Feature flags may be a useful tool to help with testing, but much more drastic changes are needed. They don&rsquo;t seem to have much interest in reducing the scope of major releases, so I would like to see them drop the annual release schedule. And, above all, make an internal commitment to quality.<\/p>\n\n<blockquote cite=\"https:\/\/www.bloomberg.com\/news\/articles\/2019-11-21\/apple-ios-14-features-changes-testing-after-ios-13-bugs\">\n<p>The testing shift will apply to all of Apple&rsquo;s operating systems, including iPadOS, watchOS, macOS and tvOS. The latest Mac computer operating system, macOS Catalina, has also manifested bugs such as incompatibility with many apps and missing messages in Mail.<\/p>\n<\/blockquote>\n\n<p>The missing Mail messages bug <a href=\"https:\/\/mjtsai.com\/blog\/2019\/10\/11\/mail-data-loss-in-macos-10-15\/#comment-3111509\">remains unfixed<\/a> in macOS 10.15.2 betas. This is the buggiest Mail release I can recall. I&rsquo;m still busy working around Catalina bugs throughout the system.<\/p>\n\n<blockquote cite=\"https:\/\/www.bloomberg.com\/news\/articles\/2019-11-21\/apple-ios-14-features-changes-testing-after-ios-13-bugs\">\n<p>Apple privately considered iOS 13.1 the &ldquo;actual public release&rdquo; with a quality level matching iOS 12. The company expected only die-hard Apple fans to load iOS 13.0 onto their phones.<\/p>\n<\/blockquote>\n\n<p>And yet customers were automatically prompted to update to 13.0, and even <a href=\"https:\/\/mjtsai.com\/blog\/2019\/10\/30\/ios-13-2-killing-background-apps-more\/\">13.2 introduced major problems<\/a>.<\/p>\n\n<p><a href=\"https:\/\/twitter.com\/steipete\/status\/1197499226948804609\">Peter Steinberger<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/steipete\/status\/1197499226948804609\">\n<p>Feature flags in teat releases are Apple&rsquo;s answer to the software quality issue? \nWhat about automated testing? And opening up hiring outside of Cupertino, to deal with the amount of radars and missing documentation.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/drunknbass\/status\/1197521874772606976\">ss&#x250;qu&#x29E;un&#x279;p<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/drunknbass\/status\/1197521874772606976\">\n<p>I can tell you from experience that these &ldquo;feature flags&rdquo; carry a lot of tech debt that these managers don&rsquo;t seem to understand. This may be the canary in the coal mine.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/michaeldupuis\/status\/1197500940565266432\">Michael Dupuis<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/michaeldupuis\/status\/1197500940565266432\">\n<p>How about slowing things DOWN? It&rsquo;s very much a feeling that they are just throwing things over the fence as fast as they can, and it shows in the horrible quality we&rsquo;ve been seeing...<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/Freerunnering\/status\/1197512430370332675\">Kyle Howells<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/Freerunnering\/status\/1197512430370332675\">\n<p>Adding feature flags to betas isn&rsquo;t the answer. It&rsquo;ll just add more work.<\/p>\n<p>Keeping the same process but adding extra steps doesn&rsquo;t generally work.<\/p>\n<p>They need to slow down, only release software when it&rsquo;s ready, and prioritise quality, documentation and fixing bugs.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/lapcatsoftware\/status\/1197525680466120705\">Jeff Johnson<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/lapcatsoftware\/status\/1197525680466120705\">\n<p>Annual OS releases are also destroying third-party software quality. We can&rsquo;t keep up with the constant churn, and the tools are never stable. We waste so much time every year just dealing with Apple&rsquo;s shit.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/lapcatsoftware\/status\/1197530304430399489\">Jeff Johnson<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/lapcatsoftware\/status\/1197530304430399489\">\n<p>Apple&rsquo;s software quality problems can&rsquo;t be solved in iOS 14. They&rsquo;ve accumulated at least 5 years of technical debt, if not more, from annual releases.<\/p>\n<p>They&rsquo;re deep down in a hole. Desperately in need of years without a major update.<\/p>\n<\/blockquote>\n\n<p>Previously:<\/p>\n<ul>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/11\/07\/apple-notes-and-icloud-syncing\/\">Apple Notes and iCloud Syncing<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/10\/30\/ios-13-2-killing-background-apps-more\/\">iOS 13.2 Killing Background Apps More<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/10\/16\/catalina-vista\/\">Catalina Vista<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/10\/15\/catalina-system-issues\/\">Catalina System Issues<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/10\/11\/mail-data-loss-in-macos-10-15\/\">Mail Data Loss in macOS 10.15<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/09\/25\/ios-13-and-13-1\/\">iOS 13 and 13.1<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/09\/25\/ios-13-full-access-keyboard-extensions-bug\/\">iOS 13 Full Access Keyboard Extensions Bug<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/09\/03\/icloud-drive-features-removedpostponed\/\">iCloud Drive Features Removed\/Postponed<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/08\/28\/ios-13-1-developer-beta-1-is-already-out\/\">iOS 13.1 Developer Beta 1 Is Already Out<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/03\/11\/the-sad-state-of-logging-bugs-for-apple\/\">The Sad State of Logging Bugs for Apple<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2018\/02\/13\/how-apple-plans-to-root-out-bugs\/\">How Apple Plans to Root Out Bugs<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2016\/02\/07\/mossberg-discovers-the-functional-high-ground\/\">Mossberg Discovers the Functional High Ground<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2015\/01\/06\/apples-software-quality-continued\/\">Apple&rsquo;s Software Quality, Continued<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2015\/05\/06\/discoveryd-is-still-buggy\/\">discoveryd Is Still Buggy<\/a><\/li>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2014\/10\/11\/apples-software-quality-decline\/\">Apple&rsquo;s Software Quality Decline<\/a><\/li>\n<\/ul>\n\n<p id=\"full-steam-ahead-but-with-feature-flags-update-2019-11-26\">Update (2019-11-26): <a href=\"https:\/\/twitter.com\/markgurman\/status\/1197545513895120896\">Mark Gurman<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/markgurman\/status\/1197545513895120896\">\n<p>iOS 13 has had 8 updates in its first two months, the most in that same period since Craig Federighi took over development with iOS 7. See chart.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/sanguish\/status\/1197575237107421189\">Scott Anguish<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/sanguish\/status\/1197575237107421189\">\n<p>It&rsquo;s a myth that Apple doesn&rsquo;t have remote writers.  They have an entire department in Seattle.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/tclementdev\/status\/1197578951931105280\">Thomas Clement<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/tclementdev\/status\/1197578951931105280\">\n<p>And this only works if Apple can detect <em>before shipping<\/em> that a feature is broken enough that it needs to be turned off.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/drdrang\/status\/1197595309150539783\">Dr. Drang<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/drdrang\/status\/1197595309150539783\">\n<p>An old saying from the making of physical products seems apropos: you can&rsquo;t inspect quality into a product.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/cdfinder\/status\/1197820651920396289\">Norbert M. Doerner<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/cdfinder\/status\/1197820651920396289\">\n<p>They need a massive OS release moratorium, and look at what they have done, and why that failed. Then start fixing the bugs, and change the crazy yearly release cycle, it is utter madness #Apple #StartFixingTheBugs<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/lapcatsoftware\/status\/1198300810087714816\">Jeff Johnson<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/lapcatsoftware\/status\/1198300810087714816\">\n<p>Months since previous Mac .0 release:<\/p>\n<p>10.1.0  6<br \/>\n10.2.0  11<br \/>\n10.3.0  14<br \/>\n10.4.0  18<br \/>\n10.5.0  18<br \/>\n10.6.0  22<br \/>\n10.7.0  23<br \/>\n10.8.0  12<br \/>\n10.9.0  15<br \/>\n10.10.0 12<br \/>\n10.11.0 11<br \/>\n10.12.0 12<br \/>\n10.13.0 12<br \/>\n10.14.0 12<br \/>\n10.15.0 12<\/p>\n<p>A sensible progression... until 10.8<\/p>\n<p>(Note that Steve died after 10.7)<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/gcaprio\/status\/1198785836021624832\">Griffin Caprio<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/gcaprio\/status\/1198785836021624832\"><p>Anyone who&rsquo;s built even a moderately complex app knows you can&rsquo;t just pepper in if\/else statements and iOS is more than moderately complex.<\/p><\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/McCarron\/status\/1198848771913859072\">Patrick McCarron<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/McCarron\/status\/1198848771913859072\">\n<p>The amount of technical debt those flags carry are no joke. Not always a clean removal either.<\/p>\n<\/blockquote>\n\n<p id=\"full-steam-ahead-but-with-feature-flags-update-2019-12-20\">Update (2019-12-20): See also: <a href=\"https:\/\/www.macrumors.com\/2019\/11\/21\/apple-software-development-changes-report\/\">MacRumors<\/a>, <a href=\"https:\/\/www.macworld.com\/article\/3455224\/software-bugs-have-become-apples-greatest-vulnerability.html\">Macworld<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>Mark Gurman (tweet, Hacker News): Apple Inc. is overhauling how it tests software after a swarm of bugs marred the latest iPhone and iPad operating systems, according to people familiar with the shift. Software chief Craig Federighi and lieutenants including Stacey Lysol announced the changes at a recent internal &ldquo;kickoff&rdquo; meeting with the company&rsquo;s software [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2019-11-21T16:15:25Z","apple_news_api_id":"0b1a8ed8-d038-4b61-83a1-93bdada01493","apple_news_api_modified_at":"2019-11-26T18:55:30Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAAAQ==","apple_news_api_share_url":"https:\/\/apple.news\/ACxqO2NA4S2GDoZO9raAUkw","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[38,126,1143,131,629,31,1667,1837,30,1666],"class_list":["post-27335","post","type-post","status-publish","format-standard","hentry","category-technology","tag-apple","tag-applemail","tag-apple-software-quality","tag-bug","tag-craig-federighi","tag-ios","tag-ios-13","tag-ios-14","tag-mac","tag-macos-10-15"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/27335","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=27335"}],"version-history":[{"count":4,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/27335\/revisions"}],"predecessor-version":[{"id":27642,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/27335\/revisions\/27642"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=27335"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=27335"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=27335"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}