David Pogue:
The PR person could switch playback from one speaker to the other without missing a beat. They even had a halo light rigged to turn on behind whichever speaker was playing, so you’d know which was which.
There was not a shred of doubt: In this side-by-side comparison, the HomePod sounded better than its competitors.
[…]
I hid the four speakers behind a curtain — a sheet of thin, sheer fabric that wouldn’t affect the sound. It took me a Sunday to figure out how to get the A/B/C/D switching to work seamlessly, but I finally managed it: All four speakers would be streaming from Spotify, all four over Wi-Fi. I’d use the Spotify app’s device switcher to hop among speakers without missing a beat.
[…]
For each song, I played the speakers in a different order (A to D sometimes, D to A sometimes). […] They held up their signs. Two of them ranked the Google Home Max (“D”) as the best. Three of them ranked the Sonos One (“A”) the best.
Nobody ranked the HomePod the best.
[…]
As far as I can tell, none of the other critics who declared HomePod No. 1 actually set up their own blind A/B/C/D tests. Maybe their conclusions wouldn’t have been so emphatic if they had.
Consumer Reports also rated HomePod lower (via MacRumors):
Overall the sound of the HomePod was a bit muddy compared with what the Sonos One and Google Home Max delivered.
All three of these speakers were impressive compared with other smart speakers we’ve tested, but they fall significantly short of our highest-rated wireless speakers, such as the Edifier S1000DB, $350, which earned an Excellent sound-quality rating.
It’s not clear to me whether they did a blind test.
Other reviewers continue to give HomePod high marks for sound. WinterCharm (ArsTechnica, Hacker News):
I am speechless. The HomePod actually sounds better than the KEF X300A. If you’re new to the Audiophile world, KEF is a very well respected and much loved speaker company. I actually deleted my very first measurements and re-checked everything because they were so good, I thought I’d made an error. Apple has managed to extract peak performance from a pint sized speaker, a feat that deserves a standing ovation. The HomePod is 100% an Audiophile grade Speaker.
See also: Jim Dalrymple.
Previously: HomePod Reviews.
Update (2018-02-14): WinterCharm:
In my review, I made a massive caveat that many news outlets when they picked this up, seemed to forget.
IN AN UNTREATED ROOM The HomePods are pushing better sound than a single X300A, as measured.
That’s an impressive feat, I was impressed by it. but my conclusion was that obviously in a properly treated room, correctly set up speakers would be better. This is why I said that the product was great for the masses, but for audiophiles, you would be better off putting these in places like your kitchen and leaving your normal listening setup intact.
Unfortunately, as often happens with the news, they skim and summarize, in a way that some of the subtlety and conditions around which my main point rested are lost in translation. When you take 5000 words and turn it into 1 headline and a 250 word article, some stuff gets lost in translation.
Second, half the news outlets were Apple news sites who have a huge pro Apple bias. They picked it up… and it’s in their best interest to misrepresent or cherry pick the review, exaggerating the claims.
Kirk McElhearn:
The much touted review of the HomePod posted by an “audiophile” on Reddit last week – and gleefully tweeted by Apple’s Phil Schiller – turns out to be a long mess of uninformed and poorly made measurements.
This reply on Reddit highlights many of the problems, notably the fact that the HomePod wasn’t measure in an anechoic room, but mainly the fact that the “reviewer” fudged the display of his graphs, making them look better than they were.
Mark Sullivan (MacRumors):
We’ve heard plenty of opinions on the HomePod’s general sound quality, so it’s a good time to measure the consistency of the HomePod’s sound distribution using some professional-grade acoustic analysis tools.
Update (2018-02-15): Kirk McElhearn:
I’ve been following this Reddit thread and its published results. It’s amazing that in a world of audiophiles who obsess over which USB cable makes their music sound better, that this person performed all of these measurements, and forgot to mention that the HomePod uses digital signal processing to alter all music that it plays. In other words, it is far from neutral, and audiophiles make a big deal about their equipment being neutral. The frequency response may be excellent, but the equalization alters the music from what it should sound like.
In fact, I think it’s highly possible that this reviewer has based the conclusions of his testing on false assumptions. The HomePod has dynamic digital signal processing; it alters the music based on the music. In other words, it’s not a fixed EQ setting, but one that changes as music is played (and according to the room where it’s played). As such, sending single frequency sine waves, or whatever he did, won’t show the results of the EQ.
Update (2018-02-20): David Pogue:
“Since the HomePod adjusts its sound to the acoustics of the room, you should not have used a piece of fabric to hide the speakers,” wrote @markbooth and others. “The fabric may have affected the HomePod’s sound.”
Well, no. The HomePod re-samples its listening position after each time it’s moved, during the first few seconds of music playback. We let the HomePod do its room listening before hanging the curtain, so it had already had the chance to adjust its sound.
Jean-Louis Gassée (Hacker News):
This is where we find a new type of difficulty when evaluating this new breed of smart speakers, and why we must be kind to the early HomePod reviewers: The technical complexity and environmental subjectivity leads to contradictory statements and inconsistent results.
Amazon Echo Audio Google Home Max HomePod Sonos Spotify
Felix Krause (tweet, Hacker News):
Any Mac app, sandboxed or not sandboxed can:
- Take screenshots of your Mac silently without you knowing
- Access every pixel, even if the Mac app is in the background
- Use basic OCR software to read the text on the screen
- Access all connected monitors
Jeff Johnson:
Nobody tell Felix that Mac apps can also read the clipboard.
This is why I think a network blocker like Little Snitch is more important for protecting users than the sandbox. Anyway, this is not really news, but it prompted some interesting comments from former Apple engineer Peter Ammon:
We did our best but the fact is that sandboxed apps run more slowly, have fewer features, are more isolated, and take longer to develop. Sometimes this cost is prohibitive (see Coda 2.5).
IMO the app sandbox was a grievous strategic mistake for the Mac. Cocoa-based Mac apps are rapidly being eaten by web apps and Electron psuedo-desktop apps. For Mac apps to survive, they must capitalize on their strengths: superior performance, better system integration, better dev experience, more features, and higher general quality.
But the app sandbox strikes at all of those. In return it offers security inferior to a web app, as this post illustrates. The price is far too high and the benefits too little.
IMO Apple should drop the Mac app sandbox altogether (though continue to sandbox system services, which is totally sensible, and maybe retain something geared towards browsers.) The code signing requirements and dev cert revocation, which has been successfully used to remotely disable malware, will be sufficient security: the Mac community is good at sussing out bad actors. But force Mac devs to castrate their apps even more, and there won’t be anything left to protect.
I still think the idea of sandboxing makes sense, but the actual implementation of it—the available entitlements, the framework bugs, the lack of documentation, and the App Store policies—were botched. And there has been little visible progress since macOS 10.7. Is this because it’s fundamentally not possible to do better, given that the Mac wasn’t designed with sandboxing in mind? Or has it simply not been a priority for Apple?
Peter Ammon:
It’s a hard UI problem. The Mac sandbox overcorrects to requiring capability resources for all file accesses, while on the other extreme we have e.g. Windows UAC which trains users to roll their eyes and click through.
But Apple doesn’t enjoy the luxury of solving this problem in a nuanced way, because Mac apps are not acting from a position of strength. I suspect you aren’t downloading lots of Mac apps today, and the reason is not insufficient sandboxing, but instead the limited selection, annoying install experience, etc. These are the problems that Apple must fix first.
[…]
Instead Apple should leverage the Mac’s unique software strengths. Aggressively evolve the Mac’s unique “UI vocabulary” and application frameworks. Empower, not punish, the dedicated and passionate developer community. Ship love to the userbase (perhaps the only one in existence) that’s willing to open their wallets for high-quality desktop software. And yes, tolerate web-tech apps too - but embarrass them!
Peter Ammon:
The theory of the Mac is to establish a set of UI conventions. When you launched a new app, you would already know how to use most of it, because it was a Mac app. It looks and behaves like other apps, so you feel at home already. And as a developer, you get the right behavior now and in the future, for free.
But if every developer builds a cross-platform app with a custom framework and appearance and behavior and UI, then the OS loses its role in defining the platform conventions. In that event, what’s the point in having more than one OS?
John Gruber (tweet):
I’m with Ammon: I think the Mac’s (relatively) recent move to cryptographically signed applications — with certificates that can be revoked by Apple — has been a win all around for security. But I don’t think the Mac sandbox has.
[…]
The whole point of the Mac is to be a great platform for native Mac apps. Sandboxing doesn’t help Mac apps do more. If the Mac devolves into a platform where people just use web browsers and cross-platform Electron apps, it might as well not exist[…]
[…]
The real problems facing the Mac are the number of developers creating non-native “Mac” apps and the number of users who don’t have a problem with them.
Andy Ihnatko (in 2011, previously):
Traditionally, the mandate of an operating system has been to enable all of a machine’s potential. Higher-level software is responsible for making a computer easy to use and sometimes that means putting the power tools high enough on a shelf that the kids can’t hurt themselves, but those resources should be there for anybody who looks for them.
[…]
The Mac must never, ever become a consumer product like the iPad, saddled with artificial limitations in the name of safety, reliability, and tidiness.
See also: Jeff Johnson, Dan Counsell, Sayz Lim, Michael Dupuis, Dave DeLong, Marcus Zarra.
Previously: Sandbox Limitation on Number of Files That Can Be Opened, Apple Rumored to Combine iPhone, iPad, and Mac Apps to Create One User Experience.
Cocoa Electron Gatekeeper History Little Snitch Mac macOS 10.13 High Sierra Optical Character Recognition (OCR) Privacy Sandboxing Screenshots Web
Mark Gurman (tweet, MacRumors, Reddit, Hacker News, ArsTechnica):
Instead of keeping engineers on a relentless annual schedule and cramming features into a single update, Apple will start focusing on the next two years of updates for its iPhone and iPad operating system, according to people familiar with the change. The company will continue to update its software annually, but internally engineers will have more discretion to push back features that aren’t as polished to the following year.
[…]
The shift is an admission of what many customers have already come to notice: Some Apple software has become prone to bugs and underdeveloped features. […] Apple has also recently released features later than it expected, as the rush to meet the annual deadline overtaxed engineers and created last-minute delays.
John Gruber:
I’m not so sure the above is a new strategy so much as a tacit admission of what’s actually been going on the last few years.
Then why should we expect any improvement?
Jeff Johnson:
The idea of postponing features a year until they’re “ready” misses the whole point. It’s very difficult to find all the bugs in a major change until after you ship it. To get to a stable operating system, you need to spend at least a year just fixing bugs after a major release.
You can’t just consider the internal costs of annual updates. There are major external costs. Third party developers play an essential role in QA. If we never get the thing until June, and you ship every fall, never enough time to fix bugs.
A lot of people are pointing to Steven Sinofsky’s comments (Reddit). He makes some good points about the “broader context,” but I think he’s completely wrong about Apple’s software quality:
In any absolute sense the quality of Mac/iOS + h/w are at quality levels our industry has just not seen before. […] On any absolute scale number of bugs—non-working, data losing, hanging mistakes—in iOS/Mac is far far less today than ever before.
I don’t see how that can be taken seriously. He doesn’t have access to Apple’s bug database, so how would he know? I really doubt that the number of open bugs is lower than in the past, and even if it were there’s no reason to assume that Radar is representative of the actual number of bugs. He later says that the list of bugs is “infinitely long,” so this whole line of argument seems nonsensical. In what way is today’s Mac/iOS quality better in “any absolute sense” than in, say, 2010? He doesn’t say, except that more people are using it:
What is different is that at scale a bug that happens to 0.01% of people is a lot of people. A stadium full or more. […] No one ever anywhere has delivered a general purpose piece of S/W+H/W at scale of 1B delivering such a broad, robust, consistent experience. We don’t have a measure for what it means to be “high quality”.
Well, we can look at how many problems an individual user runs into. Is it higher or lower than before? This measure is independent of Apple’s scale. So is the circle of people I hear complaining. Apple’s customer base has doubled many times over, but the number of family members, friends, and customers that I communicate with has not. Now you could argue that maybe we have become exceptionally unlucky and are running into more than our share of issues, but I don’t find that very convincing.
He wants to discount the actual experiences of “many super smart/skilled people” because “the more a product is used the more hyper-sensitive people get to how it works.” What does that even mean? The number of hours in a day hasn’t increased; I don’t think my Mac/iPhone usage has increased much, if at all. Hardly anyone complains to me about the “slightest changes”; I hear about things that flat out don’t work. That’s not being hyper-sensitive.
Previously: Apple Delays Features to Focus on Reliability, Performance.
Update (2018-02-13): jarjoura:
As someone who used to work on iOS at Apple, what that company honestly needs is a culture not beholden to the whims of their EPMs (project managers). They used to help organize and work with engineering to schedule things across the company’s waterfall style development. However, by the time I left, they essentially took power over engineering. Radar became the driver for the entire company and instead of thinking about a holistic product, everything became a priority number. P0 meant, emergency fix immediately, P4 meant nice to have. You get the idea.
Nothing could be worked on if it wasn’t in Radar with a priority number attached and signed off by the teams’ EPM. No room for a side project or time away from your daily duties because there were always P1s to fix. If you didn’t personally have any left for the day, you’d take one from another engineer who was likely swamped with their own list of P1s.
[…]
This is how you get bugs in shipping software. EPMs driven to schedule things and over manage engineers would decide on a whim that something was a P2. That was basically always shelved to a follow-up .1 release.
Ultimately, engineers lost the freedom to decide when a feature was ready to ship.
This point about bug prioritization came up two years ago.
Bob Burrough:
This is absolutely, 100% true, and jibes with my experience.
There was (don’t know if there still is), another really whacky problem with iOS work prioritization back then. Radar has P1, P2, P3, etc priorities. Milestones were arranged such that “No P3’s” happened (the point at which P3’s were no longer allowed to be worked on)… followed by “No P2’s” then finally “No P1’s.” At first glance that arrangement makes sense because it means the only bugs getting fixed late in the game are the really high priority ones. However, what it meant in practice was, if there was a P2 bug an engineer wanted to fix… They would scramble to make sure it gets fixed before the “No P2’s” milestone occurs…in effect, causing P2 bugs to be worked on before P1’s.
throwaway7187:
I’m a former iOS EPM (not speaking for Apple, obviously, since I don’t work there anymore), and although the Reddit commenter got the atmosphere of constant crisis right, he/she is misplacing the blame and misunderstanding the power dynamic. EPMs at Apple essentially have zero power over engineers’ workload. They take the list of stuff the engineering managers said they want to get done this year and say “You guys are crazy, you’ll never be able to do this without 3x the hours/manpower.” Then they proceed to drive the team as hard as necessary to make sure that they actually deliver what they said they were going to deliver. That’s it. The idea that there is this cabal of mighty EPMs twirling their mustaches and loading developers down with work is pretty far from reality.
It’s true that you shouldn’t be working on anything not in Radar (the bug tracker) but this is true anywhere you’ll work. Project managers however do not sign developers up for all those radars--on the contrary--we’re usually trying desperately to help you get rid of scope and get the task list down to what’s actually do-able!
One of the great things that IMHO sets Apple apart is how engineering-driven they are. I’ve never worked anywhere else where engineers had so much freedom to decide what they’re working on. The fact that they always decide to work on 3x what they can actually achieve is kind of on them. But that drive to try to do so much is part of what keeps innovation strong at Apple.
Benjamin Mayo:
It sure looks like this is a case of the feedback loop working. The Apple community complains about software quality, the executive team reviews procedures and makes structural changes.
[…]
As an outsider, I think it’s hard to really assess whether these changes are meaningful rather than empty, ambitious, words. However, I’m glad the way it is portrayed in the Bloomberg report indicates it is a deeper shift of philosophy rather than a one-time focus for iOS 12 followed by a return to the status quo.
Nick Heer:
If the changes are as modest as this report makes them out to be, how much of an improvement can we realistically expect in software quality?
Tim Bradshaw:
First Apple shareholder question is about software quality, which he says is “very unsatisfactory”. “We are getting plenty of changes but not many improvements... My solution has been to stop upgrading because I no longer trust Apple.” Apple is “losing touch with working people”
Update (2018-02-14): Riccardo Mori:
While I’m certain there are still underlying issues left unsolved in both Tiger and Leopard, in day-to-day general use, nothing prominent shows up on my radar. I turn on this PowerBook, it boots into Mac OS X 10.5.8, I open whatever apps I need for this session, and I feel I’m working in a stable, predictable environment. The only unfortunate thing I notice is that in places the hardware shows its age, or that certain features or services are too new to support this platform, but neither this particular vintage Mac nor its Mac OS X version are at fault. And it’s pretty amazing I’m still being productive with a 14-year old machine.
[…]
I’m just an outside observer, with perhaps the vantage point of having been using Apple hardware for almost 30 years. I can’t say with certainty that today both Mac OS and iOS have more bugs and issues than before. I’m also not saying that everything was 100% perfect before and now it’s all rubbish, because it’s not true. But from having extensively used (almost) each version of Mac OS and iOS, what I do notice is that behind the scenes there was a different approach to their development before a certain point in Mac OS X’s timeline, and that something changed (for the worse) after that point.
Update (2018-02-16): See also: Download.
Update (2018-02-21): See also: The Talk Show.
Apple Apple Software Quality Bug Tracking iOS iOS 12 Mac macOS 10.14 Mojave Radar and Feedback Assistant
