A Blind HomePod Test
The PR person could switch playback from one speaker to the other without missing a beat. They even had a halo light rigged to turn on behind whichever speaker was playing, so you’d know which was which.
There was not a shred of doubt: In this side-by-side comparison, the HomePod sounded better than its competitors.
[…]
I hid the four speakers behind a curtain — a sheet of thin, sheer fabric that wouldn’t affect the sound. It took me a Sunday to figure out how to get the A/B/C/D switching to work seamlessly, but I finally managed it: All four speakers would be streaming from Spotify, all four over Wi-Fi. I’d use the Spotify app’s device switcher to hop among speakers without missing a beat.
[…]
For each song, I played the speakers in a different order (A to D sometimes, D to A sometimes). […] They held up their signs. Two of them ranked the Google Home Max (“D”) as the best. Three of them ranked the Sonos One (“A”) the best.
Nobody ranked the HomePod the best.
[…]
As far as I can tell, none of the other critics who declared HomePod No. 1 actually set up their own blind A/B/C/D tests. Maybe their conclusions wouldn’t have been so emphatic if they had.
Consumer Reports also rated HomePod lower (via MacRumors):
Overall the sound of the HomePod was a bit muddy compared with what the Sonos One and Google Home Max delivered.
All three of these speakers were impressive compared with other smart speakers we’ve tested, but they fall significantly short of our highest-rated wireless speakers, such as the Edifier S1000DB, $350, which earned an Excellent sound-quality rating.
It’s not clear to me whether they did a blind test.
Other reviewers continue to give HomePod high marks for sound. WinterCharm (ArsTechnica, Hacker News):
I am speechless. The HomePod actually sounds better than the KEF X300A. If you’re new to the Audiophile world, KEF is a very well respected and much loved speaker company. I actually deleted my very first measurements and re-checked everything because they were so good, I thought I’d made an error. Apple has managed to extract peak performance from a pint sized speaker, a feat that deserves a standing ovation. The HomePod is 100% an Audiophile grade Speaker.
See also: Jim Dalrymple.
Previously: HomePod Reviews.
Update (2018-02-14): WinterCharm:
In my review, I made a massive caveat that many news outlets when they picked this up, seemed to forget.
IN AN UNTREATED ROOM The HomePods are pushing better sound than a single X300A, as measured. That’s an impressive feat, I was impressed by it. but my conclusion was that obviously in a properly treated room, correctly set up speakers would be better. This is why I said that the product was great for the masses, but for audiophiles, you would be better off putting these in places like your kitchen and leaving your normal listening setup intact.
Unfortunately, as often happens with the news, they skim and summarize, in a way that some of the subtlety and conditions around which my main point rested are lost in translation. When you take 5000 words and turn it into 1 headline and a 250 word article, some stuff gets lost in translation.
Second, half the news outlets were Apple news sites who have a huge pro Apple bias. They picked it up… and it’s in their best interest to misrepresent or cherry pick the review, exaggerating the claims.
The much touted review of the HomePod posted by an “audiophile” on Reddit last week – and gleefully tweeted by Apple’s Phil Schiller – turns out to be a long mess of uninformed and poorly made measurements.
This reply on Reddit highlights many of the problems, notably the fact that the HomePod wasn’t measure in an anechoic room, but mainly the fact that the “reviewer” fudged the display of his graphs, making them look better than they were.
We’ve heard plenty of opinions on the HomePod’s general sound quality, so it’s a good time to measure the consistency of the HomePod’s sound distribution using some professional-grade acoustic analysis tools.
Update (2018-02-15): Kirk McElhearn:
I’ve been following this Reddit thread and its published results. It’s amazing that in a world of audiophiles who obsess over which USB cable makes their music sound better, that this person performed all of these measurements, and forgot to mention that the HomePod uses digital signal processing to alter all music that it plays. In other words, it is far from neutral, and audiophiles make a big deal about their equipment being neutral. The frequency response may be excellent, but the equalization alters the music from what it should sound like.
In fact, I think it’s highly possible that this reviewer has based the conclusions of his testing on false assumptions. The HomePod has dynamic digital signal processing; it alters the music based on the music. In other words, it’s not a fixed EQ setting, but one that changes as music is played (and according to the room where it’s played). As such, sending single frequency sine waves, or whatever he did, won’t show the results of the EQ.
Update (2018-02-20): David Pogue:
“Since the HomePod adjusts its sound to the acoustics of the room, you should not have used a piece of fabric to hide the speakers,” wrote @markbooth and others. “The fabric may have affected the HomePod’s sound.”
Well, no. The HomePod re-samples its listening position after each time it’s moved, during the first few seconds of music playback. We let the HomePod do its room listening before hanging the curtain, so it had already had the chance to adjust its sound.
Jean-Louis Gassée (Hacker News):
This is where we find a new type of difficulty when evaluating this new breed of smart speakers, and why we must be kind to the early HomePod reviewers: The technical complexity and environmental subjectivity leads to contradictory statements and inconsistent results.
6 Comments RSS · Twitter
"The HomePod is 100% an Audiophile grade Speaker."
Is it because repairing the power cable is very expensive?
Is he being silly on purpose? A) the curtain would affect sound quality B) the HomePod does acoustic modelling and the setup makes it model a 1x4 foot room and adjusts the sound to the space behind the curtain.
It might have been easier, and less contaminating of the test matter, to simply blindfold the observers. (Why do they need to look at anything, curtain or otherwise?)
A curtain wouldn't affect sound quality if it were made of a fabric similar to speaker grille fabric.
"The HomePod is 100% an Audiophile grade Speaker."
So it's a cheap bit of Apple-badged no-name crap that has some lead weights inside to make it feel more substantial?
"The much touted review of the HomePod posted by an “audiophile” on Reddit last week – and gleefully tweeted by Apple’s Phil Schiller – turns out to be a long mess of uninformed and poorly made measurements."
An "audiophile" did that? You don't say.
