Explanation for Consumer Reports’ Battery Results
Rene Ritchie quoting Apple (tweet, MacRumors, AppleInsider, Hacker News, ArsTechnica):
We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test.
It sounds like they are referring to the “Disable Caches” setting in the Develop menu. It’s not front and center, but I wouldn’t call it “hidden” since it can be enabled from within Safari’s own Preferences window. It’s not a defaults write preference.
The setting definitely is used by customers, e.g. Web developers, though probably not by most consumers.
It does seem legitimate to me for Consumer Reports to turn off caching when repeatedly loading the same Web pages. A test that just reads from the cache would not be representative of real-world use. Maybe it would have been better to test a series of unique pages with the cache enabled. Either way, it’s hard to say what would be a fair, controlled test involving servers run by third parties from around the world. Consumer Reports tried to control for some factors by loading pages from their own server, but that introduces other problems.
Regardless, their main finding was the inconsistent battery life, and it sounds like that is explained by the Apple bug.
Previously: Consumer Reports on the New MacBook Pro’s Battery Life.
Update (2017-01-10): Consumer Reports:
We communicated our original test results to Apple prior to publication on Dec. 22 and afterward sent multiple rounds of diagnostic data, at the company’s request, to help its engineers understand the battery issues we saw in our testing. After investigating the issue, Apple says that the variable battery performance we experienced is a result of a software bug in its Safari web browser that was triggered by our test conditions.
[…]
Separate from Consumer Reports’ test findings, many MacBook Pro owners have posted in user forums about episodes of remarkably short battery life, and both CR’s findings as well as these consumer posts have caused much discussion and debate in the tech press and on user forums.
[…]
At Consumer Reports, we test every laptop from every manufacturer in a comparable way. Because people use laptops differently and because their usage can vary from day to day, our battery tests are not designed to be a direct simulation of a consumer’s experience. Rather, we look to control as many variables as possible, then perform a test that gives potential users a reasonable expectation of battery life when a computer’s processors, screen, memory, and antennas are under a light to moderate workload. This test has served as a good proxy for battery life on the hundreds of laptops in our ratings.
[…]
Many of these settings are set by default to extend battery life. That’s generally a good thing. But because these settings are so variable and situation-dependent, we turn several of them off during testing. For instance, we turn the screen auto-dimming features off on all laptops and set the displays to a constant level of brightness. […] We also turn off the local caching of web pages.
I still think something was/is wrong with Consumer Reports’s testing, but I don’t think it’s fair to say that disabling the caches is unfair or a flawed method. And while the preference setting is obscure, I wouldn’t call it “hidden”.
Geoff Hackworth notes that none of this explains how Consumer Reports got a high of 19.5 hours, when Apple claims only 10 hours of “wireless web.” It will be interesting to see the times when they re-test.
At Macworld we built a lot of different lab tests over the years. It’s hard to test real-world performance in automated tests. You want to produce a result that represents what regular people would experience when using the product, but it’s a constant battle against software and hardware that’s designed to reduce power consumption at every turn. You can’t just use a human to do the testing, because in addition to being wildly inefficient (these tests take a days to perform, per system), they won’t be exactly the same on all the different systems.
[…]
There are judgment calls like that at every turn when you’re building lab tests.
[…]
My guess is that this bug is more likely the cause of the battery-life disparity than anything specifically weird or unfair in Consumer Reports’ laptops tests, but I suppose we’ll see when it revisits its findings.
Consumer Reports has a spotty history with calling Apple out on product flaws. […] But almost every time, the problem they’re reporting is real — especially in retrospect, after everyone’s defensiveness has passed and we’ve lived with the products for a while. It’s just debatable how big of a deal it is in practice.
[…]
Apple’s framing here is almost Trumpian, evading responsibility for the real problem — Apple’s bug — by attempting to insult the test (“does not reflect real-world usage”), discredit and imply malice by Consumer Reports (“a hidden setting”), and disregard the bug as irrelevant (“obscure and intermittent bug”).
[…]
But disabling the browser cache during a battery test to make results more consistent is reasonable, Apple’s browser offers that feature, and it’s neither very well hidden nor unused by any customers.
They did contact Apple about the results, and Apple sent them a canned statement.
Sean:
Apple declined to help until after CR published. That’s Apple’s MO.. they don’t acknowledge problems.
It would be interesting to know how long they waited before publishing, but it’s hard to fault them for doing so if that was Apple’s only response.
The [Apple] statement goes on a little longer, but the nutshell version comprises these three sentences. And I have issues with all of them.
Update (2017-01-13): Consumer Reports (Glenn Fleishman, John Gruber, Hacker News):
With the updated software, the three MacBook Pros in our labs all performed well, with one model running 18.75 hours on a charge. We tested each model multiple times using the new software, following the same protocol we apply to hundreds of laptops every year.
There’s still no explanation for the unexpectedly long battery life that they saw. And why did the bug fix reduce the time from 19.5 to 18.75 hours?
I’d also like to point out that people seem to be misremembering Antennagate, perhaps because of Jobs’ masterful presentation. The iPhone 4, even with the bumper, had worse reception than all previous iPhones. In the area where I live, this was the difference between the phone working and not working. The Apple Store wouldn’t acknowledge this and made it difficult to return the phone.
Update (2017-02-01): Gordon Mah Ung:
In looking at other battery run-down scenarios, I ran smack into a problem that’s likely at fault for many of the confusing battery life issues with the laptop, at least in macOS Sierra 10.12.2. On occasion, the laptop’s discrete GPU would just get stuck on and consume power even when it wasn’t used. Others had reported this too, but you’d really have to stumble onto it.
I was able to reproduce the issue in Safari by opening Google Maps, which would cause the laptop to switch over to the GPU for the WebGL workload. Opening additional browser tabs and then closing the Google Maps tab would, on occasion, leave the GPU consuming up to 10 watts of power while doing absolutely nothing.
Update (2017-02-14): Benedict Slaney (via Rui Carmo):
I don’t know exactly what Apple or Intel have done right, but the power drain under low usage conditions is now around half of that that the previous MBP’s.
[…]
Given that the time remaining calculation is calculated by using nothing but the battery drain average over recent time, the only reason behind Apple’s statement that the newer CPU’s are causing unpredictable calculations that I can think of would be because sometimes the CPU moves into a higher power state for a minute or so, and that the time remaining estimate gets updated during the same period, only to be inaccurate because the CPU then switches into a low power state and spends the majority of its time there, or vice versa. But that answer just leaves more questions hanging.
[…]
This is in line with consumer reports saying that they would get around 19 hours out of a laptop on a single charge. It’s also in line with people reporting that under high load the new MBP lasts a shorter amount of time than the earlier models. It does have a 25% smaller battery, and it is more performant. Under high stress it will use its resources and drain the battery quickly.
