Apple Releases Fix to MacBook Pros in Response to Consumer Reports' Battery Test Results

Consumers Reports ^ | January 10, 2017 | By Consumer Reports

[Swordmaker]

Apple says CR tests triggered a bug in Safari

Apple has just issued a software fix through its Beta Software Program to address an issue that arose when the company's new MacBook Pro laptops were being tested in Consumer Reports' labs. Although those computers performed well in tests of display quality, performance, and other factors, we found the battery life to be so variable on the models tested that we could not recommend them to consumers. In our tests of three different MacBook Pro models, we saw battery life results as long as 19.5 hours and as short as 3.75 hours.

As a result, these laptops were the first MacBooks not to receive Recommended ratings from Consumer Reports, and the only ones in our ratings of 140 laptops to demonstrate this degree of inconsistency in battery life. We have now downloaded the software fix and are rerunning our battery tests with the fix in place on the same computers previously tested. If the battery life results are consistently high, the ratings score for MacBook Pros would rise, and those laptops will then receive Consumer Reports’ Recommended rating given their performance in all our other evaluations.

We communicated our original test results to Apple prior to publication on Dec. 22 and afterward sent multiple rounds of diagnostic data, at the company’s request, to help its engineers understand the battery issues we saw in our testing. After investigating the issue, Apple says that the variable battery performance we experienced is a result of a software bug in its Safari web browser that was triggered by our test conditions.

“We appreciate the opportunity to work with Consumer Reports over the holidays to understand their battery test results,” Apple said in a statement. “We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache.... We have also fixed the bug uncovered in this test."

We turn off caching as part of Consumer Reports' standard laptop test protocol (more on that below). Caching is a feature used by many computers to store Web pages locally on a hard drive for faster retrieval by the browser.

Apple has posted its fix for the Safari bug on this website. The fix is available to anyone who signs up for Apple’s Beta Software program; it is then pushed to users through their computer’s Software Update function. Apple says the beta fix will be a part of a broader Software Update available to all MacBook Pro users in a few weeks, regardless of whether they sign up for the beta or not.

Separate from Consumer Reports’ test findings, many MacBook Pro owners have posted in user forums about episodes of remarkably short battery life, and both CR’s findings as well as these consumer posts have caused much discussion and debate in the tech press and on user forums. Apple’s senior vice president of worldwide marketing, Phil Schiller, tweeted on December 23rd that Apple had not seen the sort of wild swings in battery life in its own testing that CR has experienced in our labs, and he has responded to individual complaints. It’s unclear whether Apple’s bug fix for Safari will impact battery life issues for those individuals. 

CR’s Laptop Testing

At Consumer Reports, we test every laptop from every manufacturer in a comparable way. Because people use laptops differently and because their usage can vary from day to day, our battery tests are not designed to be a direct simulation of a consumer’s experience. Rather, we look to control as many variables as possible, then perform a test that gives potential users a reasonable expectation of battery life when a computer’s processors, screen, memory, and antennas are under a light to moderate workload. This test has served as a good proxy for battery life on the hundreds of laptops in our ratings.

For our initial round of testing, we examined three MacBook Pros: a 13-inch model with Apple’s new Touch Bar, which sits above the keyboard; a 13-inch model without the Touch Bar; and a 15-inch model. (All 15-inch MacBook Pros come with the Touch Bar.) All three machines were bought at retail, just like all products rated by Consumer Reports. We do this to ensure that the models we test are identical to the ones a consumer would purchase.

Here’s how our battery test works: We download a series of 10 web pages repeatedly, starting with the battery fully charged, and ending when the laptop shuts down. The web pages are stored on a server in our lab and transmitted over a dedicated WiFi network. We conduct our battery tests using the browser that is native to the computer’s operating system—Safari, in the case of the MacBook Pro laptops.

Modern laptops have a variety of sophisticated battery management techniques and settings built into both their hardware and operating system software. These battery management techniques include dimming of the screen and eventually putting the computer to sleep when it is not in use. Computers also employ software strategies to speed up performance and reduce the workload of their processors and antennas, including storing web pages locally on the hard drive for quick retrieval—otherwise known as caching.

Many of these settings are set by default to extend battery life. That’s generally a good thing. But because these settings are so variable and situation-dependent, we turn several of them off during testing. For instance, we turn the screen auto-dimming features off on all laptops and set the displays to a constant level of brightness. (That’s 100 nits, for our fellow science geeks out there.) Otherwise, the screens would constantly adjust their brightness, resulting in an inconsistent strain on the battery, and would likely trigger different results in our testing every time.

We also turn off the local caching of web pages. In our tests, we want the computer to load each web page as if it were new content from the internet, rather than resurrecting the data from its local drive. This allows us to collect consistent results across the testing of many laptops, and it also puts batteries through a tougher workout.

According to Apple, this last part of our testing is what triggered a bug in the company’s Safari browser. Indeed, when we turned the caching function back on as part of the research we did after publishing our initial findings, the three MacBooks we’d originally tested had consistently high battery life results.

“This is not a setting used by customers and does not reflect real-world usage,” Apple said in its statement. “[Consumer Reports’] use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life.... This is the best pro notebook we’ve ever made, we respect Consumer Reports and we’re glad they decided to revisit their findings on the MacBook Pro.”

Once our retesting of the MacBook Pro’s batteries is complete, we will report back with our update and findings. 

[Swordmaker]

Essentially, Consumer Reports tested the MacBook Pro following their protocols which did not use the computer the way anyone else would use it, apparently thinking it works the sane way as a Windows PC would be working. So they turned OFF some of the specific parts of the Operating System essential to its efficiency, speed, and battery life. . . things such as memory cacheing, and then reported that it was not as efficient, as fast, nor as long in battery use as Apple had reported it was.

What results did they expect to get when they do that?

[relictele]

our battery tests are not designed to be a direct simulation of a consumer’s experience.

THEN WHAT'S THE BLOODY POINT!?!

Benchmarking without context is navel gazing.

[dayglored]

> Benchmarking without context is navel gazing.

Worse, it's MISLEADING because it's coming from CONSUMERS Reports, one might get the mistaken impression it's about CONSUMERS.

But NO-O-O-O-O... it's about non-real-world testing in a lab in a scenario that no CONSUMER uses.

It's a software DEVELOPER setting!

NOTE: I make the same complaint about Windows PC testing in so-called "reviews" that's done with non-real-use parameters and then touted as "user experience".

[dayglored]

That said, Apple should send Consumers Reports $100 “Bug Bounty” for discovering the bug, even though in a back-ass-wards fashion. :-)

[Swordmaker]

Consumers Reports magazine testing protocols used on laptops is flawed in that in their attempts to make the playing surface fair, as if all computers were Windows PCs and operated the same, requires the turning off of particular Operating System functions which, on a UNIX system are intended to improve speed, efficiency, and battery life. They then reported the MacBook Pro as being deficient in those areas, after those functions were turned off. What do they think would happen when they did that? Although such adjustments to the operating system are not generally user configurable, Consumers Reports did it anyway in the interest of "fairness" in comparing laptop computers, as if somehow users were better off using a computer not as the manufacturer intended it to be used, but with part of the OS disabled. — PING!

Thanks to dayglored for the heads up.

Apple MacBook Pro and the Consumers Reports' Review
Ping!

The latest Apple/Mac/iOS Pings can be found by searching Keyword "ApplePingList" on FreeRepublic's Search.

If you want on or off the Mac Ping List, Freepmail me

[Irish Eyes]

Good to see you back.

[Swordmaker]

That said, Apple should send Consumers Reports $100 “Bug Bounty” for discovering the bug, even though in a back-ass-wards fashion. :-)

Perhaps. But it was not a dangerous flaw. If I were in charge of Apple, I'd send them a substantial contribution. However, Consumers' Reports would probably reject it because they wouldn't want to seem to have accepted a bribe.

[pgyanke]

Good to have you back. Hope you’re feeling better, my FRiend.

[Squantos]

Never fart in an apple store as they have no windows BTTT !

[TexasGator]

They expect to get comparable results.

Besides it is not memory caching

[ConservativeMind]

Apple admits they have a bug that, when given the exact same settings on multiple computers, causes a battery life differential of 3.75 - 19.5 hours. You lambaste Consumer Reports for finding this bug, yet Apple thanks them.

Why do you hate Apple and Consumer Reports when they identify and fix a bug? I think that schizo.

[Falconspeed]

Yes, good to see you back. Prayers. I use my 2007 white Macbook regularly without the battery. Great machine, fragile batteries and now I just use the plug.

Steve Jobs needed big time prayers during his last few years. He tried his best to beat his illness and stubbornly rejected Stanford’s advice early on. Always pray and try every possible cure: massage, acupuncture, surgery, vitamins, etc.

[roadcat]

I use my 2007 white Macbook regularly without the battery. Great machine, fragile batteries and now I just use the plug.

Batteries are dirt cheap for these machines lately. They used to run about $100 when the machines were new. Now you can get them for $10 to $20 on eBay. For that price, order two and keep a spare. Apple batteries are better for quality. Third-party offers stronger batteries with more mAH juice, but not all use good quality electronics. I've had issues with a NewerTech replacement, but not with other knockoffs. Have replaced batteries on several 2006-2007 machines, after about 8 years they tend to need replacing after heavy use.

[dayglored]

> Never fart in an apple store as they have no windows BTTT !

LOL, not so! Mac computers are designed to run Windows, inherently via BootCamp (dual-boot), and indirectly via third party VM host software.

On the contrary, it's Microsoft that has no MacOS!

In any case, everyone knows that apple farts don't stink.

BTTT indeed.

[Falconspeed]

“Order two and keep a spare” from Ebay.

Thank you. Simple advice yet I never thought of that option. Thanks.

[dayglored]

I can't wait until Consumer Reports tests a variety of cars, and does their testing with the following "fairness" conditions:

1. Since some cars don't have turbochargers, we disabled the turbochargers on those cars that had them.
2. Since some engines only have 4 cylinders, we disabled the "extra" cylinders on the 6- and 8-cylinder engines.
3. Since some cars' gas tanks only hold 12 gallons, we only put 12 gallons of gas in the 14-, 16-, and 18-gallon tanks of other cars.

Results:

1. We determined that cars with turbochargers don't go any faster than cars without turbochargers.
2. We determined that cars with 6-cyl and 8-cyl engines don't go faster than cars with 4 cylinders, and run rougher as well.
3. We determined that cars with larger gas tanks don't get any farther range in miles than cars with smaller gas tanks.

My, my, these guys at Consumer Reports are GENIUSES! Who else would have discovered such important results?!?

Bless their little hearts.

[conservatism_IS_compassion]

Thanks for your invaluable commentary, SM. So good to see your screen name . . .