Posted on 01/11/2017 9:47:26 AM PST by Swordmaker
Macs tend to fare second best in Consumer Reports testing, partly because the magazine lives in ignorance of the differences between Apple’s computers and Windows boxes. But they’ve always been recommended, until recently. I can quibble about the way the tests appear to emphasize features over performance, usability and reliability. In fact, I have.
But it took a poor rating by CR to trigger a dialogue that revealed a serious flaw in their testing. The tests also triggered an obscure bug in Safari for macOS Sierra that might otherwise have remained undiscovered and unfixed.
It all started when CR reported wildly divergent battery life results, ranging from 3.75 hours up to 19 hours over three tests for each product. The latter is way more than Apple’s estimates, which range up to 10 hours.
Now all three MacBook Pro models exhibited similar behavior. A clue that something might be amiss was the fact that CR uses the default browser, in this case Safari. When the tests were rerun in Google Chrome, battery life was within acceptable limits.
Now Apple usually ignores test results from the media, but not CR, which has a circulation of millions of consumers and is highly influential when readers make buying decisions. A bad rating can kill or seriously hurt sales of some products. It can also accomplish good things, such as when an auto manufacturer has to go back and modify a faulty suspension system that might cause a rollover during a rapid maneuver to avoid an accident.
This time, Apple was in the hot seat. Even though a number of owners of the new MacBook Pros have reported an assortment of battery issues, CR’s results were unique. The inconsistency didn’t make sense, and thus marketing VP Philip Schiller posted a tweet — the new normal for getting the word out nowadays — saying that the results didn’t jibe with Apple’s own field tests. Apple was working with CR to figure out just what was going on.
Now CR’s tests are intended to be consistent from notebook to notebook. It involves downloading 10 sites from the company’s in-house server until the battery is spent. So just what was going on here, and was the test deliberately designed to leave Safari — and Macs — second best?
Well, that’s debatable, but to achieve consistent results, CR turns off caching on a browser. With caching on, the theory goes that the sites would be retrieved from the local cache, which presents an anomalous situation since different computers — and operating systems — might do it differently. On the other hand, it would also be using the computer normally, not in an artificial way. CR’s excuse, by the way, is that the test sequence puts greater stress on the battery: “This allows us to collect consistent results across the testing of many laptops, and it also puts batteries through a tougher workout.”
But how can such a test possibly produce results that in any way reflect what a typical user would encounter? After all, normal users might check a site several times a day, rather than constantly bring up new uncached sites. While all notebooks are being evaluated the same way, it’s a curious choice. Unfortunately, CR would have to go back and retest hundreds of computers to switch the testing scheme.
On Safari, caching is switched off via a seldom-used menu bar command, Develop, which is available in the apps preferences under the Advanced category. Clearly this is not a feature most users will ever use — or even know about. I use it to access the “Show Page Source” command from the context menu when I’m examining a site’s coding.
Now I suppose using a non-standard test scheme of this sort shouldn’t have had a disastrous effect, but it did. It seemed that the action triggered an obscure and inconsistent bug in Safari. With caching turned off, logos would reload, thus unnecessarily taxing the battery. It’s a bug that Apple discovered and fixed in the latest beta for macOS Sierra 10.12.3. You can download it if you’re a public beta tester or developer, and it will be made available for general distribution in a few weeks.
In the meantime, CR has accepted Apple’s findings: “According to Apple, this last part of our testing is what triggered a bug in the company’s Safari browser. Indeed, when we turned the caching function back on as part of the research we did after publishing our initial findings, the three MacBooks we’d originally tested had consistently high battery life results.”
It would have been nice if they said that before the review appeared, because that clearly indicated there was some sort of software issue that might be unnecessarily impacting the tests in a way that customers wouldn’t encounter. In other words, it’s an admission the test was unfair, and that the results didn’t in any way reflect a normal use case. After all, CR is testing a notebook’s battery life, not the capabilities of the default browser to render pages without caching.
In any case, CR is retesting the MacBook Pros with the revised macOS, and it shouldn’t take more than a few days to deliver the results. Assuming battery life is normal, the rating will be changed accordingly, and the new notebooks will be added to the recommended list.
Of course, CR should have realized something was amiss as soon as the battery life normalized with caching on. They could have reached out to Apple before the results were published for clarification. As it was, CR got a boatload of publicity for its decision not to recommend the MacBook Pros. Of course, that result will soon be changed if all goes well.
Will CR learn a lesson from this debacle? Probably not. After all, few companies would dare protest a bad rating. Indeed most companies who build products that don’t past muster probably deserve it.
Please stop.
Yup,last post.I've made my point and several responses have confirmed my basic position.But consider the OP's tagline and several of his/her responses here.Do they not suggest that I'm not the only who's detected a pattern...and,very possibly,an obsession?
Feel free to respond if you wish but I'm done here.Time to move on to important things...like the repeal of ObamaCare and the enforcement of immigration and residency laws (among other noteworthy issues of the day).
Thank you.
Back in the 70s I did washing machine repairs. CR rated a Kenmore as recommended, but did not recommend a Whirlpool, and the Kenmore was a rebadged Whirlpool. Don’t know what goes on under the hood at CR, but I don’t trust their ratings.
Has Apple come out with an accessory kit containing all the cables, external devices that you need to operate the 'portable' laptop?
You're sitting in an airport terminal. You want to look at a DVD or just an SD card, thumbdrive, etc. You have to drag out a certain cable, external device, drive. Crazy!
They really really missed the boat on this one.
Oh, and the backpack large enough to carry all the accessories?
I'm a Mac fanboy but this new MBP ? No thanks. A giant step backwards.
It would almost be easier for me to carry my 21" iMac than these new MBPs.
Here's an idea for you. Stay with the important things of the day, and away from technology threads. Then you won't have to waste valuable time, or stress your apparently non-existent understanding that Free Republic people can multitask.
If you don't like other topics being pursued on Free Republic, you can take your concerns to Jim and John Robinson.
There was a bug. “It’s a bug that Apple discovered and fixed “. Apple admitted there was a bug, and they issued a fix. All of your spin can’t change reality. And even the writer had to admit the truth in the midst of his attacks on CR.
Where have I denied there was a bug in a computer system YOU obviously don't use? I am not "spinning" the basic point of these articles which was that Consumer Reports modified a consumer product in such a way that their tests no longer reported on what the consumer's experience with that product would be if the consumer purchased it as the manufacturer presented it!
A factual claim is either true, or it is not. In this instance CR blew it.
Consumer Reports editors thought their readers would want to know how long laptop A's batteries will last on a single charge against laptop B, C, D, and E, each doing an identical set of specific common tasks (none of which, I might point out, was webpage design in the listed protocols which would have to be activated on the Mac to even get to the system tools modification menu.).
These consumers needed to know which out-of-the-box laptops' battery performed longest at each task, not which laptops' battery performed longest after someone turned off system features he thought were superfluous but in actuality may be important to achieving long battery life.
Instead, Consumer Reports recast itself as a Technology Hardware Review magazine and essentially decided to remove the software that differentiates the various computers from each other, by differing means of optimization.
Geeks might have an interest in the performance of just the raw hardware of the competing products, but real consumers are interested in the performance of the entire widget as designed by the manufacturer and as they can buy and use it! They are not interested in a faux product artfully modified by the decisions of Consumer Reports' engineers' very peculiar opinions of what the consumers should interested in.
Basically, what Consumer Reports did was compare Consumer Reports VERSIONS of these laptop, because no one could go buy a MacBook Pro, or a Dell, Sony, or any of the other compared laptops off-the-shelf, a claim CR prides itself in assuring its readers the products the review are to assure CR's independence, that operate in the exact same way the Consumer Reports laptops operated because the tested products were modified and were therefor, no longer representative of the off-the-shelf product!
Nowhere could you buy a Consumer Reports version of these laptops in the retail market. Nowhere. Consumer Reports was comparing NON-EXISTENT products.
I remember back in the early 80’s CR called the Coleco Adam the “Computer of the Century”. Yeah, right.
Besides erasing any tape media left in the drive when it was turned on, due to the extreme electromagnetic surge on power-up, the power supply for the computer was located in the attached failure-prone printer. So when your printer died or had to be sent off for repairs, no computing.
Also in the late 70’s CR listed as Dangerous and Unacceptable, the front-wheel drive Dodge Omni/Plymouth Horizon. The reason was that when the test driver swerved the car as hard as he could in one direction and then as hard as he could in the other, and then let go of the steering wheel, the car did not continue in straight line.
WTF??
Apparently the test driver had no idea about driving a front-wheel drive car.
We had a Horizon from 78 to 84. Put a lot of miles on it with no problems.
Of course that may have been because we never swerved the car from side to side as hard as we could and then let go of the steering wheel.
Here is one third-party option. There are others.
I quit listening to CR back when they rigged their testing to intentionally force the Suzuki Samuri they tested to “tip over” in testing. They had to finally fess up to taking extra measures to force the issue (which in turn was a reminder of the similar behavior many years before by Ralph Nader and the Chevrolet Corvair.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.