Skip to comments.Consumer Reports Deck Stacking — or Incompetence — Exposed
Posted on 01/11/2017 9:47:26 AM PST by Swordmaker
Macs tend to fare second best in Consumer Reports testing, partly because the magazine lives in ignorance of the differences between Apples computers and Windows boxes. But theyve always been recommended, until recently. I can quibble about the way the tests appear to emphasize features over performance, usability and reliability. In fact, I have.
But it took a poor rating by CR to trigger a dialogue that revealed a serious flaw in their testing. The tests also triggered an obscure bug in Safari for macOS Sierra that might otherwise have remained undiscovered and unfixed.
It all started when CR reported wildly divergent battery life results, ranging from 3.75 hours up to 19 hours over three tests for each product. The latter is way more than Apples estimates, which range up to 10 hours.
Now all three MacBook Pro models exhibited similar behavior. A clue that something might be amiss was the fact that CR uses the default browser, in this case Safari. When the tests were rerun in Google Chrome, battery life was within acceptable limits.
Now Apple usually ignores test results from the media, but not CR, which has a circulation of millions of consumers and is highly influential when readers make buying decisions. A bad rating can kill or seriously hurt sales of some products. It can also accomplish good things, such as when an auto manufacturer has to go back and modify a faulty suspension system that might cause a rollover during a rapid maneuver to avoid an accident.
This time, Apple was in the hot seat. Even though a number of owners of the new MacBook Pros have reported an assortment of battery issues, CRs results were unique. The inconsistency didnt make sense, and thus marketing VP Philip Schiller posted a tweet the new normal for getting the word out nowadays saying that the results didnt jibe with Apples own field tests. Apple was working with CR to figure out just what was going on.
Now CRs tests are intended to be consistent from notebook to notebook. It involves downloading 10 sites from the companys in-house server until the battery is spent. So just what was going on here, and was the test deliberately designed to leave Safari and Macs second best?
Well, thats debatable, but to achieve consistent results, CR turns off caching on a browser. With caching on, the theory goes that the sites would be retrieved from the local cache, which presents an anomalous situation since different computers and operating systems might do it differently. On the other hand, it would also be using the computer normally, not in an artificial way. CRs excuse, by the way, is that the test sequence puts greater stress on the battery: This allows us to collect consistent results across the testing of many laptops, and it also puts batteries through a tougher workout.
But how can such a test possibly produce results that in any way reflect what a typical user would encounter? After all, normal users might check a site several times a day, rather than constantly bring up new uncached sites. While all notebooks are being evaluated the same way, its a curious choice. Unfortunately, CR would have to go back and retest hundreds of computers to switch the testing scheme.
On Safari, caching is switched off via a seldom-used menu bar command, Develop, which is available in the apps preferences under the Advanced category. Clearly this is not a feature most users will ever use or even know about. I use it to access the Show Page Source command from the context menu when Im examining a sites coding.
Now I suppose using a non-standard test scheme of this sort shouldnt have had a disastrous effect, but it did. It seemed that the action triggered an obscure and inconsistent bug in Safari. With caching turned off, logos would reload, thus unnecessarily taxing the battery. Its a bug that Apple discovered and fixed in the latest beta for macOS Sierra 10.12.3. You can download it if youre a public beta tester or developer, and it will be made available for general distribution in a few weeks.
In the meantime, CR has accepted Apples findings: According to Apple, this last part of our testing is what triggered a bug in the companys Safari browser. Indeed, when we turned the caching function back on as part of the research we did after publishing our initial findings, the three MacBooks wed originally tested had consistently high battery life results.
It would have been nice if they said that before the review appeared, because that clearly indicated there was some sort of software issue that might be unnecessarily impacting the tests in a way that customers wouldnt encounter. In other words, its an admission the test was unfair, and that the results didnt in any way reflect a normal use case. After all, CR is testing a notebooks battery life, not the capabilities of the default browser to render pages without caching.
In any case, CR is retesting the MacBook Pros with the revised macOS, and it shouldnt take more than a few days to deliver the results. Assuming battery life is normal, the rating will be changed accordingly, and the new notebooks will be added to the recommended list.
Of course, CR should have realized something was amiss as soon as the battery life normalized with caching on. They could have reached out to Apple before the results were published for clarification. As it was, CR got a boatload of publicity for its decision not to recommend the MacBook Pros. Of course, that result will soon be changed if all goes well.
Will CR learn a lesson from this debacle? Probably not. After all, few companies would dare protest a bad rating. Indeed most companies who build products that dont past muster probably deserve it.
None the less, it was a problem that Apple had to fix and CR wasn’t the only person reporting issue.
Did Apple pull a VW with this new Safari version?
We may see.
Have you ever,ever,ever posted anything on FR connected to elections,courts,societal trends?
No,I didn't think so.
That's the sort of slimy biased language I expect from the lying leftists in Big Media.
So, according to the story, CR made two mistakes:
1)They used the default browser supplied by Apple
2) They found a bug in Apple’s software.
Apple’s apologist goes a bit over the top with this defense.
Glad to see you posting. Hope this finds you felling better than the other night.
No, Okie, CR was the only one reporting the issue. Very few users surf the internet with the Developer's menu turned on, which is what is necessary to even turn OFF Safari Caching. It was only reported after CR found it because they were doing something no-one ever does. It is NOT a standard user mode, ever. It is a DEVELOPER mode for those who DEVELOP Websites so they can test them optimized for various browsers. Most people who do that are not going to be doing that on battery mode and would never notice it.
Quit trying to make a mountain out of a molehill. I bet you haven't a clue how to enable the developer menu, do you?
I DO have it turned on because I have a specific purpose for it.
The latest Apple/Mac/iOS Pings can be found by searching Keyword "ApplePingList" on FreeRepublic's Search.
If you want on or off the Mac Ping List, Freepmail me
What is that? The fortieth time you personally have asked your insulting, snarky question? It has to be close to that.
For AT LEAST THE FORTIETH TIME, yes, I have posted many articles associated with conservatism, societal trends, particularly courts, and the election. Give it a rest, forty is enough times to keep asking that question, unless you are too stupid to understand the first first forty times I answered you, ASSHAT! Now if you want to continue to prove that YOU are too stupid to grasp that same answer, keep asking and keep proving you are that stupid.
You misquoted the actual facts, deliberately, I think.
CR also deliberately TURNED OFF SEVERAL SPECIFIC OPERATING SYSTEM FUNCTIONS which then crippled the OS's ability to function correctly because they DO NOT UNDERSTAND HOW UNIX and Apple's macOS SYSTEMS WORK! For the simple reason THEY DO NOT WORK THE SAME AS WINDOWS PCs.
CR made an ignorant, erroneous assumption that bit them in the ass.
For those who don’t know...Intel’s power management is very complex and has evolved over the years.
It is now much simpler to code for but its probably some legacy software that wasn’t quite right with the latest hardware.
Now objectively, Apple did goof on the latest Macbook Pro.
I was all set to buy one and when I saw the result, I picked up a new 2015 instead.
“here on what is primarily a *political* website”
The fact that YOU have to say “primarily” means you know that this website is not ONLY for politics, and people are allowed to post on a whole variety of topics.
If you don’t like a topic, DON’T CLICK THE LINK. Problem solved, if you are not an attention whore.
“It can also accomplish good things, such as when an auto manufacturer has to go back and modify a faulty suspension system that might cause a rollover during a rapid maneuver to avoid an accident.”
As an aside, not exactly a ‘settled’ notion:
They used the same test standards for everyone. I’ll agree that CR was wrong if Apple puts a sticker on each box saying that they are entitled to the special ed modified test instead of the standard test given to the rest of the class.
I’m quite sure that Apple could design a test they could pass. Or they could change out the default browser. Or better yet, they could modify Puppy Linux to run on their hardware and use it instead of the Apple OS, and probably double battery life.
I gave up on CR when they gave Obama and interview and supported obamacare
That’s just nonsense. There have been numerous reports of battery problems. If I remember correctly Apple sent out a patch to not show remaining time left some time back.
I know you feel obligated to promote Apple’s position but some realism is required to remain believable.
That's the Swordy I know; welcome back, glad you're feeling better, FRiend!
Yes, they tend to be watermelons - Green on the outside, Red inside. So it’s a bit of a surprise they actually went after the computer of the liberal elites.
The 2016 MacBook Pro works as advertised for 99.9% of all users of the device, unless someone deliberately turns OFF several Operating System functions.
Perhaps Apple should not have permitted that option, but exactly how is that a "goof" that effects only a tiny fraction of 2016 MacBook Pro users who do it deliberately, intending to cripple the computer from working as intended?
I want to get the new MacBook Pro, but I like to wait it out for a year to let them work out all the kinks.
Look ASSHAT, I make a very good living off of supporting Windows, Linux, and Mac PCs for multiple clients. . . with the vast majority of my income coming from Windows PC problems. I've owned a business doing that for nearly 40 years. That is not the sign of anyone who was "frightened' of something that did not exist when I was a child.
Your little ratio only reflects the fact that I maintain the Apple/Mac/IOS/IPhone/IPad Ping list of over 750 of your fellow Freepers who have, each and every one of them, ASKED me to keep them informed on those topics. Just as dayglored and ThunderSleeps maintain the Windows and Android ping lists respectively.
If you don't like Apple threads, stay the hell off of them and don't comment! It's that simple. YOU are outnumbered by the 750 Freepers who do appreciate them!
Now, I am recovering from a heart attack I had Friday and I don't need your insulting, holier-than-thou asshattery. GO AWAY!
I'm not sure. . . I got on here to enjoy some FR time and already the repeat Anti-Apple AssHats are attacking for the fortieth time like the one above, who shall go nameless. He's adding a lot to my stress levels. Thanks for your concerns. I appreciate them.
It can also accomplish good things, such as when an auto manufacturer has to go back and modify a faulty suspension system that might cause a rollover during a rapid maneuver to avoid an accident. “
Such as Piddle screaming at a driver “if you can’t roll it, I will find someone who can!” paraphrased.
CR isn’t worth the paper it is printed on or the electrons that give their lives for the digital one.
WHAT PART OF THE OPERATING SYSTEMS ARE DIFFERENT TO YOU FAIL TO GRASP? Trying to make a Mac into a Windows PC is doomed to cause problems and that is why CR always fails when doing tech comparisons by trying to make things equal. They don't know what they are doing! They are all Windows geeks!
I recall the Suzuki rollovers and the counter video of the same model going through the same course WITHOUT the outriggers CR fitted theirs with because they were afraid it would tilt, going faster without tilting at all. Their vaunted engineers didn't realize their heavy steel outriggers actually facilitated the tipping! They moved the vehicle's center of gravity!
They never asked where were all the real world tipping accidents?
My pulse rate hasn't exceeded 70 in this exchange despite have been called "asshat" (whatever that is).And yet here you are on the verge of another myocardial infarction.Perhaps you should take up yoga in addition to taking your ace inhibitor,your beta blocker and your calcium channel blocker.
Yet more proof that Apple is a religion to you while it's just a thing to most.
Just callin' 'em as I see 'em!
Welcome back Swordmaker -— I’m glad to see you have as much energy (and patience) as before. Too bad the surgeons couldn’t have also excised the bubble-headed Apple-haters who plague FR while they were fixing you.
Good to have you back!
You have no purpose on Apple threads except to spread discord and rancor and insult people who have done you no harm. Go AWAY!
pejorative metaphoric adjective; refers to those who have their heads up their asses - metaphorically speaking; as in short sighted, dimwitted or asinine or self-absorbed persons
"That selfish asshat doesn't give a damn about anyone but himself." OnlineSlangDictionary.com
Your repeated insulting, not-on-topic invasions of Apple threads have earned you that sobriquet. You have no interest in Apple topics so just stay out, as JR has ordered instead of trying to incite flame wars!
That's because Chrome doesn't have the ability to do what Safari does at the OS level in turning off functions. It's a down and dirty browser and doesn't support what Safari does.
“...Just callin’ ‘em as I see ‘em!”
Based upon your posts in this thread, you are a major league A-hole who enjoys trolling your fellow FReepers.
Just callin’ ‘em as I see ‘em.
Not what he is paid to do.
Good God...what's your pulse right now? 110? Mine's still 70.Get a hobby.A hobby that doesn't involve anything powered by a plug or a battery.
And I'd wager everything I own that your doctors would agree!
This reminds me of the Pentium floating point bug.
Intel didn’t want to replace the processor because the average joe would only run across it under certain conditions.
Rather than say thanks. Intel fought replacing it until the low roar became a din and they finally decided to replace it “upon request”.
Seems Apple could have just said, hey thanx! Fixed it and be on there merry way....and hire the guy who set up the test as a QA guru.
But there are still too many Mac v. PC fire breathers and it produces crap articles like these - blame the guys who find and report the bugs.
Apple HAS said thanks and is fixing this minor fur ball. It is not a major problem that effects users of the MacBook Pro who use it as it is MEANT to be used. 99.9% of users never turn on the Developer Menu in Safari and will never find the commands to turn them off.
And the fact is that CR did not find it, Apple did in trying to figure out what CR did to get such crappy results! Apple thanked them for discovering the crappy results under very unusual circumstances, but CR had not a clue about why it was doing it.
Another trolling liar in an Apple thread. You've also been told repeatedly I am not employed to do anything for or to benefit Apple. Give your LIES a rest.
You're still here? You've been told by several of your fellow Freepers that you aren't welcome here. Obviously you ARE as stupid as you look. My pulse rate, if it is any of your freaking business is 80, not bad for recovering from a MI! But your purpose here is to raise the hackles of every Apple user who wants to participate in the thread with out harassment. YOU MAKE THAT IMPOSSIBLE.. . which is obviously your intention.
No, you are a rude, obnoxious a$$hat glory hog -- who delights in crashing other folks' parties and tossing turds into their punchbowls.
I (an apple owner/user/developer since the Apple ][+) ASKED to be invited to these Apple ping-list threads -- and, I read them, every one.
Yet, you (and several other mentally disturbed [or Microsoft financed] uninvited trolls) post on these Apple threads far more than I do.
Get psychiatric help-- you are mentally imbalanced!
Grow up, quit interrupting adults' conversations -- and save your feces for your own punchbowl!!
And that goes for several others of your ilk, here on FR -- and all of you know da*ned well who you are...
Yup,last post.I've made my point and several responses have confirmed my basic position.But consider the OP's tagline and several of his/her responses here.Do they not suggest that I'm not the only who's detected a pattern...and,very possibly,an obsession?
Feel free to respond if you wish but I'm done here.Time to move on to important things...like the repeal of ObamaCare and the enforcement of immigration and residency laws (among other noteworthy issues of the day).
Back in the 70s I did washing machine repairs. CR rated a Kenmore as recommended, but did not recommend a Whirlpool, and the Kenmore was a rebadged Whirlpool. Don’t know what goes on under the hood at CR, but I don’t trust their ratings.
Has Apple come out with an accessory kit containing all the cables, external devices that you need to operate the 'portable' laptop?
You're sitting in an airport terminal. You want to look at a DVD or just an SD card, thumbdrive, etc. You have to drag out a certain cable, external device, drive. Crazy!
They really really missed the boat on this one.
Oh, and the backpack large enough to carry all the accessories?
I'm a Mac fanboy but this new MBP ? No thanks. A giant step backwards.
It would almost be easier for me to carry my 21" iMac than these new MBPs.
Here's an idea for you. Stay with the important things of the day, and away from technology threads. Then you won't have to waste valuable time, or stress your apparently non-existent understanding that Free Republic people can multitask.
If you don't like other topics being pursued on Free Republic, you can take your concerns to Jim and John Robinson.
There was a bug. “Its a bug that Apple discovered and fixed “. Apple admitted there was a bug, and they issued a fix. All of your spin can’t change reality. And even the writer had to admit the truth in the midst of his attacks on CR.
Where have I denied there was a bug in a computer system YOU obviously don't use? I am not "spinning" the basic point of these articles which was that Consumer Reports modified a consumer product in such a way that their tests no longer reported on what the consumer's experience with that product would be if the consumer purchased it as the manufacturer presented it!
A factual claim is either true, or it is not. In this instance CR blew it.
Consumer Reports editors thought their readers would want to know how long laptop A's batteries will last on a single charge against laptop B, C, D, and E, each doing an identical set of specific common tasks (none of which, I might point out, was webpage design in the listed protocols which would have to be activated on the Mac to even get to the system tools modification menu.).
These consumers needed to know which out-of-the-box laptops' battery performed longest at each task, not which laptops' battery performed longest after someone turned off system features he thought were superfluous but in actuality may be important to achieving long battery life.
Instead, Consumer Reports recast itself as a Technology Hardware Review magazine and essentially decided to remove the software that differentiates the various computers from each other, by differing means of optimization.
Geeks might have an interest in the performance of just the raw hardware of the competing products, but real consumers are interested in the performance of the entire widget as designed by the manufacturer and as they can buy and use it! They are not interested in a faux product artfully modified by the decisions of Consumer Reports' engineers' very peculiar opinions of what the consumers should interested in.
Basically, what Consumer Reports did was compare Consumer Reports VERSIONS of these laptop, because no one could go buy a MacBook Pro, or a Dell, Sony, or any of the other compared laptops off-the-shelf, a claim CR prides itself in assuring its readers the products the review are to assure CR's independence, that operate in the exact same way the Consumer Reports laptops operated because the tested products were modified and were therefor, no longer representative of the off-the-shelf product!
Nowhere could you buy a Consumer Reports version of these laptops in the retail market. Nowhere. Consumer Reports was comparing NON-EXISTENT products.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.