If you are looking to buy a refrigerator or other boxy, new stainless steel-clad appliance, there is likely no better – or at least better respected – reference manual in the US than Consumer Reports. It is theoretically as unbiased as a publication could be. It takes no ads, buys all its own review units, and subjects them to rigorous, super-standardised testing.
The core belief of the Consumer Reports system is that everything can be objectively evaluated, reviewed and assigned a definitive numerical score denoting its worth (or lack thereof). Objectivity is the religion of Consumer Reports. And that works well for a lot of things! Refrigerators and washing machines, for instance. Or specific attributes of products, like a computer’s battery life.
There are lot of things it doesn’t work so well for, though. Like where software is heavily involved. But even for the things that a purely “objective” review system doesn’t work so well for evaluating, the bias of this kind of rigid system is apparent and its limits easily accounted for. For the system to exist, it cannot acknowledge that there are things it cannot evaluate. The foundational belief of the system is that it can evaluate everything, and what it determines in the end is the objective truth.
So, if you go back to June, it’s pretty fascinating that Consumer Reports – rigorous and objective Consumer Reports! – hastily recommended the iPhone 4 and then just as quickly retracted its own recommendation. At best, it means Consumer Reports recommended the phone without fully testing it. The reason for the retraction: the iPhone 4 antenna’s weakspot. It was very likely the tipping point for Apple, leading to a hasty press conference where it assured the world how truly awesome the iPhone 4’s antenna was.
Fast forward to today. In Consumer Reports’ Spring electronics buyers guide, the iPhone 4 is its highest-rated smartphone on AT&T, tying with a handful of other phones – Samsung Captivate, Vibrant and myTouch 4G – for the highest score across any carrier, with a score of 76. Every phone with that score – and several phones with lower scores – carries a Consumer Reports recommended rating. Consumer Reports even recommends the older, virtually obsolete iPhone 3GS, despite its worse screen, slower guts and fatter form factor. (The Verizon iPhone 4, despite being announced 2 months ago, is missing entirely from the ratings.)
Obviously, the normally mild-mannered Consumer Reports put itself in an interesting position by staking out such a bold position on the iPhone 4. It’s hard to back down from that. But is that position really in the best interest of consumers? The iPhone 4 is better than many of the phones it recommends, deathgrip or not. (Android fans, let’s at least concede it’s much better than the iPhone 3GS. The $US100 price difference is negligible, considering the thousand-dollar-level expenses of a 2-year contract.) Is the occasional annoyance of the deathgrip a true dealbreaker in a phone that is objectively better than most phones in screen quality, battery life, apps and speed? Even Consumer Reports own scores say at face value that it’s one of the best phones, period. Yet they will not recommend it.
If Consumer Reports will not recommend its objectively highest-scoring phone, what does that say about objective reviews?