M9: Is Throttling a Non-issue? Can’t We Trust Benchmarks?

When the first reports of the M9 overheating came to light, many forum users began a collective joke-round calling the phone a popcorn machine, a grill, and other unoriginal remarks that we’ve seen with every device that presents sign of overheating, from gaming consoles to graphics cards. In this sense, the internet is not very inventive, and the cycle of rehashed jokes re-surfaces on different products every year or so. This time it was the M9’s turn and it was quite a ride.

 

The Snapdragon 810 inside had raised overheating concerns months ago with rumors going around on how Samsung would not feature the flagship chipset in their S6, unlike they have traditionally done for many years (in some variants). The company opted for their state-of-the-art Exynos chipset instead, and during the week of Mobile World Congress, head-to-head synthetic benchmark battles showed the new Galaxy trumping the M9 by previously unseen percentages in flagship cycles.

The performance coupled with overheating complaints were claimed to be results of unfinished software. Moreover, it was suspected that Samsung was, once again, cheating in their benchmarks as they notoriously had done so in the past. I want to add that Samsung was not alone in this practice, and HTC has admitted to doing so as early as last year with their M8. Ultimately, the software revision was touted to fix the overheating that set the controversy ablaze – and unsurprisingly it did.

Unfinished Software

In our feature regarding unfinished software excuses we argued in favor of HTC and their promises, as at XDA we know how far software optimizations or implementations can take the hardware, and how a clever tweak can fix seemingly unworkable problems – our developers do it all the time! But we also argued that what would most likely happen would be a limit on peak performance that would throttle the phone to prevent it from overheating. And it happened.

The phone’s maximum frequency was globally scaled by 500MHz from the listed 2GHz frequency of the high-performance A57 cores. This directly means that every marketing specification sheet you’ll see at stores or sites will list a misleading number that is virtually impossible to obtain in real-world performance – and even if you do, your phone could overheat and throttle itself. AnandTech’s tests illuminate the subject of the M9 and benchmarks, and this phrase sums up the situation: “without HTC’s CPU cheats, it’s basically impossible to get the A57 cluster beyond 1.5 to 1.6 GHz”. You read the bolded bits right, there’s cheating going on despite the lowered benchmark outputs.

HTC enables a “High Performance Mode” upon detecting a benchmark, much like Samsung had done in the past by artificially topping core frequencies when firing up a benchmark – even when the benchmark wasn’t running. In their latest software iterations, however, said cheating had ceased.

With all this being said, is the phone slow? Not by any means. if you take a look at AnandTech’s tests you’ll see that in many aspects, the chip remains superior to last year’s iterations. The more holistic reviews also tout its speed in real-world usage. The Adreno 430 GPU in it is also an improvement over last years’ Qualcomm GPUs, and this is still seen in off-screen tests that disregard the M9’s sub-1440p resolution that SD805 devices typically had. Reviews from many big Android sites say that both temperature and real-world performance are up to par, and the phone is fast. Some reviewers (like AnandTech) did notice minor framedrops, but most Android phones have micro-stutters in some way or another.

So the HTC M9 is still a good phone. But what I personally believe is underplayed is the meaning that their benchmarks – and sacrifices – have in the evolution of smartphones. The SD810 was touted to be a huge step forward, to the point where we just had to question if it could deliver on the hype. And it didn’t, no matter how much an apologist fan, a manufacturer or an Android blog tries to spin it. Qualcomm is a renowned manufacturer, and they have made chipsets that powered some of our favorite devices – my daily driver is sitting next to me right now with a Snapdragon 805 in its guts, and I love it. But the transition to 64-bit, 8 cores without their Krait design, the big.LITTLE arrangement and smaller processes proved difficult. Their upcoming Snapdragon 820 will bring custom cores again and could easily redeem Qualcomm, as could their intermediate 815 or even revised 810s. But as of now, the Snapdragon 810 seemingly under-delivered in every metric.

The Non-issue

What is perhaps most surprising is the amount of websites calling this development a “non-issue” be it in reviews, editorials or comments with many stating that standardized tests and benchmarks are not to be trusted. In this case, the benchmark tells the story perfectly: early demo and test units showed higher scores, but overheated. A throttling cap was put in place, and now performance is capped at the cost of a cooler device. In every instance, the AnTuTu benchmark scores were lower than what last year’s Snapdragon 805-powered devices can output, even with lower resolution screens, DDR4 RAM, newer software and a superior GPU. The scores show the same trend in the G Flex 2, which also has a Snapdragon 810.

On every article where we present benchmark scores, we add a statement saying that we do not believe that benchmarks represent real-world usage, and they usually don’t. Being primarily a TouchWiz user, I would know. But that doesn’t mean that you can’t trust benchmarks, as they are the way we use to quantify peak performance for processors, and the only way of standardized testing that we have. Benchmarks are truly not about a “ranking”, even if enthusiasts like to compete for higher scores. They allow manufacturers in mobile to put their technology through their paces, just as it also allows computer scientists or engineers to do the same with their creations. Then, with the results, they can compare the output to adjust the software or hardware. When a benchmark score leaks ahead of time, it is not because an OEM employee was competing with internet strangers to raise his or her ego, and the leaks can sometimes show more harm than good.

Moreover, the fact that the limited peak performance of the Snapdragon 810 means little now doesn’t mean it won’t hurt the device later on. This will affect phones in a few years, when software gets more demanding. You can argue that you’ll be switching phones before that happens, but that still doesn’t justify having an under-delivered step in performance in a highly competitive business where other manufacturers are stepping up their game. With Samsung’s unparalleled technology in the S6 (which includes industry-leading memory solutions as well as the processor), squeezing every last bit of performance out of a chip is something that would benefit Qualcomm and OEMs more than ever. The fact that the chipset has to be limited to not cause issues is not optimum.

The Issue with the Non-Issue

So to recap: The Snapdragon 810 does bring constraints in physical phones, despite their original MDP/T model showing little of this in the past. The chipset in the M9 had its most powerful cluster’s frequency capped by 25%, and this had an impact on the peak performance that would give the phone extra longevity. HTC continued to cheat on benchmarks by boosting the frequency, despite that the ability to do so outside of the cheat code is virtually impossible without lower-level tweaking. The M9’s specification sheet will list the 2GHz maximum frequency, regardless of it never being attainable in real-world usage. HTC had also sent demo and test units that behaved differently than final consumer units, which added to the disinformation. Despite the misleading specification sheet, the downgrade in performance, the reduced longevity and the cheating going on behind the scenes, many claim that there is no issue. Furthermore, many claim benchmarks are not to be trusted.

It is clear that there’s some damage control going on in the blogosphere regarding this device. As an Android lover that devours the latest news I value Truth more than anything, and downplaying the value of benchmarks is a frequent pet peeve. Scores by themselves are misleading, yes. But if you look at the other aspects of the device, and research the components’ strengths and weaknesses as well as how they interact with other specifications, benchmarks can be very illuminating. Why else do the technical sites list every score with commentary on their meaning? They aren’t a linearly intelligible number, but that doesn’t discredit their importance in technology in general.

Is the M9 a good phone? It is, no doubt about that. But a lot of decisions HTC made regarding the internals and design do have negative consequences on the user experience. There’s many users and future adopters downplaying these consequences, however. Many are going as far as saying that they are “non-issues”. But when you read news about HTC changing CEOs in front of the controversy, or their lead designer leaving ship so early, it’s easy to realize that these things are, in fact, issues.

The post M9: Is Throttling a Non-issue? Can’t We Trust Benchmarks? appeared first on xda-developers.

source: xdadevelopers