OK. The great autumn phone glut is over. The big manufacturers have revealed their flagship devices. Apple even joined in with a cheaper alternative to its flagship iPhone. I’ve spent the last few months benchmarking a lot of phones, and something has become exceedingly clear: All the CPU and GPU benchmarks in the world can’t compare to real world performance when it comes to mobile gaming. And in the real world Apple stomps its Android competitors.
Benchmarking can be one of the most exciting things about reviewing consumer electronics. You get to take a device, run some tests and boil its entire worth down to a few numbers. If you live and breathe statistical analysis like I do, then this is damn rad. It’s like understanding an entire baseball game from a glance at the box score.
But while you can benchmark a whole slew of things on a computer, mobile devices have been notoriously more difficult to benchmark. Synthetic benchmarks, like the popular AnTuTu and Geekbench, can be “gamed” by clever device manufacturers and leave you with majorly skewed results. So you often find yourself looking to PC benchmarks and adapting them for the Android/iOS environment and the numerous complications it entails.
There was one particular aspect of benchmarking that proved, for years, to be impossible to carry over from PC benchmarking. It was mobile benchmarking’s white whale.
Game benchmarks.
Game developers don’t include any kind of frames per second counter for benchmarking on mobile games, and counting FPS manually requires significant alteration to code — usually in the form of rooting/jailbreaking. There was one company, GameBench, aggressively pursuing the problem, but it wasn’t until late last year — three years after the company formed — that it caught the whale and gave us access to all kinds of goodies, like FPS, CPU, GPU usage and even power consumption.
Unfortunately, the new benchmark has also highlighted a pretty major limitation in mobile game benchmarking — mobiles games have frame per second caps. Most mobile games are capped at 30fps, with some more intense games capped at 60fps. In part this is to help save battery power, but it also means that no matter how amaaaaazing the processor in your phone is, it will only ever do a maximum 60fps.
This is incredibly antithetical to how we understand game benchmarks. They’re often for bragging rights right? “My SLI’d Quad Titan rig does 400-plus frames per second on Grand Theft Auto V. WHAT NOW SON?!”
The iPhone SE, Apple’s tiny “budget” phone, features the same top of the line guts as the iPhone 6S, and it hits the cap. Every time. Even my two year old iPhone 6 has no problem maintaining 59fps in Lara Croft Go (a wicked fun game with some sweet, if gruelling, graphics).
But the the LG G5 — which features the same processor? 42 frames per second. Even the brand-spanking new HTC 10 managed only 44 frames per second.
These aren’t two-year-old devices. These are the very best phones from their respective brands and feature one of the fastest processors available in a mobile device. So why can’t they hit the cap?
Because mobile games are made for iOS first. Android is an afterthought — even though Android continues to crush iOS in terms of smartphone marketshare.
Developing for iOS tends to be more lucrative overall, and there’s very little device fragmentation. There are just a handful of processors and screen sizes that an iOS developer needs to program for, which means devs can optimise their apps better than the poor Android developer who has to take into consideration hundreds of different devices with a wide variety of processor and display capabilities.
Sharif Sakr of Gamebench agreed with my assessment, but he was also more hopeful for the future of Android gaming.
I think the iPhone’s supremacy is a legacy of how mobile has evolved, and doesn’t necessarily dictate the future. The Android platform is moving forward (e.g. with the Vulkan API, which is the Android equivalent to Metal), and developers and middleware providers are taking Android much more seriously as a source of commercial success, so there is every hope that Android will catch up.
Yet until Android actually catches up — or apps are specifically optimised for Android — benchmarking them to compare them to iOS devices will be an exercise in futility. Apple’s always going to win.
Sorry Android fanboys.