Huawei's Nova 3, P20 and P20 Pro and the Honor Play have been delisted from the 3DMark synthetic benchmark rankings after a report found the devices artificially boosted performance when benchmarking tools were running.
According to a report by AnandTech, the Huawei phones detected when certain benchmarking software was active, and responded by boosting their power and battery usage well beyond ordinary performance.
Ignoring their thermal limitations, the phones would run in an unsustainable manner during benchmarking. Like when you hear your housemate’s car pull up and hurry to the kitchen so they’ll see you doing the dishes, this created an incorrect perception of overall performance.
After conducting their own tests and confirming the report, UL Benchmarks, the creators of benchmarking tool 3DMark, have removed several of Huawei's models from their performance rankings. In a press release, UL Benchmarks stated that they compared the phones' scores on the publicly available 3DMark app with their own private build.
According to UL Benchmarks (emphasis theirs):
We found that the scores from the public 3DMark app were up to 47% higher than the scores from the private app, even though the tests are identical.
The Huawei P20, Huawei P20 Pro, Huawei Nova 3 and the Honor Play have now been delisted, and appear on UL Benchmarks unranked and without scores.
Huawei's Dr Wang Chenglu responded to AnandTech by saying current smartphone benchmarks were limited, and that "others do the same testing, get high scores, and Huawei cannot stay silent".
"In the Android ecosystem, other manufacturers also mislead with their numbers," Dr Chenglu was quoted as saying.
All up, it sounds a lot like "benchmarks are stupid anyway" combined with the old "they did it first" chestnut. But while it isn't unheard of for manufacturers to pad benchmarks, that still doesn't make it OK.
Huawei more clearly admitted to the performance boosting in a subsequent statement to Android Authority, saying, "In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to 'Performance Mode' and delivers optimum performance."
Huawei further stated that it plans to bring "Performance Mode" to their phones, so users "can use the maximum power of their device when they need to".
But that Performance Mode isn't here yet, and it wasn't what was benchmarkers were led to believe they were assessing.
Gizmodo has reached out to Huawei for comment, and will update this article when we receive a response.
Update - September 8, 9:52AM AEST: Huawei and UL Benchmarks have sent us a joint statement, saying that Huawei's phones will be relisted once Performance Mode is rolled out:
Huawei and UL (creators of 3DMark) have held comprehensive discussions on benchmarking practices this week, and have reached a positive agreement on the next steps in working together.
In the discussion, Huawei explained that its smartphones use an artificial intelligent resource scheduling mechanism. Because different scenarios have different resource needs, the latest Huawei handsets leverage innovative technologies such as artificial intelligence to optimize resource allocation in a way so that the hardware can demonstrate its capabilities to the fullest extent, while fulfilling user demands across all scenarios.
UL understands the intent of Huawei’s approach, but is opposed to forcing the use of a “Performance Mode” by default when a benchmarking application is detected by the device. UL rules require a device to run the benchmark as if it were any other application.
Huawei respects consumers’ right to choose what to do with their devices. Therefore, Huawei will provide users with open access to “Performance Mode” in EMUI 9.0, so that the user can choose when to use the maximum power of their device.
Huawei and UL have also discussed current common benchmark testing methodologies in general. UL and Huawei would like to participate in an industry movement to develop benchmarking standards that best serve the needs of manufacturers, press, and consumers.
To prevent confusion around current benchmarking results, after discussion, UL and Huawei have temporarily delisted the benchmark scores of a range of Huawei devices, and will reinstate them after Huawei grants all users of Huawei handsets access to the Performance Mode.