The company behind PCMark’s benchmarking rankings has delisted roughly 50 devices from its system after reports surfaced that their RealTek processors were boosting performance during testing. This includes phones from major manufacturers such as Oppo, Realme, Xiaomi and Sony.
This discovery was made by AnandTech, which found that certain phones containing RealTek processors could detect when benchmarking tools were active.
The journalists realised this when the source code for the processors revealed power management tweaks for certain apps. Amongst them was a variety of benchmarking tools such as GeekBench, AnTuTu, 3DBench, PCMark, Quadrant and Master Lu.
“We find the APK ID for PCMark, and we see that there’s some power management hints being configured for it, one common one being called a ‘Sports Mode’,” reports AnandTech.
The mode forced the devices to boost power capabilities beyond regular everyday performance and therefore provide more favourable benchmarking scores. When the journalists used a private, unnamed version of PCMark to retest the chipsets, the results were entirely different as the Sports Mode wasn’t automatically triggered.
UL, creators of PCMark, verified Anandtech’s findings with its own tests and has temporarily delisted over 50 devices from 25 vendors that contain the following RealTek Chipsets:
- MediaTek Helio G90
- MediaTek Helio G70
- MediaTek Helio P95
- MediaTek Helio P90
- MediaTek Helio P65
- MediaTek Helio P60
- MediaTek Helio P20
- MediaTek Helio A22
The following phones have been reportedly affected, with UL specifically mentioning the Realme 6, Oppo Reno3 Pro and Xiaomi Redmi Note 8 Pro in a release early Thursday morning.
- Oppo Reno3 Pro
- Oppo Reno Z
- Oppo F15
- Oppo F9 Pro
- Vivo S1
- Xiaomi Note 8 Pro
- Realme C3
- Realme 6
- iVoomi i2 Lite
- Sony XA1
UL is open to relisting these devices, which is why they are only temporarily delisted. There is also the potential for more devices to be impacted by the discoveries.
“We are investigating further since whitelisting at the SoC level potentially impacts dozens to possibly even hundreds of devices,” said a UL spokeperson in an email to Gizmodo Australia.
“If MediaTek starts following the rules and removes the optimisation, then we would look to relist devices that are updated. This is what we have done in the past with the manufacturers who have changed their ways.”
It’s unclear which devices may be relisted at this stage, but Oppo reportedly took strides to ensure accurate benchmark scores for the Reno3 back in February.
“Through our own testing we identified discrepancies between the benchmark scores of OPPO Reno3 Pro earlier this year and on the 14th of February removed the benchmark list and undertook our own internal testing. This resulted in an OTA update (version number: CPH2035PU_11.A.08_0080_202003010045) being pushed out to handsets on the 6th of March.”
MediaTek responded to the findings, defending the performance boosting modes by saying it showed the full capabilities of its chipsets.
“MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit.
Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.
Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimise power and performance to provide the best user experience possible while maximising battery life.
If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience.
Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.
We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.”
Realme has since echoed these sentiments.
“There are numerous independent benchmarking tests available depending on the market you operate in and as a brand, we have to respect all of them. But how you measure performance is challenging, especially when modern chipsets and smartphones are designed to dynamically adjust to the user behaviourAnandTech made a similar argument, pointing out that MediaTek’s argument ignores benchmarks that test more than just the chipset, like a phone’s storage or filesystem speed.
“[The argument] falls apart in the face of cheating benchmarks that not only target the actual hardware components of a SoC ” like how GeekBench is testing the CPU speeds or how GFXBench checks out the how fast a GPU can be, but also benchmarks which actively try to be user experience benchmarks, such as PCMark. This is a real-world mimicking workload that tries to convey the responsiveness of a phone as a whole, not just the chipset,” said Anandtech.
UL also took issue with MediaTek’s response, particularly around the claim that it follows accepted industry standards and that the tests accurately represent the capabilities of MediaTek chipsets.
“Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimisations is not an ‘accepted industry standard.’ It is, in fact, the very opposite of the accepted standard,” UL said.
The benchmark creators delisted Huawei’s Nova 3, P20 and P20 Pro and Honor Play phones from its 3DMark benchmarking rankings in 2018 after similar artificial boosting was found.
This story was originally published on April 16. It has been updated to include statements from UL, Realme and Oppo.