Activity trackers, whether standalone devices like the Fitbit or as a feature of other wearables like Apple Watch, are a popular health and fitness trend among consumers, and scientists and doctors have increasingly been using activity trackers in their research or to monitor patients away from the office. But a new study published last week in PloS-One suggests – not for the first time – that these devices aren’t as reliable as they claim.
European researchers recruited 15 healthy men and women (all white) to cycle on a stationary bike for 10 minutes. As they cycled, their heart rate was measured by the Fitbit Charge 2, one of the company’s most recently released products, as well as by an electrocardiograph (ECG) machine. ECGs measure the electrical activity of a heartbeat via electrodes placed near the heart, and are considered the gold standard for heart rate measurement.
The researchers found heart rate measurements taken by the Fitbit Charge 2 were routinely different than those taken by the ECG. The average heart rate recorded by the Fitbit was around 6 beats per minutes (BPM) slower than the average collected by ECG. But some individual measurements overestimated the ECG by as much as 16.8 BPM, and underestimated the ECG by as much as 28.5 BPM.
The study is the latest to indicate activity trackers aren’t doing a great job at measuring at least some kinds of activity. The researchers cite small studies that have found older trackers can underestimate an ECG reading on average by anywhere from 2.5 BPM to 9.3 BPM. Other studies have also found trackers can underestimate calories burned during exercise, too.
Though Fitbit’s specific algorithm for tracking heart rate is a trade secret, they and other activity trackers generally rely on measuring how light is differently absorbed by the skin as blood flows underneath it, which is known as a photoplethysmogram (PPG). While PPGs are an accurate, low-cost method for measuring someone’s resting heart rate, the researchers and others argue that they start to flounder when measuring someone in the middle of moderate to intensive exercise.
Other researchers have argued back that these performance lags, even if real, are relatively minor and still represent a significant improvement over past consumer devices that measure heart rate.
In 2016, Fitbit was hit by a class-action lawsuit, filed by lawyers from the legal firm Lieff Cabraser. The lawsuit argued that the alleged inaccuracy of Fitbit’s Charge and Surge device was grounds for “economic injury” and could even be dangerous for people who mistakenly push themselves too far while exercising. Fitbit has said the lawsuit has no merit, while also stating that its products aren’t meant to be used as medical devices.
“This appears to be another study commissioned by Lieff Cabraser, the plaintiff’s lawyers who are suing Fitbit, which casts serious doubts into the study’s inherent bias in terms of design methodology and interpretation of the findings,” Fitbit said in a statement to Gizmodo, citing a press release issued by the firm touting the study findings.
Lieff Cabraser did commission a 2016 study from California State Polytechnic University that found the earlier Fitbit Charge underestimated ECGs by 10 per cent when measuring users whose heart rates jumped past 132 BPM. But Jonathan Selbin, one of the attorneys involved in the lawsuit, told Gizmodo that the law firm was not involved in the new study.
“Zero, zippo, zilch. We had nothing to do with it at all. The journal forwarded us a copy because they thought we’d be interested given our lawsuit,” Selbin said via email.
The statement from Fitbit added, “We stand behind our heart-rate monitoring technology and all our products. Fitbit is the most trusted wearable by scientists and clinical researchers across the world and, to date, more than 500 published studies have utilised a Fitbit device, which is more than any other consumer wearable brand.”
Simone Benedetto*, lead author of this study and a user experience researcher at the TSW Experience Lab in Italy, says the reason why he felt compelled to perform his research in the first place was because of his own experience with the product. “I used the device and realised it did not perform so well,” he told Gizmodo via email.
“These devices measure a vital parameter, which is heart rate,” said Benedetto. “[W]e cannot play with it… the accuracy is crucial!”
He said that while the TSW Experience Lab provided the team with research materials, she reiterates it had no role in the study design or their decision to publish the findings. He also denied that Lieff Cabraser provided him with any funding, calling Fitbit’s claim “ridiculous.”
“Many people who bought these devices mistrust them as well as many others employing PPG,” he said. “We simply provided scientific evidence regarding its accuracy.”
According to Selbin, the law firm is still pursuing its civil case against Fitbit, on behalf of plaintiffs who have opted out of an arbitration clause with the company.
The researchers, in their paper, say they plan to pursue larger trials that will test a variety of activity trackers currently on the market.
[PloS-One]
*This article previously referred to Simone Benedetto using female pronounces, but Benedetto is in fact male. We regret the error.