Using the internet to diagnose your latest health concern is, perhaps unsurprisingly, not all that effective. A new study from scientists in Australia tested more than two dozen so-called symptom checker apps and found that they correctly diagnosed a hypothetical person’s illness barely over one-third of the time. Apps designed to tell someone if they need emergency or urgent medical care, on the other hand, performed better.
The researchers used free symptom checker apps popular in Australia and worldwide for their study. These included apps developed by Drugs.com, the Mayo Clinic, and WebMD. Many of these are available through the Apple store and Android’s Google Play store, while others are available on the websites of several medical centres and hospitals, including Johns Hopkins. Some of the apps also included a triage function, meant to tell the user whether and how much their symptoms merited medical attention.
To test the apps, the authors ran them through 48 different patient vignettes, 30 of which had been used for an earlier study of symptom checkers published in 2015. The illnesses in these stories ran from being life-threatening, such as appendicitis or a heart attack, to needing urgent care, like pneumonia, to being short-lasting or easily treatable on their own, such as athlete’s foot or cold sores. It also included illnesses native to Australia, such as the tickborne disease Queensland tick typhus. The researchers did more than 1,000 individual tests of the apps to see if they picked the right diagnosis and over 600 tests to see if they urged the right level of triage care.
In total, 27 symptom checker apps were evaluated for their diagnostic prowess. On average, the apps listed the correct diagnosis as their first possible option 36 per cent of the time. When looking at the top three choices, they fared better, with 52 per cent accuracy. Expanding to the top 10 provided a hit rate only slightly better, at 58 per cent. No single app was significantly better than the rest, though some were far worse. The average accuracy for the first diagnosis ranged from 12 per cent to 61 per cent among individual apps.
The findings, published Monday in the Medical Journal of Australia, line up with previously research suggesting that these apps aren’t very accurate. That includes the 2015 study, which found that the apps suggested the correct illness first an average of 34 per cent of the time. So it looks like not much has changed since then.
“While it may be tempting to use these tools to find out what may be causing your symptoms, most of the time they are unreliable at best and can be dangerous at worst,” said lead author Michella Hill, a graduate student at Edith Cowan University, in a statement released by the university.
There are a few silver linings from Hill and her team’s study. Apps that relied on AI algorithms appeared to do better on average than those without them, though still not better than 50 per cent. And apps that included triage advice performed better at that function, ranging from 44 per cent to 54 per cent. This accuracy improved for conditions that needed emergency (63 per cent) or urgent care (56 per cent), but was only accurate a third of the time for less urgent problems.
The findings might be especially relevant now, given that the covid-19 pandemic has left many people often stuck at home wondering if they might have it. Online symptom checkers specifically meant for covid-19 have been released, but there isn’t any data available yet to measure just how accurate they really are. While the authors do think that these checkers may provide some benefits, they caution anyone from taking their results without a dose of scepticism.
“The reality is these websites and apps should be viewed very cautiously as they do not look at the whole picture—they don’t know your medical history or other symptoms,” Hill said. “For people who lack health knowledge, they may think the advice they’re given is accurate or that their condition is not serious when it may be.”