Science is a messy, error fraught business, which is why reproducibility is so essential. Unfortunately, that doesn't appear to be one of psychology's strong suits, according to a massive analysis published yesterday in Science.
A years-long effort to reproduce more than 100 psychology studies across three leading journals paints a pretty dismal picture. When re-tested by independent research psychologists, the conclusions of more than 60 studies on personality, relationships, learning, and memory, turned out to be far less whelming. Strongly significant findings often became weaker, while weakly significant findings became non-existent.
"I think we knew or suspected that the literature had problems, but to see it so clearly, on such a large scale — it's unprecedented," statistician Jelte Wicherts of Tilburg University in the Netherlands told The New York Times.
In other words, depressing as this sounds, the news isn't totally surprising. In the past few years, the Times notes that the credibility of the social sciences has been bruised over and over:
A star social psychologist was caught fabricating data, leading to more than 50 retracted papers. A top journal published a study supporting the existence of ESP that was widely criticised. The journal Science pulled a political science paper on the effect of gay canvassers on voters' behaviour because of concerns about faked data.
But most of the published works examined in the new Science paper were not maliciously manipulated. Rather, on re-analysis, the findings simply weren't as strong or convincing as the authors had originally claimed. There are a lot of possible explanations here, including issues with sample size and experimental design, or misinterpretation and misuse of statistical tools. These are problems that have long plagued experimental, data-driven sciences, from economics to cancer research:
Dr. John Ioannidis, a director of Stanford University's Meta-Research Innovation Center, who once estimated that about half of published results across medicine were inflated or wrong, noted the proportion in psychology was even larger than he had thought. He said the problem could be even worse in other fields, including cell biology, economics, neuroscience, clinical medicine, and animal research.
And that's something that should worry us much more than a few bad apples. If the tools of tools of scientific inquiry are being misused on the regular, it calls an awful lot of knowledge into question. Knowledge, mind you, that's already made its way into textbooks, that doctors use to prescribe medication, and that psychologists use to treat patients.
So, do we throw up our hands in despair and renounce the scientific institution? Certainly not. But there's an important wake-up call here. If hundreds of studies have all come to the same conclusion, then there's a pretty good chance we can believe it. But if one study makes the cover of Science for a splashy, new, radical claim that no one else has verified — then we should all be sure to keep our excitement tempered, and remember that scientists are human beings, too.
Top image via Shutterstock