There could be a crisis in science, especially in the social and even medical sciences: Results are difficult to reproduce. Groundbreaking papers might reveal a first set of results, but attempts to repeat the studies reveal a different and sometimes even conflicting new set.
Image: US Air Force photo/Staff Sgt. Jerilyn Quintanilla
There are many factors as to why the same experiment might produce two different sets of results – but a new review confirms that the experimenter’s gender could play an important role. The researchers behind the review think that these biases could skew the results of clinical trials and misrepresent how well medications work. The authors suggest some policies, such as new kinds of reporting and experimental controls, that they hope will help the problem in the future.
“I think the big takeaway is that [experimenter gender] is something that should be reported and tracked,” Colin Chapman, the study’s first author from Uppsala University in Sweden told Gizmodo. “Right now it isn’t, and it’s a very simple thing to control and report for. There’s really no excuse.”
Chapman’s team’s paper in Scientific Advances finds a number of concerning correlations. Children seem to do better on IQ tests when experimenters are female. Some studies find that men report less pain when the experimenter is female, but women report more pain when experimenters are male. Researchers studying sex find that men report more sexual encounters when a woman administers the questionnaire.
Chapman’s study tries to explain the difference with a few hypotheses, such as the psychological and social stress of dealing with the opposite gender, or hoping to look more fit to a potential mate. Obviously these results are pretty rooted in the overarching majorities and social norms – but those social norms are biasing science.
The study concludes that it’s up to journals, funding agencies and universities to require that experimenter gender be reported, and that an analysis of potential gender effects is included in research. Chapman also suggested a 50/50 gender split when it comes to selecting experimenters. Perhaps one day, computerised experimenters could reduce the bias.
If this seems unsurprising, it should be. It’s been a known problem for decades, Robert Rosenthal, distinguished professor of psychology at the University of California, Riverside, told Gizmodo. He published a book about experimenters and how they could affect behavioural research back in 1966. Chapman’s new paper serves as a reminder and to offer solutions.
Other sources I spoke to still found the analysis important. But how much should study authors control the gender of experimenters? After all, there’s no controlling for the gender of the people you interact with in real life. “Replicability is a very important virtue of scientific experiments, but so is explanatory power and accuracy,” Carolyn Neuhaus, research scholar at the Hastings Center, told Gizmodo. “Standardising experimenter gender might increase replicability of studies, but then the experimental results may not adequately explain how human mind, body, and behaviour work in everyday life.” You can’t control the gender of the person who treats you in the emergency room, for example.
Then of course, there’s the fact that social interaction is way more complex than just gender. Gender isn’t really binary, for one thing – many people are intersex or gender fluid. (Chapman told Gizmodo that for now, it might be easier to report experimenter biological sex, rather than gender, and look to a more broader picture in the future). Then there are other factors at play, too. “I imagine race, ethnicity, age, that all of those things could have important effects on how research participants perform in a research study,” Kristina Gupta, assistant professor in women’s, gender and sexuality studies at Wake Forest, told Gizmodo.
Ultimately, science is not an exact truth, but a tool to help establish the truth. And all experiments have a major weakness: They’re not real life. “It’s often the best thing we have to go on, but it’s not infallible,” said Gupta. “You have to go based on the best available scientific evidence, but you should always be aware it’s not a complete truth of the world.”
As for reporting experimenter gender (or sex), it’s just another variable in the scientific experiment that could affect the outcome, said Pheobe Friesen, a placebo effect expert and PhD candidate in philosophy at the CUNY Graduate Center.
“The lesson there is that it’s important to control as many factors as possible across conditions, and to make sure to report what one is unable to control. It seems the lesson here is largely the same.”