As a purveyors of supposedly factual content, it’s the job of science journalists to make sure we’re accurate about how bad things like trans fats actually are. A whole lot of our fellow fact-farters seem to have dropped the ball on that one this week.
Image: wackystuff/flickr
Lots of places wrote about a single new science study that came out on Wednesday saying that banning trans fats in a handful of New York counties led to a decrease in heart attacks and strokes. “Trans Fat Ban Saved Lives in New York, Study Shows,” was one headline. “NYC’s trans-fat ban has actually started saving lives” says another. In reality, the study is so full of confounding factors that, while its authors have done lots of data analysis, they’re not clearly demonstrating a successful trans fat ban. The news coverage needs to be better about reporting studies’ limitations.
“If this was submitted to a nutrition journal it would have been rejected,” David Klurfeld, a national program leader at the USDA in many nutrition and food safety studies told Gizmodo. He saw all the coverage this story got the other day and was similarly surprised when he read the actual paper.
Regular fatty acids have long chains of carbon atoms at the end, each linked with one or two connectors. Saturated fatty acids form straight lines because all their carbon atoms are connected with a single connector. Unsaturated fatty acids have pairs of connectors which might add kinks in their chains. Trans fatty acids are unsaturated fatty acids that still form straight lines. They’re found naturally in small amounts in foods like dairy, but a hundred years ago food companies started forming them via a process called hydrogenation, which gets rid of some of the double connectors and straightens the chains. Generally, the more straight chains, the more likely something will be solid at room temperature, so over the past century companies have used hydrogenated trans fats to make solid fats like margarine or to replace lard and beef tallow.
Research has since shown that hydrogenated trans fats seem to cause an increase in LDL (the bad) cholesterol levels, and other inflammatory responses, and it’s noted that these fats could be contributing to cardiovascular disease. The FDA doesn’t recognise trans fats from partial hydrogenation as safe, so the molecules won’t be allowed in American food unless otherwise authorised starting in 2018.
But it’s important to note that researchers aren’t sure how bad trans fats really are. In controlled feeding tests using hydrogenated trans fats, participant’s “LDL cholesterol goes up and HDL (the good) cholesterol goes down. Not a large amount, probably not enough to account for the supposed risk of trans fat for heart disease,” said Klurfeld. As for the health effects of diary-based trans fat, the science isn’t conclusive yet.
Regardless, five boroughs of New York City still restricted trans fats in eateries (restaurants, cafeterias, soup kitchens, etc.) starting in 2007. NYC suburbs of Westchester and Rockland Counties, as well as Long Island, Albany, and Broome Counties (which includes the college city of Binghamton) joined in the years following. In the new study, researchers wanted to know if those bans did anything.
So, they collected lots of data from New York State databases on heart attacks, strokes and deaths in hospitals from 2002 to 2013 and compared counties that did and didn’t restrict trans fats. They tried their best to correct for commuters, existing disease trends, how urban a county was, and other factors, and published their results in JAMA Cardiology on Wednesday.
When they looked at counties with and without regulations, the researchers found that heart attack and stroke rates decreased across the board, regardless of the trans fat restrictions, during the years from 2002 to 2013. The combined rates in counties with trans fats regulations decreased around six per cent during the time after 2006. This effect was only statistically significant, meaning there was a less than a five per cent chance it could have happened randomly, in the case of heart attacks. It was not statistically significant for strokes.
The comprehensive study analyses a ton of data and does lots to correct for confounding factors. For instance, it track patients based on where they’re from, rather than the hospital they got admitted in — that means that if they got sick on a trip to New York or commuted between zones, they get counted in the right place.
But the authors themselves also list so many potential complicating factors! They couldn’t, for instance, track whether populations in their study actually ate less trans fat over time. Race and ethnicity aren’t well-reported in the data. The populations within the regulated versus non regulated counties have changed over time, too. Were changes in household income what drove down heart attack rates? Was there better access to healthcare in counties that happened to have trans fats restrictions? Doesn’t it sound weird that a change in restaurants would have a long-term health effect so soon? And what if trans fats were the problem, but those people had just been avoiding trans fats given the bad press?
Trans fats are bad. I have no opinion about banning them. I’m just saying that single observational studies on how foods change our health don’t prove that bans are saving lives. Klurfeld agreed.
“Your’e exactly right. I saw the headlines, I heard it on the radio, I looked at it on the abstract and it doesn’t show anything,” he said. You can’t just say, these new regulations are in place so people must be eating less trans fats and look, now there are fewer heart attacks.
I emailed the first author of the paper, Eric Brandt from Yale University. He thought that the news coverage was mostly accurate, but did point out that “a few reports I read discussed the results of my study regarding deaths and preventing deaths, which we did not study.” He also discusses how they removed New York City in one analysis, which he thought demonstrated that any New York City-specific issues, like the smoking ban, didn’t have an effect on the results, which should strengthen their conclusions.
But, he said, “any study could have further discussion on the drawbacks.”
So, don’t freak out, and if you see other people freak out, maybe give the researchers a call or an email (their email address is almost always in the paper, which is almost always linked in the news story) and ask them what the limitations of their study are.