Why we shouldn’t trust news stories on scientific studies about what foods are healthy

by Alan Cohen

The blog newsofthetimes has a nice recent post on how new scientific studies are constantly changing conventional wisdom about how to eat healthily. As a health researcher who does some work with nutrition, I have often remarked on exactly the problem she points out, and I have asked myself who is at fault for our rapidly changing guidelines to healthy eating. I think there are largely 2 guilty parties. First is the media. They know that studies like this make good headlines, even when scientists view the results as tentative. They’ve got us jumping this way and that every time there is a new study, which is often.

The second guilty party is researchers themselves (ourselves). We all know that correlation is not causation, and most of the analyses we do go to great lengths to take this into account. But often these lengths are not great enough, or fail to appreciate how complex the world is.

Let’s say that I try to study the link between beta-carotene (a nutrient found in many fruits and veggies) and cancer risk. I conduct blood tests to measure beta-carotene and follow patients for years to see who gets cancer. Well, we all know that in modern western countries rich people eat more fruits and vegetables, and that being rich has lots of other intangible benefits which are likely to lower cancer risk. So we had better control for socio-economic status (SES), to make sure we don’t find a link between beta-carotene and lower cancer risk that is just due to the additional benefits of being rich.

Unfortunately, our measures of SES are imperfect (income and education don’t fully describe the cultural differences of class), so we are likely to only partially succeed in controlling for SES in the analysis. This problem is widely under-appreciated by researchers. At the same time, because rich people eat more fruits and veggies, when we control for SES we may inadvertently get rid of the statistical pattern we want to see – the link between consumption of fruits and veggies containing beta-carotene and cancer risk.

So after our (long, expensive) study, we publish results in some fancy journal showing that there is (or isn’t) an effect of beta-carotene on cancer risk. The results are highly significant, and the media eats up the story. The readers eat up their fruits and veggies, boosting sales of blueberry juice and acai extract. But in the end, we don’t know anything: if we find an effect, it could be just due to bias, because we haven’t sufficiently controlled for SES. If we don’t find an effect, it could be because we controlled away the real effect when we included SES in the model. So a few years later, someone else will come along, find the opposite, and conventional wisdom will change again.

We expect the effects of individual items in the diet to be relatively modest, and this makes the problem worse because we are unable to detect really strong effects of any one food.

My advice: read Michael Pollan, eat like our (ancient) ancestors, and ignore all health advice you hear in the news. The interactions of different foods are far too complex to be properly studied with the approaches normally used, the risk of unforeseen biases too great, and most such studies are a waste of money.