Stats 101: interpreting medical studies


As of September 2017, new Sweat Science columns are being published at Check out my bestselling new book on the science of endurance, ENDURE: Mind, Body, and the Curiously Elastic Limits of Human Performance, published in February 2018 with a foreword by Malcolm Gladwell.

- Alex Hutchinson (@sweatscience)


I have a new article over at the Men’s Health website that tries to explain a bit about how the process of “statistical adjustment” works in large observational studies, and what some of the pitfalls are. My favourite example from the piece: a 2009 study of over half a million people that found that red meat intake is associated with increased risk of death — even after “adjusting” for potentially confounding factors like age, education, race, BMI, smoking, exercise, vegetable consumption and so on. Needless to say, that study got lots of press at the time. But when you dig into the study’s stats (as Stanford prof Kristin Sainani pointed out to me), you find out that red meat also increases your risk of sudden accidental death from causes like car crashes and guns!

“Unless red-meat eaters are swerving to avoid cows, that doesn’t make any sense,” Sainani says. Instead, the most likely explanation is that red-meat eaters take more risks in other areas of life. But the study didn’t collect any data on the driving habits and gun collections of its volunteers, so the researchers were unable to adjust for these factors—and as a result, the conclusions were skewed.

Another interesting nugget is a rough estimate of the size of effect needed to be fairly sure you’re not seeing the effects of residual confounding, according to statistical simulations:

Bad data can easily generate an apparent risk increase of up to 60 percent, according to a research paper on statistical adjustment published in the journal PM&R. Effects bigger than that are very difficult to explain without serious errors in the design of the study.

Anyway, check out the whole article for more!