Home > Uncategorized > Known knowns, unknown unknowns and the limits of science

Known knowns, unknown unknowns and the limits of science

January 22nd, 2011

Every once in a while, I’m contacted by product reps who offer to send me the latest high-tech sports gear to try out and review on the site. While I sincerely appreciate their offers, I decided when I started the site that I wouldn’t do first-person reviews. That’s partly so that readers can assume that when I do criticize or compliment a product, it’s an unbiased opinion. But it’s also because I really don’t place much value on n=1 experiments. (If I try a pair of compression tights, I can obviously judge whether they’re comfortable and come in pretty colours, but it’s impossible for me to tell in any meaningful way whether they made me faster or less sore. Or more precisely, if I run faster and feel less sore, it’s impossible for me know what really caused it.)

This question of what constitutes meaningful evidence is a constant undercurrent on this blog, and it occasionally pops to the surface. For example, I’ve been arguing about the fundamental mechanisms of weight loss for most of this week — and I think I generally agree with the guy I’m arguing with, we just differ on the extent to which the science is settled. Or, for a different take on the nature of evidence, here’s a few lines from a comment that popped up yesterday on an old post about “cryotherapy”:

[Y]ou see people getting in and out of them constantly… like a revolving door! Why would people put themselves in such cold temperatures if it doesnt help them with something?? I only went in once so I dont know if I reaped any long term benifits but I loved the way I felt after I got out! I couldnt stop giggling and I didnt know why! […] I hope you stubborn North Americans learn to see past your ignorance to the rest of the world and try something before you knock it

All of this is on my mind these days because of a few very interesting recent articles on the systematic problems inherent in the medical literature. (And let’s not kid ourselves, the sports science literature has a long way to go before it reaches even that level of unreliability!) So on that note, some suggested weekend reading:

If I could summarize the articles in a few lines, I would. But they’re not simple — particularly Lehrer’s article, which covers some pretty broad territory beyond the obvious problems of publication bias, study design, bad statistics, and so on. If it’s a topic that interests you at all, they’re both worth a read.

Finally, for a slightly more practical (and optimistic) take on how these somewhat abstract concerns intersect with real life, read Steve Magness’s post on “Science Vs. Practice: Should our training be evidence based?

(Thanks to Amby B., Ian R. and Amy M. for pointing these articles out to me.)

  1. No comments yet.
Comments are closed.