The Streetlight Effect
This came up in the comments of a previous post, but I thought it was worth a post of its own. I was trying to explain the dangers of the “Streetlight Effect,” and came across this article from Discover magazine by David Freedman, the author of Wrong, which does a very nice job of explaining it:
The fundamental error here is summed up in an old joke scientists love to tell. Late at night, a police officer finds a drunk man crawling around on his hands and knees under a streetlight. The drunk man tells the officer he’s looking for his wallet. When the officer asks if he’s sure this is where he dropped the wallet, the man replies that he thinks he more likely dropped it across the street. Then why are you looking over here? the befuddled officer asks. Because the light’s better here, explains the drunk man.
How does this relate to scientific research? It points out that researchers are always likely to focus on quantities they can measure (i.e. where the light is good), regardless of whether they’re the most important. An example:
A bolt of excitement ran through the field of cardiology in the early 1980s when anti-arrhythmia drugs burst onto the scene. Researchers knew that heart-attack victims with steady heartbeats had the best odds of survival, so a medication that could tamp down irregularities seemed like a no-brainer. The drugs became the standard of care for heart-attack patients and were soon smoothing out heartbeats in intensive care wards across the United States.
But in the early 1990s, cardiologists realized that the drugs were also doing something else: killing about 56,000 heart-attack patients a year. Yes, hearts were beating more regularly on the drugs than off, but their owners were, on average, one-third as likely to pull through. Cardiologists had been so focused on immediately measurable arrhythmias that they had overlooked the longer-term but far more important variable of death.
This particular example brings to mind last month’s discussion about the increased incidence of arrhythmias among elite cross-country skiers. Should we be worried about those arrhythmias? Or should we focus on the “more important variable of death,” since studies of the same skiers found that more skiing led to longer life?
More generally, the Streetlight Effect is one of the key reasons why extrapolating “real world advice” from lab studies has such a low batting average. Last week, I blogged about an interview with Asker Jeukendrup, a sports scientist who I have a tremendous amount of respect for. But I disagreed with one of his comments: “I think if the physiological changes are there, the performance must ultimately follow.” All it takes is a stroll down the supplements aisle of a pharmacy or health-food store to remind me that promising physiological changes don’t always translate into measurable health or performance benefits.