Fact-checking medical claims
In 2007/08, the work of Nicholas Christakis and James Fowler revealed that human behaviors, and even states of mind, tracked through social networks much like infectious disease.
Or put another way, both obesity and happiness worm their way into connected communities just like the latest internet meme, the best Charlie Sheen rumors, or the workplace gossip about Johnny falling down piss-drunk at the company’s holiday party.
But according to a new research study, incorrect medical facts may be no different, galloping from person to person, even within the confines of the revered peer-reviewed scientific literature. And by looking at how studies cite facts about the incubation periods of certain viruses, a new study in PLoS ONE has found that quite often, data assumed to be medical fact isn’t based on evidence at all.
How many glasses of water are we supposed to drink each day? Eight – everyone knows it’s eight. But according to researchers from the schools of Public Health and Medicine at Johns Hopkins University, this has never been proven true. In fact, they argue there’s not one single piece of data that supports this claim.
Digging a little deeper, the research team dove into scientific papers looking for places where researchers quoted the incubation period of different viruses, from influenza to measles. Every time a claim was made, they traced the network of citations back to the original data source (and provided a cool visualization of the path, to boot). For example, many studies will set the stage for their own research by saying that it’s commonly known that the incubation period for influenza is 1-4 days, and next to that statement, they’ll put a small reference in parenthesis, which signals where they obtained that information.
The problem is, many articles cited another study, that cited another study, which in turn cited yet another – you get the picture. It’s like a really bad version of the “telephone game" played by kids. And 50% of the time, the researchers found no original source of incubation period data when they started backtracking. Scary stuff.
By factoring in review articles, which are supposed to be a comprehensive analysis of a field of research, the team found that 65% of viral incubation data never gets cited again after its first publication. 65%! Granted, review articles have to factor in the quality of the research done in individual experiments. So is that much crappy research being done, or is the majority of science in this particular arena simply falling into the growing chasm of “dark data”?
I’ve been chewing on this article for a while, waiting for the right time to write something about it. Today, a tweet by Nieman Lab caught my attention, and spurred me into action.
The tweet pointed to a post on Doc Searls’ blog asking media outlets to do a better job linking to original sources (I, like Searls, get super-frustrated with the NYT, when they either don’t link to a source, or you click on the underlined blue text thinking you’ll be enlightened by profound insight, only to find you’ve been swept away to some vaguely-related post authored by another NYT staffer).
Time to add scientists to your list of offenders, Doc.
Photo via Flickr / Dan Zen
Citation: Reich NG, Perl TM, Cummings DAT, Lessler J, 2011 Visualizing Clinical Evidence: Citation Networks for the Incubation Periods of Respiratory Viral Infections. PLoS ONE 6(4): e19496. doi:10.1371/journal.pone.0019496
** Update, 18 May 2011: The statistics cited in this post (50% of original data not traced back to source, 65% of studies never cited again) apply, in this case, to viral incubation data only. The authors didn't extrapolate these findings to other medical claims. I updated the statements above to make this explicitly clear. -bjm
Brian Mossop is currently the Community Editor at Wired, where he works across the brand, both magazine and website, to build and maintain strong social communities. Brian received a BS in Electrical Engineering from Lafayette College, and a PhD in Biomedical Engineering from Duke University in 2006. His postdoctoral work was in neuroscience at UCSF and Genentech.
Brian has written about science for Wired, Scientific American, Slate, Scientific American MIND, and elsewhere. He primarily cover topics on neuroscience, development, behavior change, and health.
Contact Brian at brian.mossop@gmail.com, on Twitter (@bmossop), or visit his personal website.