How lasting are scientific findings? In the previous blog
post, I quoted Simon LeVay from his book, When
Science Goes Wrong, where he suggests that it would be interesting to look
at a random collection of scientific papers from a couple of decades or so ago and
ask, “Were they right in their main findings and conclusions, and were they as
original as their authors claimed?” This post will deal with the first
question.
This topic has been receiving some attention in the past few
years, especially in the fields of medicine and nutrition. Two years ago The Atlantic magazine ran a fascinating feature article on John Ioannidis, a Greek medical school professor, who has
made a career of studying the staying power of medical research findings (Freedman, D. H. “Lies, damned lies, and medical
science.” The Atlantic, 306(4), p.
76-86, Nov. 2010). In a
nutshell, what he has discovered is that for medical articles, the answer to
LeVay’s question, “Were they right in their main findings and conclusions”, is
usually no. In fact, Ioannidis came to prominence with the publication of a thorough analysis in PLOS Medicine
with the blunt title, “Why most published research findings are false” (vol. 2(8), e-page e124, 2005, DOI: 10.1371/journal.pmed.0020124).
He comes to several interesting conclusions, including “The greater the
financial and other interests and prejudices in a scientific field, the less
likely the research findings are to be true,” (perhaps not so surprising) and “The
hotter a scientific field (with more scientific teams involved), the less
likely the research findings are to be true.”
Way back in 1972, physician-author William A. Nolen had
something interesting and revealing to say about medical research results in
his book A Surgeon’s World (Random
House, 1972):
There is, however, one trap into which
doctors engaged in research fall with some frequency: that of deciding
prematurely that what they want to be true is true. They formulate some theory
that sounds interesting, and which if true, will be of great value to medicine,
and then set out to prove it is true. In the university hospitals it's sort of
a joke—which, like most jokes, has an element of truth to it—that when the
professor sends his interns and residents into the dog lab to investigate one
of his new ideas, it's their job to prove he's correct. The assistant resident
who comes out of the dog lab with a pile of data proving that the professor's
theory is all wet is not apt to become chief resident. [pp. 240-241]
(This book is one of my favorites from way back, but
unfortunately we do not have it at the University of Regina library and it’s
out of print. You can, however, get it through interlibrary loan.)
A related phenomenon which has been seen in many fields is
the decline effect, where the strength of research findings appears to decline
over time. For example, the therapeutic power of some medicines has seemed to
diminish as time has passed. A fascinating aspect of this phenomenon is that
individual researchers sometimes see this in their own research as they
continue to study an observed effect. An article in the New Yorker
by Jonah Lehrer (he of fabricated-Dylan-quotes fame; see the previous blog entry)
discusses this in detail (“The truth wears off,” Dec. 13, 2010). Many
explanations have been proposed, but whether any of them, or all of them
together, are sufficient to explain the decline effect, is very much open to
question.
In an article in Nature,
University of California-Santa Barbara psychology professor Jonathan Schooler,
who has personally experienced the decline effect in his own research, proposes
a database that would help get a handle on the mystery. He includes this
intriguing tidbit: “Although the laws of reality are usually understood to be
immutable, some physicists, including Paul Davies, director of the BEYOND:
Center for Fundamental Concepts in Science at Arizona State University in
Tempe, have observed that this should be considered an assumption, not a
foregone conclusion.” (Schooler, J. “Unpublished results hide the decline effect.”
Nature, 470(7335), p. 437, Feb. 24,
2011.) The universe might be stranger than we think.
