This does not mean that the original studies are wrong, however; it just serves as a reminder that science often fails at one of its most fundamental requirements, that an experiment in one lab, should be able to be reproduced in another - and this is dangerous as it could have big health implications.
"Reproducibility is a central feature of how science is supposed to be," says Brian Nosek, who led the research at the Centre for Open Science.
Only two out of five studies could be reproducedNosek looked at cancer biology labs after two high-profile studies, from pharmaceutical companies Bayer and Amgen, reported dismal results when they attempted to reproduce a few cancer studies. Only 25% of the papers Bayer examined were reproduced and Amgen could only replicate six out of 53 studies.
"Those were earthshaking reports, in the sense that the community responded very strongly to these reports of challenges to reproduce some of these core findings," Nosek says. However, Bayer's and Amgen's scientists would not disclose which experiments were examined, so their work raised more questions that other scientists could not answer.
"Three of the five show very, very striking differences from the original," says Timothy Errington, a biologist at the Centre for Open Science and collaborator in the project. One could not generate the same results, while the other two are debatable.
In one case, scientists of the original research offered the same drug and tumour cells that
So was the original science wrong, or did the scientists who tried to reproduce the result make an error?
Initial findings can still be correct, despite irreproducibilityThe review project carried out experiments in commercial labs or university "core facilities", which does a lot of research on mice and work to standards required by the FDA. However, researching with live organisms is not simple as there are many possible sources of variation - animals or cells or details of lab technique. Exactly when a study's findings can be considered to have been reproduced is vague as well.
"One of the difficulties of the reproducibility project is they have limited time and resources to spend on any one study," says Sean Morrison, a Howard Hughes Medical Institute investigator at the University of Texas Southwestern Medical Centre. "As a result, they can't go back and do these things over and over again when the first results turn out to be un-interpretable."
Therefore Errington says that a single failure to replicate results does not prove that initial findings were wrong, and it should not put a
"If we see someone else's evidence as making it hard for the person who did the original research, there is something wrong with our culture," he said.
The need to take reproducibility projects seriouslyHowever, for
He also argues, "If we, and the many laboratories who have reproduced our results, are right and the reproducibility study is wrong - which I think is the case - they will not be doing a favour to cancer patients."
Ruoslahti and other scientists expressed concern and urged the scientific community to take reproducibility projects seriously.
“Careers are on the line here if this comes out the wrong way,” says Atul Butte, a computational biologist at the University of California, San Francisco, whose own paper was validated by the replication team.
Unfortunately, Nosek says, there are very few incentives for scientists to repeat experiments of other studies. The glamour and fame
Online tool reveals up to 100% of research trials are never disclosed
Only 16% of medical news found to have independent expert commentary
Fraudulent data unveiled in 33 research publications authored by Japanese scientist