Taking cancer biology as an example: The importance of reproducibility in science

20170126110000, Brenda Lau
Scientists expressed concern and urged the scientific community to take reproducibility projects seriously.
Many scientists have called for more negativity in science, to enhance the quality of science. Recently, preliminary results of a major project have reemphasised why this matters. The project measured the reliability of cancer research, but found that laboratories trying to repeat published experiments, more often than not, cannot.

This does not mean that the original studies are wrong, however; it just serves as a reminder that science often fails at one of its most fundamental requirements, that an experiment in one lab, should be able to be reproduced in another - and this is dangerous as it could have big health implications.

"Reproducibility is a central feature of how science is supposed to be," says Brian Nosek, who led the research at the Centre for Open Science.

Only two out of five studies could be reproduced

Nosek looked at cancer biology labs after two high-profile studies, from pharmaceutical companies Bayer and Amgen, reported dismal results when they attempted to reproduce a few cancer studies. Only 25% of the papers Bayer examined were reproduced and Amgen could only replicate six out of 53 studies.

"Those were earthshaking reports, in the sense that the community responded very strongly to these reports of challenges to reproduce some of these core findings," Nosek says. However, Bayer's and Amgen's scientists would not disclose which experiments were examined, so their work raised more questions that other scientists could not answer.

Nosek's project on the other hand, showed how it picked and which studies were reproduced, as well as publishing methods and study plans in advance - providing more substance for discussion. The results of the first five attempts have been published.

"Three of the five show very, very striking differences from the original," says Timothy Errington, a biologist at the Centre for Open Science and collaborator in the project. One could not generate the same results, while the other two are debatable.

In one case, scientists of the original research offered the same drug and tumour cells that was used to reduce potential sources of error. However, the replicating lab ended up with different results. The scientists were so confident in two of the original studies that drug companies have already invested millions of dollars for clinical trials, yet the follow-up experiments for one of those could not validate the original results.

So was the original science wrong, or did the scientists who tried to reproduce the result make an error?

Initial findings can still be correct, despite irreproducibility

The review project carried out experiments in commercial labs or university "core facilities", which does a lot of research on mice and work to standards required by the FDA. However, researching with live organisms is not simple as there are many possible sources of variation - animals or cells or details of lab technique. Exactly when a study's findings can be considered to have been reproduced is vague as well.

"One of the difficulties of the reproducibility project is they have limited time and resources to spend on any one study," says Sean Morrison, a Howard Hughes Medical Institute investigator at the University of Texas Southwestern Medical Centre. "As a result, they can't go back and do these things over and over again when the first results turn out to be un-interpretable."

Therefore Errington says that a single failure to replicate results does not prove that initial findings were wrong, and it should not put a stain on individual papers.

"If we see someone else's evidence as making it hard for the person who did the original research, there is something wrong with our culture," he said.

The need to take reproducibility projects seriously

However, for Dr. Erkki Ruoslahti, at the non-profit Sanford Burnham Prebys Medical Discovery Institute in California, the failure to reproduce his results might affect funding for the clinical development of DrugCendR, his drug.

He also argues, "If we, and the many laboratories who have reproduced our results, are right and the reproducibility study is wrong - which I think is the case - they will not be doing a favour to cancer patients."
Ruoslahti and other scientists expressed concern and urged the scientific community to take reproducibility projects seriously.

“Careers are on the line here if this comes out the wrong way,” says Atul Butte, a computational biologist at the University of California, San Francisco, whose own paper was validated by the replication team.

Unfortunately, Nosek says, there are very few incentives for scientists to repeat experiments of other studies. The glamour and fame lies in publishing new ideas and results. He argues that if the scientific community would consider taking reproducibility seriously, attempts to reproduce the findings of others should firstly be, valued as part of scientific contribution. MIMS

Read more:
Online tool reveals up to 100% of research trials are never disclosed
Only 16% of medical news found to have independent expert commentary
Fraudulent data unveiled in 33 research publications authored by Japanese scientist

Sources:
http://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304
http://www.npr.org/sections/health-shots/2017/01/18/510304871/what-does-it-mean-when-cancer-findings-cant-be-reproduced
https://www.statnews.com/2017/01/18/replication-cancer-studies/
https://www.sciencenews.org/article/cancer-studies-get-mixed-grades-redo-tests