A large scale study has identified what could be a big problem in psychological research – a lack of reproducibility.

The study sought to replicate 100 findings published in three prominent psychology journals, but found that across multiple criteria independent researchers could replicate less than half of the original findings.

In some cases it calls into question the validity of the findings, but it may also show how hard it is to conduct effective replications and achieve reproducible results.

It also highlights the media-driven distortion of science and science journals, which creates a world in which exciting findings are more likely to get coverage than accurate ones.

The review study was conducted by more than 270 researchers on five continents, and all the data has been made available at the official site for the Reproducibility Project: Psychology. 

Reproducibility means that the results recur when the same data are analysed again, or when new data are collected using the same methods.

Science is unique from other ways of gaining knowledge because it relies on reproducibility to gain confidence in ideas and evidence.

So, researchers around the world established an initial estimate of the rate of reproducibility in psychology, and identified evidence of possible influences on reproducibility.

It is one of the most basic scientific tenets that credibility accumulates through independent replication and elaboration of ideas and evidence.

But the experts say failure to reproduce does not necessarily mean the original report was incorrect.

The replication teams needed to have a complete understanding of the methodology used for the original research, and be aware of shifts in the context or conditions of the research. This can be a serious pitfall, and any small change could be unrecognised but vital for observing the result

The study shows that a problem for psychology, as well as in other disciplines, is that incentives for scientists are not consistently aligned with reproducibility.

“Scientists aim to contribute reliable knowledge, but also need to produce results that help them keep their job as a researcher,” said Brian Nosek, a psychology professor and coordinator of the study.

“To thrive in science, researchers need to earn publications, and some kinds of results are easier to publish than others, particularly ones that are novel and show unexpected or exciting new directions.”

As a consequence, according to Nosek and his co-authors, many scientists pursue innovative research in the interest of their careers, even at the cost of reproducibility of the findings.

The authors say that research with new, surprising findings is more likely to be published than research examining when, why or how existing findings can be reproduced.

Overall, the Reproducibility Project team successfully replicated fewer than half of the original findings. Investigators suggested this could be due to three basic reasons:

1. Though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where or how the replication was carried out might have influenced the results.

2. The replication might have failed, by chance, to detect the original result.

3. The original result might have been a false positive.

The findings demonstrate that reproducing original results may be more difficult than many assumed, and that more openness and transparency is needed to improve reproducibility.

The journal Nature is already on this wagon, running a long series on reproducibility in many fields.