The present results suggest that there is room to improve reproducibility in psychology. Any temptation to interpret these results as a defeat for psychology, or science more generally, must contend with the fact that this project demonstrates science behaving as it should. Hypotheses abound that the present culture in science may be negatively affecting the reproducibility of findings.

Assuming a two-tailed test and significance or α level of 0.05, all test results of original and replication studies were classified as statistically significant (P ≤ 0.05) and nonsignificant (P > 0.05). However, original studies that interpreted nonsignificant P values as significant were coded as significant (four cases, all with P values < 0.06). Using only the nonsignificant P values of the replication studies and applying Fisher’s method (26), we tested the hypothesis that these studies had “no evidential value” (the null hypothesis of zero-effect holds for all these studies). We tested the hypothesis that the proportions of statistically significant results in the original and replication studies are equal using the McNemar test for paired nominal data and calculated a CI of the reproducibility parameter. Second, we compared the central tendency of the distribution of P values of original and replication studies using the Wilcoxon signed-rank test and the t test for dependent samples. For both tests, we only used study-pairs for which both P values were available.

Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error. Direct replication is the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding (7, 8) and is the means of establishing reproducibility of a finding with new data.