In 2010, Fanelli reported a positive result rate of 91.5% for psychology papers; these authors found only 42.65% positive results papers in the Registered Reports they reviewed

Scheel, A. M. (2019, March 12). Positive result rates in psychology: Registered Reports compared to the conventional literature. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2390

Abstract
Background: Several studies have found the scientific literature in psychology to be characterised by an exceptionally high rate of publications that report 'positive' results (supporting their main research hypothesis) on the one hand, and notoriously low statistical power on the other (Sterling, 1959; Fanelli, 2010; Maxwell, 2004). These findings are at odds with each other and likely reflect a tendency to under-report negative results, through mechanisms such as file-drawering, publication bias, and 'questionable research practices' like p-hacking and HARKing. A strong bias against negative results can lead to an inflated false positive rate and inflated effect sizes in the literature, making it difficult for researchers to build on previous work and increasing the risk of ineffective or harmful 'evidence-based' applications and policies. In 2013, Registered Reports (RRs) were developed as a new publication format to reduce under-reporting of negative results by mitigating file-drawering, publication bias, and questionable research practices: Before collecting and analysing their data, authors submit a protocol containing their hypotheses and methods to a journal, where it gets reviewed and, if successful, receives 'in-principle acceptance' which guarantees publication once the results are in, regardless of the outcome. Given their bias-reducing safeguards, we should expect a lower positive result rate in RRs compared to the non-RR literature, but to date no structured comparison of RRs and non-RRs has been offered.

Objectives: Fanelli (2010) presented a simple method to assess the positive result rate in a large sample of publications. We used his method to replicate his results for the (non-RR) psychology literature since 2013 and compare it to all published RRs in psychology.

Hypothesis: Using Fanelli's method, we tested the hypothesis that published RRs in psychology have a lower positive result rate than non-RRs in psychology published in the same time range (2013-2018). We would reject this hypothesis if the difference between RRs and non-RRs were found to be significantly smaller than 6%. Method: To obtain the non-RR sample, we applied Fanelli's (2010) sampling strategy: We searched all journals listed in the 'Psychiatry/Psychology' category of the Essential Science Indicators database for the phrase 'test* the hypothes*' and picked a random sample of 150 publications of all search results, deviating from Fanelli only in restricting the year of publication to 2013-2018. To obtain the RR sample, we relied on a list of published RRs curated by the Center for Open Science (https://www.zotero.org/groups/479248/osf/items/collectionKey/KEJP68G9?) which at the time had 152 entries, and excluded all publications that were not in psychology or not certainly RRs, leaving 81 publications. The positive result rate was determined by identifying the first hypothesis mentioned in the abstract or full text and coding whether it was (fully or partially) supported or not supported, and then for each group calculating the proportion of papers that reported support. Methods and analyses were preregistered at https://osf.io/s8e97/.

Results: Eight non-RRs and 13 RRs were excluded because they either did not test a hypothesis or could not be coded for other reasons, leaving 142 non-RRs and 68 RRs. The positive result rate was 95.77% for non-RRs and 42.65% for RRs. The proportion difference was significantly different from zero (one-sided Fisher's exact test, alpha = .05), p < .0001, and not significantly smaller than our smallest effect size of interest of 6% in an equivalence test, Z = -7.564, p > .999. For an exploratory analysis we also coded whether or not a paper contained a replication of previous work and found that none of the non-RRs, but two thirds (42/68) of the RRs did. The positive result rate for replication RRs was slightly lower (35.71%) than for original RRs (53.85%), but this difference was not significant, p = .112.

Conclusions and implications: In 2010, Fanelli reported a positive result rate of 91.5% for the field of psychology. Using the same method, we found a rate of 95.77% for the time between 2013 and 2018, suggesting that the rate has not gone down in recent years. In contrast, with only 42.65% the new population of Registered Reports shows a strikingly lower positive result rate than the non-RR literature. This difference may be somewhat smaller when focussing only on original work, but the RR population is currently too small to draw strong conclusions about any differences between replication and original studies. Our conclusions are limited by the different sampling procedures for RRs and non-RRs and by the observational nature of our study, which did not allow us to account for potential confounding factors. Nonetheless, our results are in line with the assumption that RRs reduce under-reporting of negative results and provide a first estimate for the difference between this new population of studies and the conventional literature.