False-Positive Social Psychology: How Deviations from Preregistrations Affect False-Positive Significance Rates

False-Positive Social Psychology: How Deviations from Preregistrations Affect False-Positive Significance Rates. Terry Cheng, Honors Thesis, UCSB, Mar 14 2022. https://osf.io/2fwz6

Abstract: Numerous solutions have been proposed to address the replication crisis, in which numerous high-profile empirical research studies cannot be replicated by other research teams. One possible explanation is that researchers have the option to adjust their data analyses after viewing the results, inflating false positive rates. One popular solution is study preregistration, the prac-tice of developing the data analysis plan before the data is collected. However, preregistrations only alleviate replication problems if researchers are held accountable to their analysis plans. Across two related studies, we explore the effectiveness of preregistration in its current form. In Study 1, we audit recent preregistered publications from a major psychology journal and observe deviations in 19 of 32 papers. In Study 2, we simulate the effects of generic deviations on the false-positive rate. We find that deviations that run more or more varied tests cause larger changes, tripling the false-positive rate in the most extreme case. We note that auditing preregistrations requires an inconsistent amount of time depending on their length and format, which we suspect contributes to the enforcement issues we observe. We suggest that researchers and journals alike adopt the asPredicted.org template for preregistrations.