Laypeople Can Predict Which Social Science Studies Replicate

Hoogeveen, Suzanne, Alexandra Sarafoglou, and Eric-Jan Wagenmakers. 2019. “Laypeople Can Predict Which Social Science Studies Replicate.” PsyArXiv. September 25. doi:10.31234/osf.io/egw9d

Abstract: Large-scale collaborative projects recently demonstrated that several key findings from the social science literature could not be replicated successfully. Here we assess the extent to which a finding’s replication success relates to its intuitive plausibility. Each of 27 high-profile social science findings was evaluated by 233 people without a PhD in psychology. Results showed that these laypeople predicted replication success with above-chance performance (i.e., 58%). In addition, when laypeople were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%. We discuss the prediction patterns and apply signal detection theory to disentangle detection ability from response bias. Our study suggests that laypeople’s predictions contain useful information for assessing the probability that a given finding will replicate successfully.