Failed replication of Vohs & Schooler 2008: Manipulating free will beliefs in a robust way is more difficult than has been implied by prior work, & the proposed link with immoral behavior may not be as consistent as suggested

Nadelhoffer, Thomas, Jason Shepard, Damien Crone, Jim A. C. Everett, Brian D. Earp, and Neil Levy. 2019. “Does Encouraging a Belief in Determinism Increase Cheating? Reconsidering the Value of Believing in Free Will.” OSF Preprints. May 3. doi:10.31219/osf.io/bhpe5

Abstract: A key source of support for the view that challenging people’s beliefs about free will may undermine moral behavior is two classic studies by Vohs and Schooler (2008). These authors reported that exposure to certain prompts suggesting that free will is an illusion increased cheating behavior. In the present paper, we report several attempts to replicate this influential and widely cited work. Over a series of four high-powered studies (sample sizes of N = 162, N = 283, N = 268, N = 804) (three preregistered) we tested the relationship between (1) anti-free-will prompts and free will beliefs and (2) free will beliefs and immoral behavior. Our primary task was to closely replicate the findings from Vohs and Schooler (2008) using the same or similar manipulations and measurements as the ones used in their original studies. Our efforts were largely unsuccessful. We suggest that manipulating free will beliefs in a robust way is more difficult than has been implied by prior work, and that the proposed link with immoral behavior may not be as consistent as previous work suggests.


4. General Discussion
The free will debate has gone mainstream in recent years in the wake of scientific advances that on some accounts seem to undermine free will. Given the traditional associations between free will and moral responsibility, a great deal may hang on this debate. In a high-profile paper on the relationship between free will beliefs and moral behavior, Vohs and Schooler (2008) cautioned against public pronouncements disputing the existence of free will, based on their findings concerning the relationship between free will beliefs and cheating. Our goal in this paper was to replicate their landmark findings. Across four studies, we had mixed results. While we were able to influence people’s beliefs in free will in one of the four studies, we failed in our efforts to find a relationship between free will beliefs and behavior. When coupled with the work of other researchers who have had difficulty replicating the original findings by Vohs and Schooler, we think this should give us further pause for concern.
That said, there are four primary limitations of our studies. First, in light of the results from Study 4, it is possible that there is a link between free will belief and moral behavior—we just failed to detect it because our two behavioral studies were not high powered enough. Perhaps a very high-powered (800+ participants) behavioral experiment would replicate Vohs and Schooler’s original findings. That is certainly possible, but we are doubtful that simply running another high-powered experiment would yield the desired effect. After all, our pooled data analyses have 1,089 and 551 pooled participants, respectively. Moreover, Monroe, Brady, and Malle (2016) had mixed results manipulating free will beliefs in very high-powered studies. And even when they did manage to decrease free will beliefs, they did not find any behavioral differences. So, we are not convinced that insufficient power explains our failures to replicate—especially given that in Vohs and Schooler’s original studies were underpowered (N = 15-30 per cell) and yet they found very large effects both with respect to manipulating free will beliefs (d = 1.20) and influencing cheating behavior (d = 0.88). By our lights, we have done enough in this paper—when coupled with the other mixed results from attempts to replicate Vohs and Schooler (2008)—to weaken our collective confidence in the proposed relationship between free will beliefs and moral behaviors. That is not to say there is no relationship, however, it suggests that if there is one, it likely not a relationship we should be especially worried about from the dual standpoints of morality and public policy.
The second potential problem with our studies is that we ran them online rather than using a convenience sample, as Vohs and Schooler did. While we tried to ensure that we mimicked their original work as much as possible, follow up studies with a convenience sample would certainly be valuable. However, the differences in sample should not deflate the importance of our replication attempts. After all, the effect (and its societal implications) are claimed to be pervasive. If directly communicating skepticism about free will barely undermined people's beliefs and (going beyond our own data) at most resulted in only a trivial increase in bad behavior (or affected behavior in a very limited range of contexts), then the effect is arguably unimportant and unworthy of the substantial attention it has received so far. A third limitation is that we only used American participants. However, this limitation is an artifact of our goal of trying to replicate the work by Vohs and Schooler. Because they used an American sample, we used an American sample. Figuring out whether their work replicates in a non-American sample is a task for another day. That said, we would obviously welcome cross-cultural studies that implemented our paradigms to see whether our findings are cross-culturally stable.
The fourth and final limitation our experimental design is the possibility that MTurk participants may not be as attentive as in-lab participants. To guard against this, we used an attention check and excluded any participants who failed it. We also used two items designed to encourage participants to pay attention by reminding them that they would be asked to write about the content of the vignette they read. While these measures can obviously not guarantee participants are paying attention, we’d like to think that they reduce the likelihood of inattention. Additionally, many lab tasks that are particularly susceptible to lapses in attention have been replicated using MTurk populations, including tasks that depend on difference in reaction times on the scale of milliseconds (e.g., Erikson Flanker tasks) and memory tasks that are heavily attention-dependent (see Woods, et al., 2015 for a review).
Setting these limitations aside, we nevertheless think we have made a valuable contribution to the literature on the relationship between free will beliefs and moral behavior. Minimally, our findings serve as a cautionary tale for those who fret that challenging free will beliefs might undermine public morality. Future research on this front will have to take into consideration the difficulty of replicating both standard manipulations of belief in free will and the purported link between free will skepticism and morality. Contrary to our initial expectations, the association between free will beliefs and moral behavior appears to be elusive. As such, worries about the purported erosion of societal mores in the wake of recent advances in neuroscience are likely to be misplaced. The belief in free will appears to be more stable, robust, and resistant to challenge than earlier work suggests. While some scientists may think that their research undermines the traditional picture of agency and responsibility, public beliefs on this front are likely to be relatively slow to change. Even if beliefs about free will were to incrementally change, given the lack of association between dispositional free will beliefs and moral behavior reported by Crone and Levy (2018), it is unclear that people would have difficulty integrating such beliefs into a coherent worldview that permits the same level of moral behavior.