Crowdsourcing hypothesis tests: Making transparent how design choices shape research results. Justin F. Landy et al. http://www.socialjudgments.com/docs/Crowdsourcing_Hypothesis_Tests_Manuscript_Oct_31_2019.pdf
Abstract: To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.
Keywords: Crowdsourcing, scientific transparency, stimulus sampling, forecasting, conceptual replications, research robustness
Hypothesis 5: The tendency to make deontological judgments is positively correlated with
happiness.
In order to bridge the normative-descriptive divide between the fields of
philosophical ethics (how should people morally behave) and moral psychology (how and why
do people morally behave) cognitive science must map out how variation in moral cognitions are
systematically related to variances in outcomes related to human flourishing. The goal of this
original research was to contribute to this endeavor by examining how the tendency to make
utilitarian versus deontological moral judgments (Bentham 1970/1823; Kahane, 2015; Kant,
1993/1785; Mill, 1861) relates to personal happiness and well-being (Kahneman, Diener, &
Schwarz, 1999; Ryff, 1989; Waterman, 1993). The idea that happiness and morality are tightly
intertwined has a long history in philosophy (see, e.g., Annas, 1993; Aristotle, 340 BCE/2002;
Foot, 2001; Kraut, 1979), and recent empirical work suggests that people consider moral
goodness to be an element of what “happiness” consists of (Phillips, Freitas, Mott, Gruber, &
Knobe, 2017; Phillips, Nyholm, & Liao, 2014). However, prior work has not examined the
relationship (if any) between specific moral orientations and happiness.
Hypothesis 5 posits that people who are more inclined to base their moral judgments on
the violation of rules, duties, and obligations (deontological judgments) versus material
outcomes (utilitarian judgments) are also more likely to experience happiness in their lives. This
prediction is based on philosophical and scientific evidence that has demonstrated shared
psychological and neurological mechanisms between these dimensions (e.g., Everett, Pizarro, &
Crockett, 2016; Greene, 2013; Lieberman, 2013; Phillips et al., 2017; Singer, 2005). To test this
hypothesis, Sowden and Hall (2015) asked participants to judge several morally questionable
behaviors that pitted utilitarian and deontological considerations against one another (Greene et
al. 2001) and compared an index of those judgments to how they responded to measures of
subjective well-being (Diener et al., 1985; Watson et al., 1988) and eudaimonic happiness
(Waterman et al., 2010). The crowdsourced project posed the research question to independent
researchers, who separately designed studies relating moral judgments to individual happiness.
Results from Hyp 5
Hypothesis 5: Deontological judgments predict happiness. Although the original pattern of results once again directly replicated using the original materials, the hypothesis that individuals who tend to make deontological (vs. utilitarian) judgments report different levels of personal happiness was not supported overall by the crowdsourced conceptual replications. Although a statistically significant directional effect in support of H5 was reported in the Main Studies, the aggregated estimate was close to zero, and the effect did not reach statistical significance in the Replication Studies. Overall, the Bayesian analysis found strong evidence against this original prediction. There has not previously been a systematic review or metaanalysis of the relationship between moral stance and happiness, though prior research has linked both processes to emotional and intuitive responding (e.g., Everett et al., 2016; Greene, 2013; Lieberman, 2013; Phillips et al., 2017; Singer, 2005). These results fail to find support for an association between deontological moral judgments and hedonic happiness that has been suggested – although not empirically confirmed – by this prior work. Although laypeople appear to believe that part of what brings happiness is living a moral life (Phillips et al., 2017; Phillips et al., 2014), adherence to deontological vs. utilitarian ethical principles does not seem to relate to one’s overall happiness.
No comments:
Post a Comment