Drawing on dual-process theory, we suggest that the benefits that arise from combining several quantitative individual judgments will be heightened when the judgments are based on different cognitive processes

Enhancing the Wisdom of the Crowd With Cognitive-Process Diversity: The Benefits of Aggregating Intuitive and Analytical Judgments. Steffen Keck, Wenjie Tang. Psychological Science, September 22, 2020. https://doi.org/10.1177/0956797620941840

Rolf Degen's take: https://twitter.com/DegenRolf/status/1308645051766116352

Abstract: Drawing on dual-process theory, we suggest that the benefits that arise from combining several quantitative individual judgments will be heightened when these judgments are based on different cognitive processes. We tested this hypothesis in three experimental studies in which participants provided estimates for the dates of different historical events (Study 1, N = 152), made probabilistic forecasts for the outcomes of soccer games (Study 2, N = 98), and estimated the weight of individuals on the basis of a photograph (Study 3, N = 3,695). For each of these tasks, participants were prompted to make judgments relying on an analytical process, on their intuition, or (in a control condition) on no specific instructions. Across all three studies, our results show that an aggregation of intuitive and analytical judgments provides more accurate estimates than any other aggregation procedure and that this advantage increases with the number of aggregated judgments.

Keywords: decision making, judgment, cognitive processes, wisdom of the crowd, judgment aggregation, group judgments, dual-process theory, open data

General Discussion

The results of three experimental studies showed that forming crowds with a high level of cognitive-process diversity—by aggregating a combination of intuitive and analytical individual judgments—improved the quality of crowd wisdom, compared with crowds formed by an aggregation of only analytical judgments, only intuitive judgments, or judgments made in a control condition without specific manipulation of judges’ cognitive processes. Moreover, we found that whereas the benefits of cognitive-process diversity generally held for both smaller and larger crowds, the magnitude of these benefits increased with crowd size and eventually approached its maximum as crowds became very large. Providing supporting evidence for the suggestion that the benefits of cognitive-process diversity are driven by higher levels of judgment-error independence, the results of Studies 1 and 2 revealed a lower average correlation in signed errors between judges employing an intuitive cognitive process and those employing an analytical cognitive process, compared with judges relying on the same cognitive process or judges in the control condition.

One particularly interesting finding of Study 3 is that analytical-intuitive crowds still outperformed purely analytical crowds even though individual analytical judgments were more accurate than individual intuitive judgments—implying that in this specific context the benefits of adding more uncorrelated judgments outweighed the detrimental effects of adding less accurate judgments. It is, however, important to note that there are likely a number of domains (e.g., tasks that require the application of formal logic) in which intuitive judgments would be much less accurate than analytical ones and hence adding highly inaccurate though less correlated judgments to a crowd is likely not beneficial (e.g., Mannes et al., 2014).

Previous research has suggested ways to improve judgment aggregation, such as by selecting better performing individuals (e.g., Budescu & Chen, 2014; Mannes et al., 2014) or by refining the aggregation procedure (e.g., Jose & Winkler, 2008; Palley & Soll, 2019). By contrast, our approach focused on increasing independence between individual judgment errors by manipulating the cognitive process employed by individual judges to form their judgments. It thus also complements recent work by de Oliveira and Nisbett (2018), who investigated the possibility of improving crowd wisdom by amplifying the demographic diversity of crowds and found that this approach was largely ineffective. A likely explanation for this difference in results is that we directly manipulated the cognitive process by which judgments were being made, whereas demographic differences frequently might not be associated with differences in individual cognition.

One limitation of our work is that we manipulated judgments to either be predominantly intuitive or predominantly analytical. However, in practice, judgments and decisions might frequently be based on a process in the middle of a continuum with analytical and intuitive processes at the boundaries (e.g., Hammond, 1996). Thus, an interesting direction for future research would be to compare our approach with one in which a crowd is formed by aggregating judgments that are each based on a mixture of analytical and intuitive processes. A related important limitation of our results is that we did not provide direct insights into differences in participants’ exact cognitive processes, such as the use of different judgment rules or reliance on different pieces of information (e.g., Herzog & von Helversen, 2018; Hoffmann et al., 2013). Such differences might explain the higher independence between analytical and intuitive judgments observed in our studies.

A final interesting avenue for future research would be to explore whether our approach toward improving the wisdom of crowds might also help to increase the effectiveness of combining judgments that are made by the same individual (e.g., Herzog & Hertwig, 2009, 2014; Vul & Pashler, 2008).

General Discussion

The results of three experimental studies showed that forming crowds with a high level of cognitive-process diversity—by aggregating a combination of intuitive and analytical individual judgments—improved the quality of crowd wisdom, compared with crowds formed by an aggregation of only analytical judgments, only intuitive judgments, or judgments made in a control condition without specific manipulation of judges’ cognitive processes. Moreover, we found that whereas the benefits of cognitive-process diversity generally held for both smaller and larger crowds, the magnitude of these benefits increased with crowd size and eventually approached its maximum as crowds became very large. Providing supporting evidence for the suggestion that the benefits of cognitive-process diversity are driven by higher levels of judgment-error independence, the results of Studies 1 and 2 revealed a lower average correlation in signed errors between judges employing an intuitive cognitive process and those employing an analytical cognitive process, compared with judges relying on the same cognitive process or judges in the control condition.

One particularly interesting finding of Study 3 is that analytical-intuitive crowds still outperformed purely analytical crowds even though individual analytical judgments were more accurate than individual intuitive judgments—implying that in this specific context the benefits of adding more uncorrelated judgments outweighed the detrimental effects of adding less accurate judgments. It is, however, important to note that there are likely a number of domains (e.g., tasks that require the application of formal logic) in which intuitive judgments would be much less accurate than analytical ones and hence adding highly inaccurate though less correlated judgments to a crowd is likely not beneficial (e.g., Mannes et al., 2014).

Previous research has suggested ways to improve judgment aggregation, such as by selecting better performing individuals (e.g., Budescu & Chen, 2014; Mannes et al., 2014) or by refining the aggregation procedure (e.g., Jose & Winkler, 2008; Palley & Soll, 2019). By contrast, our approach focused on increasing independence between individual judgment errors by manipulating the cognitive process employed by individual judges to form their judgments. It thus also complements recent work by de Oliveira and Nisbett (2018), who investigated the possibility of improving crowd wisdom by amplifying the demographic diversity of crowds and found that this approach was largely ineffective. A likely explanation for this difference in results is that we directly manipulated the cognitive process by which judgments were being made, whereas demographic differences frequently might not be associated with differences in individual cognition.

One limitation of our work is that we manipulated judgments to either be predominantly intuitive or predominantly analytical. However, in practice, judgments and decisions might frequently be based on a process in the middle of a continuum with analytical and intuitive processes at the boundaries (e.g., Hammond, 1996). Thus, an interesting direction for future research would be to compare our approach with one in which a crowd is formed by aggregating judgments that are each based on a mixture of analytical and intuitive processes. A related important limitation of our results is that we did not provide direct insights into differences in participants’ exact cognitive processes, such as the use of different judgment rules or reliance on different pieces of information (e.g., Herzog & von Helversen, 2018; Hoffmann et al., 2013). Such differences might explain the higher independence between analytical and intuitive judgments observed in our studies.

A final interesting avenue for future research would be to explore whether our approach toward improving the wisdom of crowds might also help to increase the effectiveness of combining judgments that are made by the same individual (e.g., Herzog & Hertwig, 2009, 2014; Vul & Pashler, 2008).