Those of higher scoring on the Cognitive Reflection Test were more discerning in their social media use: They followed more selectively, shared news content from more reliable sources, and tweeted about weightier subjects

Mosleh, Mohsen, Gordon Pennycook, Antonio A. Arechar, and David G. Rand. 2019. “Digital Fingerprints of Cognitive Reflection.” PsyArXiv. October 17. doi:10.31234/osf.io/qaswn

Abstract: Social media is playing an increasingly large role in everyday life. Thus, it is of both scientific and practical interest to understand behavior on social media platforms. Furthermore, social media provides a unique window for social scientists to deepen our understanding of the human mind. Here we investigate the relationship between individual differences in cognitive reflection and behavior on Twitter in a sample of large N = 1,953 users recruited via Prolific Academic. In doing so, we differentiate between two competing accounts of human information processing: an “intuitionist” account whereby reflection plays little role in daily life, and a “reflectionist” account whereby reflection (and, in particular, overriding intuitive responses) does play an important role. We found that people who score higher on the Cognitive Reflection Test (CRT) – a widely used measure of reflective thinking – were more discerning in their social media use: They followed more selectively, shared news content from more reliable sources, and tweeted about weightier subjects. Furthermore, a network analysis indicated that the phenomenon of echo chambers, in which discourse is more likely with like-minded others, is not limited to politics: we observe “cognitive echo chambers” in which people low on cognitive reflection tend to follow the same set of accounts. Our results help to illuminate the drivers of behavior on social media platforms, and challenge intuitionist notions that reflective thinking is unimportant for everyday judgment and decision-making.

---
Discussion
Together, these results paint a fairly consistent picture. People in our sample who engaged in more cognitive reflection were more discerning in their social media use: They followed more selectively, shared higher quality content from more reliable sources, and tweeted about more weighty subjects. These results have numerous implications. Returning to the debate between those who have claimed a limited role for cognitive reflection in determining everyday behaviors (intuitionists) and those who emphasize the importance of the (perhaps distinctly) human capacity to use reflection to override intuitions (reflectionists), the results are plainly more consistent with the latter perspective. We find that reflective thinking (as measured in our survey study) is associated with a wide range of naturally occurring social media behaviors. Furthermore, each of these associations has important theoretical implications in their own right that we will now enumerate – and together, they paint a consistent picture of reflective thinking as an important positive force in judgment and decision-making outside of the laboratory.

One line of prior work which the current results bear on has to do with media truth discernment. Past work has shown that people who are more analytic and reflective are better at identifying true versus false news headlines, regardless of whether the headlines align with their ideology (e.g., (Pennycook and Rand 2019c)). However, these studies have relied entirely on survey experiments, where participant responses may be driven by experimenter demand effects or expressive responding. Additionally, in these experiments, participants judge a comparatively small set of headlines (pre-selected by the experimenters to be balanced on partisanship and veracity). Thus, these prior results may be idiosyncratic to the specific headlines (or approach for selecting headlines) used in designing the survey. Furthermore, these studies have focused on contrasting true headlines with blatantly false headlines (which may be comparatively rare outside the laboratory, (Grinberg et al. 2019, Guess et al. 2019), rather than articles which are misleading but not entirely false (e.g., hyper-partisan biased reporting of events that actually occurred (Pennycook and Rand 2019b)). Thus, the results may not generalize to the kinds of misinformation more typically encountered online. Finally, these studies have focused on judgments of accuracy, rather than sharing decisions. Thus, whether these previously documented associations extended to actual sharing in the naturally occurring social media environment is an open question – particularly given that the social media context may be more likely to active a political identity (as opposed to accuracy or truth) focus (Brady, Crockett and Van Bavel 2019, Van Bavel and Pereira 2018). Yet, despite these numerous reasons to think that prior findings may not generalize outside the survey context, we do indeed find that participants who perform better on the CRT share news from higher quality news sources. This observation substantially extends prior support for a positive role of reasoning in news media truth discernment.

Our results are also relevant in similar ways for prior work regarding the role of cognitive sophistication in political engagement. Prior evidence using survey experiments suggests that people who are more cognitively sophisticated (e.g., higher CRT, more educated, higher political knowledge) show higher rates of engagement with politics (Pennycook and Rand 2019a, Galston 2001). However, it has also been suggested that this relationship may be the result of social desirability bias, such that more cognitively sophisticated people simply over-report political engagement to please the experimenter (Holbrook, Green and Krosnick 2003, Enamorado and Imai 2018). Our results, however, suggest that more reflective people are indeed actually more engaged with politics on social media. This supports the inference that analytic thinking is associated with increased political engagement.

More broadly, cognitive reflection has been associated with lower gullibility – that is, less acceptance of a large range of epistemically suspect beliefs (such as conspiracy theories, paranormal claims, etc. – see (Pennycook et al. 2015b) for a review), including decreased susceptibility to pseudo-profound bullshit (Pennycook et al. 2015a). Again, however, these findings are rooted in survey evidence and not real-world behavior, and could reflect socially desirable responding. Here we find that low CRT is associated with increased following of and tweeting about money-making scams and get-rich-quick schemes. This supports the conclusion that more intuitive people are indeed more gullible.

One of the most intriguing results that we uncovered was the clustering of accounts followed by lower versus higher CRT participants. In particular, there was a cluster of accounts that were predominantly followed by low CRT participants. This observation is particularly interesting in the context of the extremely extensive discussion of partisan echo chambers, in which supporters of the same party are much more likely to interact with co-partisans (Stewart et al. 2019, Barberá et al. 2015, Garimella and Weber 2017). Our network analysis indicates that the phenomenon of echo chambers is not limited to politics: the cognitive echo chambers we observe have potentially profound implications for how information flows through social media. Furthermore, it is likely that cognitive echo chambers are not confined to social media – future work should investigate this phenomenon more broadly.

There are, of course, important limitations of the present work. Most notably, we were only able to consider the Twitter activity of a tiny subset of all users on the platform. Thus, it is important for future work to examine how our results generalize to other sets of users – and in particular, to users who did not opt in to a survey experiment. One potential approach that may be fruitful in this endeavor is training a machine learning classifier to estimate users’ CRT scores based on their social media activity. Relatedly, it will be important to test how the results generalize to other social media platforms (e.g. Facebook, LinkedIn), and to users from non-Western cultures. Future work should also examine how the results obtained here generalize to other measures of cognitive sophistication beyond the CRT.

In sum, here we have shed light on social media behavior using the lens of cognitive science. We have provided evidence that one’s extent of analytic thinking predicts a wide range of social media behaviors. These results meaningfully extend prior survey studies, demonstrating that analytic thinking plays an important role outside the laboratory.