Measuring Cognitive Reflection Without Maths: Developing and Validating the Verbal Cognitive Reflection Test

Sirota, Miroslav, Lenka Kostovičová, Marie Juanchich, Chris Dewberry, and Amanda C. Marshall. 2018. “Measuring Cognitive Reflection Without Maths: Developing and Validating the Verbal Cognitive Reflection Test.” PsyArXiv. December 6. doi:10.31234/osf.io/pfe79

Abstract: The Cognitive Reflection Test (CRT) measures the ability to suppress an initial (incorrect) intuition and to reflect when solving three mathematical problems. It rapidly became popular for its impressive power to predict how well people reason and make decisions. Despite the popularity of the CRT, a major issue complicates its interpretation: the numerical nature of the CRT confounds reflection ability with mathematical ability. In addition, the statistical and psychometric properties of the CRT are suboptimal and an increasing proportion of participants have become familiar with it. We have addressed these issues by developing the Verbal CRT (CRT-V), a novel 10-item measure of cognitive reflection, using non-mathematical problems with good statistical and psychometric properties and with low familiarity. First, we selected suitable items with relatively low familiarity and optimal difficulty as identified in two different populations (Studies 1 and 2) and with high content validity as judged by an expert panel (Study 3). Second, we demonstrated a good criterion and construct validity for the test in different populations with a wide range of variables (Studies 4-6) and a good internal consistency and test-retest reliability (Study 7). The Verbal CRT was less associated with numeracy than the original CRT and was not biased against women as was the case with the original CRT. The Verbal CRT can complement existing tests of cognitive reflection; it will be especially appropriate for use in general adult populations and in populations that are less educated and mathematically anxious.