Requisite Skills and the Meaningful Measurement of Cognition Among the Lowly Educated or Non-Medicated Schizophrenics

Requisite Skills and the Meaningful Measurement of Cognition. Richard S. E. Keefe, Michael F. Green, Philip D. Harvey. JAMA Psychiatry. Published online July 8, 2020. doi:10.1001/jamapsychiatry.2020.1618

In 1931, the noted neuropsychologist Alexander Luria, along with his supervisor Lev Vygotsky, led an expedition from Moscow to Uzbekistan with an honorable objective: to understand cognition in a population with low educational levels.1 However, they concluded that the minimal educational background of the population could prevent these individuals from engaging in the basic elements of a cognitive evaluation and that existing cognitive tests were not valid for their study population, so they created specialized tests that would be more appropriate. The study by Stone et al2 shares features of this expedition 90 years ago. Some of our era’s leading scientists have conducted an equally honorable collaborative project in the rural province of Ningxia, China, to assess individuals with chronic schizophrenia who have never received antipsychotic medications. By examining this population, the authors aimed to address the problem of the exclusion of underserved individuals from research on serious mental illness and to examine the longitudinal cognitive trajectory of schizophrenia in its natural untreated condition. The authors found that, in their cross-sectional sample, the duration of untreated psychosis was associated with worse cognitive performance. They specifically reported the association between cognition and the duration of chronic untreated psychosis as having partial Spearman correlation coefficients of 0.35 for the Brief Assessment of Cognition in Schizophrenia, Symbol Coding subtest; 0.24 for the Neuropsychological Assessment Battery, Mazes subtest; and 0.02 for the Brief Visuospatial Memory Test–Revised. These 3 assessments are conceptualized in western cultures as tests of processing speed, reasoning and problem solving, and visual memory. As with the findings of Luria and Vygotsky, given the distinct population in the Stone et al2 study, it is important to consider whether the assessments were suitable to the investigators’ purpose. The most relevant considerations are the validity of the tests for this group of individuals with low educational levels and the consequences of factors that could be associated with the passage of time itself, such as aging and changes in educational quality and opportunity.

To assess cognition in this study sample, the authors chose the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Consensus Cognition Battery (MCCB), which has been used to understand cognitive treatment response in clinical trials of cognition among patients with schizophrenia3 and has been accepted as a criterion-standard measure by the US Food and Drug Administration. The MCCB has been translated into more than 20 languages and has demonstrated reliability and validity in a wide variety of populations with schizophrenia across the world. Stone et al2 properly used a culturally adapted version of the MCCB that had been validated and normed in China and compared the results of participants with chronic untreated schizophrenia with those of a cohort of individuals without mental illness who had similar ages and educational histories. However, as described in standards published by the American Psychological Association,4 one of the important assumptions when administering tests such as the MCCB is that all participants have sufficient experience with the basic elements of testing to understand the intent of the test and perform the test procedures. Based on the educational level of the participants in this study, it is likely that these assumptions were not met, and the study’s primary findings may be associated with changes in education over time.

Any cross-sectional study of the association between longitudinal variables and cognition must recognize that performance on static tests of cognitive ability has been reported to improve (approximately 0.2 SDs per decade) over subsequent generations.5 There are many possible reasons for this improvement, which is called the Flynn effect6; these reasons include improvements in education, knowledge, health, nutrition, poverty, and environmental stimulation, among others.7 While the Flynn effect may be reversing in some western countries,6 it can be substantial in a rapidly evolving culture such as China, especially in the midst of an educational revolution. The median educational level of the participants in the Stone et al2 study was 3 years, and the median age of the participants was 52 years, with a range of 19 to 81 years. The study participants attended school between 1942 and 2004. Over the course of those 62 years, 2 rapid advances in education occurred that outpaced population growth. During the first period, from 1957 to 1960, the number of primary students attending school increased from 55 million to more than 90 million, and the number of middle school students doubled; during the second period, from 1966 to 1976, the number of middle school students increased from less than 15 million to more than 65 million.8 These increases in educational availability and quality were likely even more substantial in underdeveloped regions like Ningxia. Thus, given that the disease onset for most individuals with schizophrenia occurs during a narrow window in late adolescence, the older participants in this study would have been raised in a less advanced educational system. Early education focuses on the basic tenets of reading, writing, and arithmetic. Testing people who lack these fundamental skills creates challenges for the evaluation of cognition.

How, specifically, does the absence of relevant experience create challenges for the validity of a cognitive assessment battery? As an obvious example, consider a processing speed test with quickly moving stimuli that requires respondents to type letters on a smart phone. If test performance were compared between an individual aged 80 years who had no experience with the technology and an individual aged 25 years who had 15 years of experience with the technology, performance differences would likely be associated with the respondent’s experience with that technology, which was substantially different. Few, if any, investigators would consider the test results a valid measure of the participants’ cognition.

To those who received education in societies in which advanced education is the norm, a pencil would not be considered an advanced technology; however, those with limited experience in the use of a pencil will likely have difficulty completing tests that require drawing figures from memory or writing numbers quickly, and such adept use of a pencil is required for 4 of the 10 tests in the MCCB. Previous research on performance-based tests in China reported that, with regard to test performance, the consequences of low educational level were greater than those of a schizophrenia diagnosis, and the most difficult task for test respondents was writing their names using a pencil.9 Measuring the association between the duration of illness and cognition in a cross-sectional study is problematic when older participants have limited experience with the tools required for performance.

Revising the traditional instructions of a test by including additional practice trials and explanations, as done in the Stone et al2 study, may be insufficient. Just as children who are not exposed to human voices during early developmental periods are likely to struggle with language through adulthood, children who do not master fundamental skills, such as writing, during early education are likely to have the remainder of their lives shaped by the absence of those skills, and they may never be able to acquire the skill levels of those who received the requisite training.

The results of the Stone et al2 study illustrate the importance of requisite skills for cognitive assessment. The tests that were associated with duration of illness shared a common feature: all 3 of them required the use of a pencil. Only 1 of the 7 tests that were not associated with illness duration involved pencil use. Those tests mostly involved verbal interaction, which is, of course, a part of everyday life among people with lower and higher educational levels alike, so formal education for that skill was not necessary.

Leading anthropologists have asserted that the neuroanatomy and cognitive capacity of humans as a species have not changed in 30 000 years.10 However, our methods for assessing cognitive ability have changed substantially based on the extent and quality of the education we have received, which has allowed us to perform well on those same assessments, and methods will continue to change as the technologies we use to perform assessments advance. Conducting studies of cognition among people, near and far, who live in regions with underdeveloped educational systems is an important endeavor, but these studies may require special considerations for assessment and the development of specific tests that allow measurement of cognition that is independent of the confounding factor of inexperience with the requisite tools. Cognitive assessments may otherwise remain confounded by variability because of cultural advantages that provide early experience with the requisite tools for only a portion of the population.