USA: Evidence of a negative Flynn effect on the attention/working memory and learning trials; as expected, education level, age group, and ethnicity were significant predictors of California Verbal Learning Test performance

Cohort differences on the CVLT-II and CVLT3: evidence of a negative Flynn effect on the attention/working memory and learning trials. Lisa V. Graves, Lisa Drozdick, Troy Courville, Thomas J. Farrer, Paul E. Gilbert & Dean C. Delis. The Clinical Neuropsychologist, Dec 12 2019. https://doi.org/10.1080/13854046.2019.1699605

Abstract
Objective: Although cohort effects on IQ measures have been investigated extensively, studies exploring cohort differences on verbal memory tests, and the extent to which they are influenced by socioenvironmental changes across decades (e.g. educational attainment; ethnic makeup), have been limited.

Method: We examined differences in performance between the normative samples of the CVLT-II from 1999 and the CVLT3 from 2016 to 2017 on the immediate- and delayed-recall trials, and we explored the degree to which verbal learning and memory skills might be influenced by the cohort year in which norms were collected versus demographic factors (e.g. education level).

Results: Multivariate analysis of variance tests and follow-up univariate tests yielded evidence for a negative cohort effect (also referred to as negative Flynn effect) on performance, controlling for demographic factors (p = .001). In particular, findings revealed evidence of a negative Flynn effect on the attention/working memory and learning trials (Trial 1, Trial 2, Trial 3, Trials 1–5 Total, List B; ps < .007), with no significant cohort differences found on the delayed-recall trials. As expected, education level, age group, and ethnicity were significant predictors of CVLT performance (ps < .01). Importantly, however, there were no interactions between cohort year of norms collection and education level, age group, or ethnicity on performance.

Conclusions: The clinical implications of the present findings for using word list learning and memory tests like the CVLT, and the potential role of socioenvironmental factors on the observed negative Flynn effect on the attention/working memory and learning trials, are discussed.

Keywords: Cohort differences, Flynn effect, verbal memory, California Verbal Learning Test


Discussion
In the present study, we examined differences in performance on the immediate- and
delayed-recall trials between the CVLT-II and CVLT3 normative samples. Specifically,
we explored the extent to which verbal learning and memory skills were influenced
by the cohort year in which norms were collected (i.e. 1999 for the CVLT-II versus
2016–2017 for the CVLT3) versus differences in education level. Of note, differences in
education level between the CVLT-II and CVLT3 normative samples mirrored an
increase in the proportion of U.S. adults who completed post-secondary education
during the time period spanning the development of the CVLT-II and CVLT3.
The present study revealed evidence of a negative Flynn effect on the attention/
working memory and learning trials of the CVLT-II/CVLT3, with the CVLT3 cohort performing
significantly worse than the CVLT-II cohort on Trial 1, Trial 2, Trial 3, Trials 1–5
Total, and List B). In contrast, no significant cohort differences were found on the
delayed-recall trials. Consistent with past research, education level, age group, and
ethnicity were shown to be significant predictors of overall CVLT performance.
Education level and age group were positively and negatively associated with CVLT-II/
CVLT3 performance, respectively. With regard to ethnicity, performance on multiple
immediate- and delayed-recall trials was significantly higher among White and
Hispanic individuals relative to African-American individuals. Nevertheless, none of these
demographic variables were shown to have an interactive effect with cohort year of
norms collection on performance.
The present study overcomes some of the limitations of previous studies that examined
Flynn or cohort effects on learning and memory of word lists (e.g. use of relatively
small sample sizes; limited age ranges; confounding time of testing with changes in the
target words; using data harmonization techniques to convert Logical Memory scores to
CERAD Word List scores). Of note, the present study offers the advantage of using the
same word lists administered to large normative samples that represent a wide age
range and that were matched to the demographic makeup of the U.S. census at the time
that the testing occurred in order to explore potential cohort effects on a standardized
measure of verbal learning and memory. Further, the present findings are in line with
recent research suggesting that a negative Flynn effect may be occurring not only on IQ
tests, but also on measures of auditory attention/working memory and learning of word
lists. That is, given that negative cohort effects were observed only on immediate-recall
trials (and appeared to be driven by cohort differences on the first three learning trials in
particular), the present findings provide further evidence that the attention/working
memory aspects of verbal memory may be particularly vulnerable to negative cohort
effects (Wongupparaj et al., 2017).
As discussed above, the present study indicated that the CVLT3 normative sample
was more highly educated, on average, than the CVLT-II normative sample, and this
difference mirrored the increase over the past two decades in the proportion of U.S.
adults who completed post-secondary education. However, while the present study
yielded evidence for a significant positive (albeit relatively small) association between
education level and CVLT-II/CVLT3 performance, the evidence for a negative cohort
effect on performance persisted even after accounting for differences in education
level and cohort year education interactions (which were nonsignificant).
Furthermore, the observed negative cohort effect on immediate recall was present
across all age and ethnic groups, with cohort year x age group and cohort year x ethnicity
interactions also being nonsignificant.
The present results are consistent with the findings from a meta-analysis of cohort
effects on attention and working memory measures conducted by Wongupparaj et al.
(2017), who found a gradual decline in more complex auditory attention/working
memory skills (e.g. digit span backward) over the past four decades. Although the current
findings differ from those of Dodge et al. (2017), in which a positive cohort effect
was reported for word list learning and memory performance (including immediate
and delayed recall), there were a number of limitations in that study that make it difficult
to directly compare results from the two investigations (e.g. investigating only
older adults [65 years or older]; notable differences in the proportions of adults in
each age cohort within the pooled sample who were tested in the first versus second
data collection period; not addressing the fact that age cohort and period of testing
were potentially confounded; and using data harmonization techniques to convert
Logical Memory scores from a subset of its data pool into CERAD Word List scores to
facilitate analyses of cohort differences on verbal memory performance).
The results of the present study raise intriguing questions about the effects of socioenvironmental
changes that have unfolded during the time period spanning the
development of the second and third editions of the CVLT. In particular, the present
findings suggest that socioenvironmental changes may have occurred since 2000 that
(a) might be negatively impacting working memory and verbal learning skills, (b) are
not disproportionately affecting certain age or ethnic groups, and (c) are occurring
independent of generational changes in educational attainment. While education level
was examined in the present study, a number of researchers have highlighted distinctions
between educational attainment and education quality, and have suggested that
“educational attainment” as a homogeneous variable may have become diluted in
recent years due to varying standards and quality required for degrees across educational
settings (Allen & Seaman, 2013; Bratsberg & Rogeberg, 2018; Hamad et al., 2019;
Jaggars & Bailey, 2010; Nguyen et al., 2016; Rindermann et al., 2017). The lack of an
observed cohort year x education level interaction effect found in the present study
may reflect, in part, these recent concerns about the homogeneity of educational
attainment. This is important to consider for the present findings given that the negative
Flynn effect that has recently been found on IQ measures has been partly attributed
to reduced quality of education in those studies (Allen & Seaman, 2013; Jaggars
& Bailey, 2010).
It is difficult to escape the observation that the time period spanning the development
of the second and third editions of the CVLT (1999/2000 versus 2016/2017) also
coincided with a profound societal change: the digital revolution. As noted in recent
reviews (Rindermann et al., 2017; Wilmer, Sherman, & Chein, 2017), the use of digital
technology, while offering multiple advantages, may have subtle but significant
adverse effects on working memory and rote memorization skills. While relationships
between the use of digital technology and verbal learning and memory performance
were not formally investigated in the present study, the current findings invite the
intriguing hypothesis that increased use of digital tools may inadvertently have an
adverse effect on working memory and learning abilities. Unfortunately, there has
been a paucity of studies investigating associations of self-reported and performancebased
internet use with cognition. While there is evidence that the ability to perform
different tasks on the internet is significantly correlated with performance on cognitive
tests (Woods et al., 2019), no studies have directly investigated whether varying
degrees of internet, mobile phone, or other digital technology usage may positively or
negatively affect the development and maintenance of different domains of cognition.
Future research should explore potential differences between high and low internet
users on neuropsychological test performance. In addition, the present study was also
limited in that we were unable to assess relationships between other socioenvironmental
changes that may have occurred in the years spanning the development of
the CVLT-II and CVLT3 (e.g. generational changes in healthcare or standard of living;
see Dutton & Lynn, 2013; Rindermann et al., 2017).
The present findings were likely related to true cohort differences in verbal learning
and memory skills, and not to differences between the makeup of the CVLT-II versus
the CVLT3, given that (a) the lists of target words are identical across the two versions
of the test; (b) the negative cohort effect was only observed on select trials, thereby
indicating that one version is not uniformly harder or easier than the other; and (c)
other recent studies have also found evidence for a negative Flynn effect on attention/
working memory components of verbal memory (Wongupparaj et al., 2017). One
question that arises is whether the observed negative cohort effect found on the CVLT
in the present study was due to a negative Flynn effect specifically on attention/working
memory and learning skills versus a broader effect on IQ in general, which has
also been reported in recent years (Bratsberg & Rogeberg, 2018; Dutton & Lynn, 2013,
2015; Dutton et al., 2016; Flynn & Shayer, 2018; Pietschnig & Gittler, 2015; Shayer &
Ginsburg, 2007, 2009; Sundet et al., 2004; Teasdale & Owen, 2005, 2008; Woodley &
Meisenberg, 2013). Given that the CVLT-II and CVLT3 were not co-normed with IQ
tests, we cannot directly investigate this relationship. However, IQ has been shown to
correlate robustly with education level, and education was not shown to drive or moderate
any of the observed cohort effects in the present study. These findings suggest
that the present findings were related to true cohort differences in attention/working
memory and learning skills independent of any cohort changes that might also be
occurring for IQ functions in general.
It is also worth noting that the negative cohort effects observed in the present
study were associated with relatively small effect size estimates (i.e. gp
2 < .010 on
immediate-recall trials). However, the cohort effects are unlikely due to random chance
given the robust statistical power rendered by our large sample size. Moreover, from a
clinical perspective, even a small difference in raw scores can have a notable impact
on the conversion to standardized scores, which in turn can impact decisions about
an examinee’s level of cognitive functioning. For example, for an individual within the
age range of 45–54 years, a raw score of 4 on Trial 1 yields a z-score of –1.5 based on
CVLT-II norms versus a scaled score of 7 based on CVLT3 norms (note that the CVLT3
now uses scaled scores rather than z-scores); thus, this individual’s Trial 1 performance
could be interpreted as mildly impaired using CVLT-II norms and low average using
CVLT3 norms.
The present results have other important implications for clinical practice. In a
recent position paper, Bush et al. (2018) discussed the advantages and disadvantages
of using newer versus older versions of neuropsychological tests. The authors note
that an advantage of an older version of a neuropsychological test is that it may be
grounded more in empirical data supporting its validity, whereas a newer version may
lack such empirical support. Additionally, older versions of tests offer the advantage of
increased familiarity and ease of interpretation for clinicians. However, Bush et al.
(2018) also note that if cohort differences are found in the normative data between
the older and new versions of a test, then the use of the older version may provide
inaccurate standardized scores in a present-day evaluation (see also Alenius et al.,
2019). Given the present findings, the continued use of the CVLT-II’s 1999 norms in
today’s assessments may provide artificially lower standardized scores on indices of
attention/working memory and learning across the immediate-recall trials (e.g. Trial 1,
Trial 2, Trial 3, Trials 1–5 Total, List B). Further, given that the target lists and Yes/No
Recognition trial are the same on the CVLT3 as those used on the CVLT-II, 1) the validity
studies that have been conducted to date for the CVLT-II (over 1,000 published
studies; Delis et al., 2017) likely still have relevance for the CVLT3, and 2) familiarity
and ease of interpretation should be relatively equivalent across the two test versions.
Finally, the present results also suggest that the normative data that are currently
being used for other verbal learning and memory tests (e.g. California Verbal Learning
Test – Children’s Version; Rey Auditory Verbal Learning Test; Hopkins Verbal Learning
Test), which were initially collected before 2000 and have not undergone any major
revisions since the early 2000s, may also have become outdated and are in need of
re-norming in the near future.
In summary, the current study found evidence of a negative Flynn effect on the
attention/working memory and learning trials of the CVLT-II/CVLT3. The findings have
clinical implications for the use of word list learning and memory tests like the CVLT,
and raise intriguing questions about the possible adverse effects of recent socioenvironmental
changes on attention, working memory, and learning skills.