Poor test-retest reliability of task-fMRI: New empirical evidence and a meta-analysis

Poor test-retest reliability of task-fMRI: New empirical evidence and a meta-analysis. Maxwell L. Elliott, Annchen R. Knodt, David Ireland, Meriwether L. Morris, Richie Poulton, Sandhya Ramrakha, Maria L. Sison, Terrie E. Moffitt, Avshalom Caspi, Ahmad R. Hariri. bioRxiv preprints, Jun 24 2019. https://doi.org/10.1101/681700

Abstract: Identifying brain biomarkers of disease risk and treatment response is a growing priority in neuroscience. The ability to identify meaningful biomarkers is fundamentally limited by measurement reliability; measures that do not yield reliable values are unsuitable as biomarkers to predict clinical outcomes. Measuring brain activity using task-fMRI is a major focus of biomarker development; however, the reliability of task-fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments with 1,088 participants reporting 1,146 ICCs for task-fMRI revealed poor overall reliability (mean ICC = .397). Second, the test-retest reliabilities of activity in a priori regions of interest across 11 commonly used fMRI tasks collected in the Human Connectome Project and the Dunedin Longitudinal Study were poor (ICCs = .067 - .485). Collectively, these findings demonstrate that commonly used task-fMRI measures are not currently suitable for brain biomarker discovery or individual differences research in cognitive neuroscience (i.e., brain-behavior mapping). We review how this state of affairs came to be and consider several avenues for improving the reliability of task-fMRI.

Significance Statement A biomarker with the potential to be useful in predicting clinical outcomes must yield values that are repeatable. We performed a comprehensive meta-analysis of the test-retest reliability of task-fMRI measures, which are widely adopted for biomarker discovery in neuroscience. We found that the meta-analytic reliability of task-fMRI was poor. We also investigated the reliability of many of the most commonly used task-fMRI measures in two datasets recently collected with cutting-edge scanners and methods. We found generally poor reliability for these task-fMRI measures. These findings indicate that many task-fMRI measures are not currently suitable for biomarker discovery or individual differences research in cognitive neuroscience.