Quality of Research on Mental Health Effects of COVID-19: Low compliance with open science recommendations, which raise concerns about the validity, generalisability, and reproducibility of the findings

Nieto, Ines, Juan F Navas, and Carmelo Vazquez. 2020. “The Quality of Research on Mental Health Effects of the COVID-19 Pandemic: A Note of Caution After a Systematic Review.” PsyArXiv. July 14. doi:10.31234/osf.io/ndgkj

Abstract
Background: SARS-CoV-2 pandemic has spurred scientific production in diverse fields of knowledge, including mental health. Yet, the quality of current research may be challenged by the urgent need to provide immediate results to understand and alleviate the consequences of the pandemic. This systematic review aims to examine compliance with basic methodological quality criteria and open science practices on the mental health effects of the COVID-19 pandemic.

Methods: A systematic search was performed using PubMed and Scopus databases on the 13th of May. Empirical studies, published in peer-reviewed journals in English, between February and May 2020, were included. The dependent variable(s) required to be quantitative and related to mental health. Exclusion criteria included clinical pharmacological trials, and studies using psychophysiological or biological recordings. The study protocol was previously pre-registered in https://osf.io/bk3gw/.

Findings: Twenty-eight studies were identified. More than 75% met the requirements related to reporting key methodological and statistical information. However, 89.3% used convenience samples and 92.86% lacked of a priori power analysis. There was low compliance with open science recommendations, such as pre-registration of studies (0%) and availability of databases (3.57%), which raise concerns about the validity, generalisability, and reproducibility of the findings.

Interpretation: While the importance of offering rapid evidence-based responses to mitigate mental health problems stemming from the COVID-19 pandemic is undeniable, it should not be done at the expense of sacrificing scientific rigor. The results of this study may stimulate researchers and funding agencies to try to orchestrate efforts and resources and follow standard codes of good science.