Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017). Tom E. Hardwicke et al. Perspectives on Psychological Science, March 8, 2021. https://doi.org/10.1177/1745691620979806

Abstract: Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.

Keywords: transparency, reproducibility, meta-research, psychology, open science

Discussion

Our evaluation of transparency and reproducibility-related research practices in a random sample of 250 psychology articles published between 2014 and 2017 shows that, although many articles were publicly available, crucial components of research—protocols, materials, raw data, and analysis scripts—were rarely made publicly available alongside them. Preregistration remained a nascent proposition with minimal adoption. The disclosure of funding sources and conflicts of interest was modest. Replication or evidence synthesis via meta-analysis or systematic review was infrequent (although, admittedly, only a relatively short time had elapsed since the articles had been published). Although there is evidence that some individual methodological reform initiatives have been effective in specific situations (e.g., Hardwicke et al., 2018; Nuijten et al., 2017; for review, see Hardwicke, Serghiou, et al., 2020), the findings of the current study imply that their collective, broader impact on the psychology literature during the examined period was still fairly limited in scope.

For most of the articles (65%) we examined, we could access a publicly available version (open access). This is higher than recent open-access estimates obtained for biomedicine (25%; Wallach et al., 2018) and the social sciences (40%; Hardwicke, Wallach, et al., 2020), as well as a large-scale automated analysis that suggested that 45% of the scientific literature published in 2015 was publicly available (Piwowar et al., 2018). Limiting access to academic publications reduces opportunities for researchers, policymakers, practitioners, and the general public to evaluate and make use of scientific evidence. One step psychologists can take to improve the public availability of their articles is to upload them to the free preprint server PsyArXiv (https://psyarxiv.com/). Uploading a preprint does not preclude publication at most journals (Bourne et al., 2017), although specific policies regarding open access can be checked on the Sherpa/Romeo database (http://sherpa.ac.uk/romeo/index.php).

The reported availability of research materials was modest in the articles we examined (14%), which is comparable to recent estimates in the social sciences (11%; Hardwicke, Wallach, et al., 2020) and lower than in biomedicine (33%; Wallach et al., 2018). Several reportedly available sets of materials were in fact not available because of broken links, an example of the “link-rot” phenomenon that has been observed by others trying to access research resources (Evangelou et al., 2005; Rowhani-Farid & Barnett, 2018). We also did not find any study protocols (an additional document detailing the study methods); however, it is unclear to what extent this results from a difference in norms between, for example, biomedicine (in which prespecified protocols are increasingly promoted; Ioannidis, Greenland, et al., 2014) and psychology (in which there may not be an expectation to provide methodological details in a separate protocol document). We did not examine whether sufficient methodological information was provided in the Method sections of articles, as this would have required domain-specific expertise in the many topics addressed by the articles in our sample. The availability of original research materials (e.g., survey instruments, stimuli, software, videos) and protocols enables the comprehensive evaluation of research (during traditional peer review and beyond; Vazire, 2017) and high-fidelity independent replication attempts (Open Science Collaboration, 2015; Simons, 2014), both of which are important for the verification and systematic accumulation of scientific knowledge (Ioannidis, 2012). Furthermore, reusing materials and protocols reduces waste and enhances efficiency (Chalmers & Glasziou, 2009; Ioannidis, Greenland, et al., 2014). Psychologists can share their materials and protocols online in various third-party repositories that use stable permalinks, such as the Open Science Framework² (OSF; see Klein et al., 2018). One observational study found that when the journal Psychological Science offered authors an open-materials badge there was a subsequent increase in the sharing of materials (Kidwell et al., 2016).

Data-availability statements in the articles we examined were extremely uncommon. This is consistent with accumulating evidence that suggests that the data underlying scientific claims are rarely immediately available (Alsheikh-Ali et al., 2011; Iqbal et al., 2016), although some modest improvement has been observed in recent years in biomedicine (Wallach et al., 2018). Although we did not request data from authors directly, such requests to psychology researchers typically have a modest yield (Vanpaemel et al., 2015; Wicherts et al., 2006). Most data appear to be effectively lost, including for some of the most influential studies in psychology and psychiatry (Hardwicke & Ioannidis, 2018b). Vanpaemel et al. (2015), for example, could not obtain 62% of the 394 data sets they requested from authors of papers published in four American Psychological Association journals in 2012. The sharing of raw data, which is the evidence on which scientists base their claims, enables verification through the independent assessment of analytic or computational reproducibility (Hardwicke, Bohn, et al., 2020; Hardwicke et al., 2018; LeBel et al., 2018) and analytic robustness (Steegen et al., 2016). Data sharing also enhances evidence synthesis, such as through individual participant-level meta-analysis (Tierney et al., 2015), and can facilitate discovery, such as through the merging of data sets and reanalysis with novel techniques (Voytek, 2016). Psychologists can improve data availability by uploading raw data to third-party repositories such as the OSF (Klein et al., 2018). Data sharing must be managed with caution if there are ethical concerns, but such concerns do not always preclude all forms of sharing or necessarily negate ethical motivations for sharing (Meyer, 2017). Furthermore, when data cannot be made available it is always possible to explicitly declare this in research articles and explain the rationale for not sharing (Morey et al., 2016). Journal policies that use badges to encourage data sharing (Kidwell et al., 2016) or mandate data sharing (Hardwicke et al., 2018; Nuijten et al., 2017) have been associated with marked increases in data availability in the journals that adopted them.

Of the articles we examined, only one shared an analysis script, a dearth consistent with assessments in biomedicine (Wallach et al., 2018), the social sciences (Hardwicke, Wallach, et al., 2020), and biostatistics (Rowhani-Farid & Barnett, 2018). Analysis scripts (a step-by-step description of the analysis in the form of computer code or instructions for recreating the analysis in point-and-click software) provide the most veridical documentation of how the raw data were filtered, summarized, and analyzed. Verbal descriptions of analysis procedures are often ambiguous, contain errors, or do not adequately capture sufficient detail to enable analytic reproducibility (Hardwicke, Bohn, et al., 2020; Hardwicke et al., 2018; Stodden et al., 2018). Psychologists can share their analysis scripts on a third-party repository, such as the OSF (Klein et al., 2018), and educational resources are available to help researchers improve the quality of their analysis code (Wilson et al., 2017). Sharing the computational environment in which analysis code successfully runs may also help to promote its longevity and trouble-free transfer to other researchers’ computers (Clyburne-Sherin et al., 2018).

Preregistration, which involves making a time-stamped, read-only record of a study’s rationale, hypotheses, methods, and analysis plan on an independent online repository, was rare in the articles we examined. Preregistration fulfills a number of potential functions (Nosek et al., 2019), including clarifying the distinction between exploratory and confirmatory aspects of research (Kimmelman et al., 2014; Wagenmakers et al., 2012) and enabling the detection and mitigation of questionable research practices such as selective-outcome reporting (Franco et al., 2016; John et al., 2012; Simmons et al., 2011). Preregistration is relatively new to psychology (Nosek et al., 2018, 2019), but similar concepts of registration have a longer history in the context of clinical trials in biomedicine (Dickersin & Rennie, 2012), in which they have become the expected norm (Zarin et al., 2017). However, clinical trials represent only a minority of biomedical research, and estimates suggest that preregistration is rare in biomedicine overall (Iqbal et al., 2016; Wallach et al., 2018). Preregistration is also rare in the social sciences (Hardwicke, Wallach, et al., 2020). There is no doubt that the number of preregistrations (and the related Registered Reports article format) is increasing in psychology (Hardwicke & Ioannidis, 2018a; Nosek et al., 2018); however, our findings suggest that efforts to promote preregistration may not yet have had widespread impact on routine practice. It is important to note that because there is a time lag between registration and study publication, our measures may underestimate adoption. Although norms and standards for preregistration in psychology are still evolving (Nosek et al., 2019), several dedicated registries, such as the OSF, will host preregistrations, and detailed guidance is available (Klein et al., 2018).

Our findings suggest that psychology articles were more likely to include funding statements (62%) and conflict-of-interest statements (39%) than social-science articles in general (31% and 15%, respectively; Hardwicke, Wallach, et al., 2020) but less likely than biomedical articles (69% and 65%, respectively; Wallach et al., 2018). It is possible that these disclosure statements are more common than most other practices we examined because they are often mandated by journals (Nutu et al., 2019). Disclosing funding sources and potential conflicts of interest in research articles helps readers to make informed judgments about the risk of bias (Bekelman et al., 2003; Cristea & Ioannidis, 2018). In the absence of established norms or journal mandates, authors may often assume that such statements are not relevant to them (Chivers, 2019). However, because the absence of a statement is ambiguous, researchers should ideally always include one, even if it is to explicitly declare that there were no funding sources and no potential conflicts of interest.

Of the articles we examined, 5% claimed to be a replication study—slightly higher than a previous estimate in psychology of 1% (Makel et al., 2012) and a similar estimate of 1% in the social sciences (Hardwicke, Wallach, et al., 2020) but comparable to a 5% estimate in biomedicine (Wallach et al. 2018). Only 1% of the articles we examined were cited by another article that claimed to be a replication attempt; of these articles, 11% were included in a systematic review, and 7% were included in a meta-analysis. Replication and evidence synthesis through systematic reviews and meta-analyses help to verify and build on the existing evidence base. However, it is unclear what an ideal frequency of these activities would be because they depend on many factors, such as how often studies are sufficiently similar to be amenable to synthesis methods. Although the current findings suggest that routine replication and evidence synthesis is relatively rare in psychology, many high-profile replication attempts have been conducted in recent years (Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012). In addition, because the articles we examined were published relatively recently, there may be some time lag before relevant replication and evidence-synthesis studies emerge. For example, in biomedicine at least, there is a geometric growth in the number of meta-analyses, and in many fields multiple meta-analyses are often conducted once several studies appear on the same research question (Ioannidis, 2016).

The current study has several caveats and limitations. First, our findings are based on a random sample of 250 articles, and the obtained estimates may not necessarily generalize to specific contexts, such as other disciplines, subfields of psychology, or articles published in particular journals. However, this target sample size was selected to balance informativeness with tractability, and the observed estimates have reasonable precision. Second, although the focus of this study was transparency and reproducibility-related practices, this does not imply that the adoption of these practices is sufficient to promote the goals they are intended to achieve. For example, poorly documented data may not enable analytic reproducibility (Hardwicke, Bohn, et al., 2020; Hardwicke et al., 2018), and inadequately specified preregistrations may not sufficiently constrain researcher degrees of freedom (Claesen et al., 2019; Bakker et al., 2020). Third, we relied only on published information. Direct requests to authors may have yielded additional information; however, as noted earlier, such requests to research psychologists are often unsuccessful (Hardwicke & Ioannidis, 2018a; Vanpaemel et al., 2015; Wicherts et al., 2006). Fourth, a lack of transparency may have been justified in some cases if there were overriding practical, legal, or ethical concerns (Meyer, 2017). However, no constraints of this kind were declared in any of the articles we examined. Last, the study can gauge the prevalence of the assessed practices only during a particular time period. The effect of reform initiatives introduced after the examined time period, such as the founding of the Society for Improving Psychological Science (http://improvingpsych.org), will not be represented in our findings.

The current findings imply the minimal adoption of transparency and reproducibility-related practices in psychology during the examined time period. Although researchers appear to recognize the problems of low credibility and reproducibility (Baker, 2016) and endorse the values of transparency and reproducibility in principle (Anderson et al., 2010), they are often wary of change (Fuchs et al., 2012; Houtkoop et al., 2018) and routinely neglect these principles in practice (Hardwicke, Wallach, et al., 2020; Iqbal et al., 2016; Wallach et al., 2018). There is unlikely to be a single remedy to this situation. A multifaceted approach will likely be required, with iterative evaluation and careful scrutiny of reform initiatives (Hardwicke, Serghiou, et al., 2020). At the educational level, guidance and resources are available to aid researchers (Crüwell et al., 2019; Klein et al., 2018). At the institutional level, there is evidence that funder and journal policies can be effective at fomenting change (Hardwicke et al., 2018; Nuijten et al., 2017), and these initiatives should be translated and disseminated where relevant. Heterogeneous journal policies (Nutu et al., 2019) may currently be disrupting efforts to establish norms and promote better standards in routine practice. The Transparency and Openness Promotion initiative promises to encourage the adoption and standardization of journal policies related to transparency and reproducibility (Nosek et al., 2015), but it remains to be seen how effective this initiative will be in practice. Aligning academic rewards and incentives (e.g., funding awards, publication acceptance, promotion, and tenure) with better research practices may also be instrumental in encouraging wider adoption of these practices (Moher et al., 2018).

The current study is one of several to examine the prevalence of transparency and reproducibility-related research practices across scientific disciplines (Hardwicke, Wallach, et al., 2020; Iqbal et al., 2016; Wallach et al., 2018). Here, we have sketched out some of the topography of psychology’s territory. Additional studies will be required to fill in areas of the map that have yet to be explored and increase the resolution in specific areas (e.g., subfields of psychology). Future studies can also add a temporal dimension by comparing new data with the baseline established here, allowing us to explore the evolution of this landscape over time.