A Model for Public Access to Trustworthy and Comprehensive Reporting of Research

A Model for Public Access to Trustworthy and Comprehensive Reporting of Research. Marina Broitman, Harold C. Sox, Jean Slutsky. JAMA, March 28, 2019. doi:10.1001/jama.2019.2807

The Patient-Centered Outcomes Research Institute (PCORI) was authorized by Congress in 2010 to fund comparative clinical effectiveness research. The legislation required the institute to guarantee peer review of all research results and to make those results publicly accessible within 90 days of their receipt, requirements that were the first of their kind for a US-based research funding organization. The authorizing legislation further stipulated that the peer review should assess the scientific integrity of each study and its adherence to the methodological standards established by the PCORI methodology committee.1 The law broadly defined the forms of peer review that would be acceptable.

The PCORI board of governors, after comment from patients, clinicians, professional interest groups, and journal editors, established a peer review process2 requiring all awardees to submit a comprehensive final report for PCORI-based external peer review. After final approval, the institute would post on its website lay and technical abstracts of the report; the complete, approved final report and study protocol; and a summary of the peer review critiques and the authors’ responses to those critiques. This Viewpoint describes the peer review system and the comprehensive report, which are the essential elements of this program, and discusses the problems each raise and the potential benefits.

The PCORI peer review process is similar to the review process at some journals in its use of methods experts and clinician-scientists as external reviewers. However, PCORI peer review is unique in its recruiting of patients, caregivers, health care professionals, and policy makers to critique the relevance and usefulness of the research. The institute engaged a contractor to manage the external peer review process to help ensure its impartiality. Each report receives 4 to 5 external reviews and a comprehensive review from the peer review editor.

The final research report is structured like a journal article, but is 3 to 4 times longer to accommodate a complete account of all study aims, methods, and results. The report also includes an account of the study’s adherence to the PCORI methodology standards1 and a section for patient engagement in research, in which the authors describe their partnership and collaborations with interested groups and individuals (including patients and other health care decision makers) in the development, implementation, and interpretation of the study. Per legislative requirements, the report must describe the study limitations and how the comparative effectiveness of the study interventions differs among various subgroups of study participants.

From October 6, 2016, to February 28, 2019, there were 275 final reports submitted for peer review and 206 had completed all revisions following peer review. As of February 28, 2019, PCORI has posted 177 publicly available lay and technical abstracts based on those reports as well as 41 of the full reports. Posting the study results could jeopardize journal publication; therefore, the institute waits 12 months after completion of peer review to post the final research report on its website unless the main study results have already been published in a journal article.

The institute met its legislative mandates. It established a peer review process and the requirement to write a comprehensive report, and it publishes lay and technical abstracts of the final report within 90 days of completing peer review. The vision of the sponsors of the legislation (ie, prompt public access to all study results following peer review) is achievable but at what cost and with what benefits? It is possible now to describe some of the challenges and speculate about potential benefits.

To assess the effects of requiring a comprehensive report, the institute surveyed 191 investigators about their experiences with final research report preparation and peer review and 52% responded. Some awardees raised complaints about the burden of writing a comprehensive final report and responding to peer review when they are preparing manuscripts to submit to journals. Others reported that writing the final report helped them to identify topics, text, and tables for focused articles. Many did not see an added value from the comprehensive final report, which increased their frustration with the time and effort involved. Survey respondents estimated that their research teams spent between 160 to 255 hours preparing the final report and responding to reviewer critiques.

The peer review process is taking longer than originally expected. Instead of the 4 to 6 months estimated by the PCORI board of governors,2 peer review takes a median of 8 to 9 months from submission to final acceptance. Unlike journals that can quickly reject unsuitable articles, the institute must accept and publicly post all reports. Some reports require many rounds of review and revision, which delays completion of peer review and public release of the results.

These problems are present and real, whereas the most substantial benefits lie in the future. First, the public is likely to benefit from a full report of research even when the results are unlikely to change practice. Because inconclusive studies are the least likely to be published in a scientific journal,3 the published body of evidence is incomplete and is likely biased toward positive results,4 which is a problem for systematic reviewers, guideline developers, and many others who rely on evidence to make decisions, including clinicians and patients. Journal articles and research registries each play a key role in informing the public, but a comprehensive final research report can fill a gap in public reporting by providing an in-depth account of a study, including all results and lessons learned from doing the research, regardless of study success.

Second, study authors may benefit from the institute’s peer review of their final research report before they submit manuscripts to a journal. Researchers can use the critiques by the PCORI peer reviewers to anticipate the concerns of journal reviewers and shape journal manuscripts accordingly. When the final research report for the institute is prepared (after publication of a journal article), authors can include information that journals typically do not publish such as a narrative account of how the study unfolded (including explanations of key decisions) and a full account of the patients and others who partnered with the researchers to conduct the study.

Dual peer review and publication could cause problems, especially when the description of research in a journal article conflicts with the final report. These differences are best handled by transparent reporting of the reasons for the discrepancies. When report authors disagree with the recommendations from the peer reviewers, both sides of the disagreement are captured in the peer review summary.

Third, a comprehensive final report supports the goal of increasing the efficiency of research by providing essentially unlimited space in which to describe how a 3- to 5-year study developed and evolved and to transmit all results and lessons learned to other researchers and funding agencies.5 Posted final research reports that provide all study results, including unpublished secondary or inconclusive results, may provide leads for others to investigate, describe pitfalls to avoid, or describe a line of investigation that led nowhere and should be avoided.

Fourth, the funder that requires a final research report can learn from systematically examining the research that it has supported, especially if reports have been peer reviewed. Ongoing review of its body of funded research could inform many aspects of the work of a funding agency such as the choice of high-priority research topics for funding announcements, instructions to applicants, the evaluation of research proposals, postaward negotiations to strengthen the study, and monitoring research while it is in progress. Peer review of adherence to the PCORI methodology standards has identified several recurring methodological shortcomings that the institute must address.

In the future, these potential benefits of requiring a peer-reviewed, publicly available comprehensive final report will be weighed against the costs. In a few years, it will be possible to determine if these final reports have expanded the reach of each research study, whether they are used in systematic reviews, and how lessons learned from peer review of the final reports have changed the way that the institute does its work. For the present, it may be enough to hope that an openly accessible peer-reviewed comprehensive final research report will help to increase public trust in research. It is possible that the sponsors of the legislation were on to something important when they set the institute on this path. However, the realization of their vision is in its infancy and its putative benefits are largely speculative. There is still much to learn.