Quotation errors in high-impact general science journals: Found a total error rate of 25%, which tracks well with error rates found in similar studies in other academic fields

Quotation errors in general science journals. Neal Smith and Aaron Cumberledge. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, October 14 2020. https://doi.org/10.1098/rspa.2020.0538

Abstract: Due to the incremental nature of scientific discovery, scientific writing requires extensive referencing to the writings of others. The accuracy of this referencing is vital, yet errors do occur. These errors are called ‘quotation errors’. This paper presents the first assessment of quotation errors in high-impact general science journals. A total of 250 random citations were examined. The propositions being cited were compared with the referenced materials to verify whether the propositions could be substantiated by those materials. The study found a total error rate of 25%. This result tracks well with error rates found in similar studies in other academic fields. Additionally, several suggestions are offered that may help to decrease these errors and make similar studies more feasible in the future.

4. Discussion

This study is the first review of quotation errors in high-impact general science journals. Errors were found to exist in considerable numbers. This demonstrates a weakness in the current use of references in scientific writing. There may be several reasons for these errors. Stochastic modelling suggests that 70–90% of references are copied second-hand from other articles' reference lists [26]. In addition, it has been argued through analysis of misprints that only about 20% of authors citing a paper have actually read the original [27]. As suggested by other quotation error researchers, authors could avoid errors through greater diligence [1,4–5,9]. There is also a lack of agreement regarding the correct reasons to include citations in scientific papers [28]. This could contribute to the citing of inappropriate references. Finally, quotation errors may occur though deliberate malpractice with the goal of increasing the citation metrics for the cited references [4].

Regardless of the cause, the most pragmatic approach to improving this problem is to improve the review and verification of references [1,20]. In the current state of academic literature, this is a very time-consuming task. In this study, it took two reviewers months of work to examine only 250 citations. The 500 articles from which we randomly selected our sample had a total of 26 344 references (many of which were cited multiple times). This suggests that it is unfeasible for editors or reviewers to thoroughly check all citations for substantiation. Therefore, we present two suggestions that would make systematic checking of references far more feasible.

First, most importantly, journals should change their citation styles to require page numbers. None of the high-impact journals reviewed require or even allow the inclusion of page numbers with in-text citations. In the verification process, a huge amount of time is spent searching through references to find the information being cited. Some books and reports are hundreds or thousands of pages long. Furthermore, even relatively short journal articles of 8–10 pages can be very dense and take a long time to thoroughly examine. Requiring page numbers (or paragraph numbers, etc.) places a slightly higher burden on the authors in exchange for significantly lightening the workload of potential reviewers. Lengthy references are often used to cite one specific piece of information, and it is not reasonable to expect reviewers to search through them to find that information. Page numbers should be required. One possible exception to this rule could be when referring to a study as a whole. However, even in those cases, propositions can nearly always be substantiated by referring to the page number of the introduction or abstract of a paper. This makes quotation errors easier to check for, increasing the likelihood of detection both before and after publication.

Requiring page numbers with in-text citations would constitute a significant change for academic publishers. The five journals in our study all use numbered endnotes, with a single endnote used for each reference regardless of how many times it is cited. To require page numbers in the text, these journals would have to either require page numbers to be included in each in-text citation (along with an endnote reference number), require separate endnotes containing page numbers for each citation of the reference, or abandon endnote citation altogether for some style of parenthetical citation. However, the continued prevalence of quotation errors is a significant problem that more than justifies the one-time cost of journals adopting new in-text citation policies.

We are not necessarily suggesting that systematic review of all quotations should be done by reviewers/editorial staff. However, systematic review of quotations would have benefits. There is a reason that the academic review process exists: to verify and improve the quality of scientific literature. Minimizing quotation errors is certainly one way to do that, and reference verification by journal staff has been significantly correlated with fewer quotation errors [10]. However, even in the absence of such a system of editorial review, including page numbers would give readers and reviewers in studies such as ours a better chance at successfully detecting quotation errors when they happen. Furthermore, the simple act of requiring authors to specifically locate and cite a specific page would necessitate them taking more care with their use of citations.

Our second suggestion refers specifically to the Impossible to Substantiate category. We are not aware of any previous studies that include an Impossible to Substantiate category, so further explanation and justification for its inclusion is in order. Essentially, this category refers to statements being cited that either lack a clear proposition or contain a proposition that cannot be substantiated through an outside reference. For example, an article might merely mention a novel material and cite a reference discussing that material. There is no specific proposition being made. The reference is simply giving additional background information. Therefore, substantiation is impossible. In other cases, statements cannot possibly be substantiated with a reference. For example, it was not uncommon in the articles surveyed for the methods section to be replaced (in whole or in part) with a citation. Here, there is a claim: ‘The methods from this reference were used’. However, it is not possible to substantiate this claim, because the article does not include the details of the methods used for comparison.

Some may consider this approach to be overly fastidious. However, there is no good reason to allow this type of inexact and non-verifiable referencing to pervade scientific literature. The most likely reason for this type of citing is to shorten articles to save printing space. This is a weak justification in the digital age. If background information is so unimportant that it does not merit a few words in the text (‘discussed in reference 15’ or ‘see reference 15 for the history of material X’ for example), then instead of using a propositionless citation, the information should be edited out of the paper proper and included as a supplement. The citing of methods sections and other unsubstantiatable claims could be dealt with in the same manner.

Of the previous quotation error studies reviewed, 71% did not mention string citations at all, and 14% specifically excluded string citations from their research [2–5,7–23]. Only one study specifically noted a difference in error rate between single and string citations. Surprisingly, that study came to a directly opposite conclusion regarding string citations, finding major errors more common in string citations [9]. The reason behind this discrepancy is unclear, although it may be related to the study's enormous sample size (more than six times larger than the other studies reviewed) or its very limited topic focus (peer-reviewed orthopedic literature related to the scaphoid). It is also not methodologically clear if the study required each reference in a string citation to substantiate the entire proposition being made. Our study did not require this. It required only that all the references in the string—as a whole—substantiate the entire proposition and for the reference being checked to contribute to that substantiation. References mentioned in string citations tend to make overlapping points and are often redundant [29]. Therefore, using our methodology, it seems reasonable to expect string citations to be more likely to be Fully Substantiated, not less. Still, the connection between string citations and substantiation needs further investigation.

Previous research has found quotation errors in the physical, life and social sciences [1–23]. This study extends that research to a cross section of high-impact general science journals, finding a similar rate of errors. However, further research is needed to more fully understand the problem. This paper reviewed only a total of 250 citations, which is less than 1% of the citations included in the five target journals over the course of a year. Although this sample is in keeping with the sample size of similar studies [2–5,7–8,10–23], a larger sample could produce more meaningful results. The main barrier to using a larger sample is the time cost involved. By improving citation and referencing standards for journal articles, reviewers should be able to check references more quickly. Furthermore, in this study the reviewers were not experts in the scientific disciplines to which the references belonged. Even though only two references (0.08%) were deemed too difficult to understand, some classifications required extensive research on the part of the reviewers. Expert reviewers should be able to work at a significantly faster pace, allowing for larger sample sizes. Further review of references can better show the extent of quotation errors in scientific literature. A better understanding of these errors can help decrease them, leading to better, more rigorously supported science.