2022 年 4 巻 2 号 p. 49-52
Systematic reviews (SRs) are considered the most reliable forms of evidence to guide clinical decision-making. However, the number of SRs has substantially increased over the last 20 years and led to massive production of unnecessary, misleading, and conflicted SRs, which is becoming a problem [1–3].
Researchers often refer to the International Prospective Register of Systematic Reviews (PROSPERO), a registry for SR protocols, before beginning a new SR to avoid the publication of duplicate studies that address the same research questions [4, 5]. Researchers can choose to register protocols outside PROSPERO because PROSPERO limits the word counts for SR contents and does not accept the registration of SRs “without an outcome of clear relevance to the health of humans” [6]. Moreover, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2020 suggested that systematic reviewers can register SR protocols in other platforms such as Open Science Framework (OSF) Registries [7]. Until 2019, 44 preprint servers have been launched for medical and biomedical sciences [8]. Protocols.io, ClinicalTrials.gov, the International Clinical Trials Registry Platform (ICTRP), and OSF Registries also allow the registration of SR protocols. Here, we performed a meta-epidemiological study to determine how often SR protocols are registered outside of PROSPERO and the characteristics of these SRs. This short report was necessary because this study could change the methodology in SR. We hypothesized that identifying SR protocols would prevent 21% of redundant SRs because a previous study suggested that 21% of published SRs followed SR protocols [9].
This was a methodological study, and the details of the methodology are described in Supplementary Text 1. The study protocol is registered in protocols.io (dx.doi.org/10.17504/protocols.io.bhxpj7mn) [10]. We searched all SR protocols registered on both PROSPERO and non-PROSPERO platforms, which included 48 platforms with keywords related to SRs. The search period was from January 1, 2011 to September 8, 2020. All protocols on PROSPERO were included in this study. For non-PROSPERO protocols, we screened the titles of records and assessed their eligibility (Supplementary Text 1). We tabulated the number and proportion of SR protocols in the PROSPERO and non-PROSPERO platforms by calendar years. Additionally, we randomly selected 100 samples from each of the included PROSPERO and non-PROSPERO protocols and reported their characteristics. Moreover, we similarly investigated the duplications on PROSPERO among non-PROSPERO registrations among the random samples to clarify what percentage of protocols can only be found by searching non-PROSPERO platforms. We conducted a pre-specified sensitivity analysis to exclusively focus on the proportion of SR protocols related to coronavirus disease 2019 (COVID-19). The numerator of the proportion was the number of SR protocols related to COVID-19 in non-PROSPERO platforms. In contrast, the denominator of the proportion was the number of SR protocols related to COVID-19 in PROSPERO plus non-PROSPERO platforms.
Differences between the Protocol and the ManuscriptWe reconsidered the sample size of the random sampling and changed 300 samples to 200 because the reporting characteristics of the included PROSPERO and non-PROSPERO protocols are less important than the primary outcome, the proportion of SR protocols registered outside of PROSPERO.
A flowchart of the study selection is shown in Fig. 1. We obtained 89,442 and 560 records from the PROSPERO and non-PROSPERO platforms, respectively. The numbers and proportions of protocols from each platform are summarized in Table 1. The proportion of protocol registrations in the non-PROSPERO platforms remained at 1% from 2011 to 2020. However, the total number of records in OSF Registries significantly increased in 2020 (Table 1). In the sensitivity analysis, after focusing only on protocols related to COVID-19, the proportion of non-PROSPERO protocols increased from 0% in 2019 to 3% in 2020. The total number of COVID-19 studies was one in 2019 and 2110 in 2020. In the non-PROSPERO platform, details regarding the country and funding were unclear in 23% and 29% of protocols, respectively (Supplementary Table 1). Twelve percent of non-PROSPERO protocols were registered in PROSPERO as duplicate records.
Abbreviations: PROSPERO = International Prospective Register of Systematic Reviews; ICTRP = International Clinical Trials Registry Platform; OSF = Open Science Framework.
| Year | 2011 (n = 288) |
2012 (n = 769) |
2013 (n = 1,534) |
2014 (n = 2,713) |
2015 (n = 5,612) |
2016 (n = 8,298) |
2017 (n = 10,841) |
2018 (n = 15,709) |
2019 (n = 14,644) |
2020 (n = 29,594) |
|---|---|---|---|---|---|---|---|---|---|---|
| PROSPERO | 284 (99) | 759 (99) | 1,525 (99) | 2,707 (100) | 5,600 (100) | 8,279 (100) | 10,822 (100) | 15,663 (100) | 14,569 (99.5) | 29,234 (99) |
| Non-PROSPEROa | 4 (1) | 10 (1) | 9 (1) | 6 (0) | 12 (0) | 19 (0) | 19 (0) | 46 (0) | 75 (0.5) | 360 (1) |
| arXiv | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 1 (0) | 0 (0) | 1 (0) | 0 (0) | 2 (0) | 0 (0) |
| ClinicalTrials.gov | 3 (1) | 9 (1) | 6 (0) | 3 (0) | 5 (0) | 11 (0) | 5 (0) | 11 (0) | 6 (0) | 11 (0) |
| F1000 Research | NA | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 2 (0) | 3 (0) |
| HRB Open Research | NA | NA | NA | NA | NA | NA | NA | 1 (0) | 11 (0) | 12 (0) |
| ICTRP | 1 (0) | 1 (0) | 3 (0) | 1 (0) | 6 (0) | 6 (0) | 8 (0) | 13 (0) | 10 (0) | 44 (0) |
| medRxiv | NA | NA | NA | NA | NA | NA | NA | NA | 7 (0) | 44 (0) |
| OSF Preprints | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 1 (0) | 2 (0) | 18 (0) |
| OSF Registries | NA | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 3 (0) | 5 (0) | 22 (0) | 204 (1) |
| PeerJ Preprints | NA | NA | 0 (0) | 1 (0) | 0 (0) | 1 (0) | 1 (0) | 3 (0) | 0 (0) | NA |
| Protocols.io | NA | NA | NA | 0 (0) | 0 (0) | 0 (0) | 1 (0) | 9 (0) | 5 (0) | 11 (0) |
| PsyArxiv | NA | NA | NA | NA | NA | 0 (0) | 0 (0) | 2 (0) | 3 (0) | 2 (0) |
| SSRN | 0 (0) | 0 (0) | 0 (0) | 1 (0) | 0 (0) | 1 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| SciELO Preprints | NA | NA | NA | NA | NA | NA | NA | 0 (0) | 0 (0) | 4 (0) |
| SocArXiv | NA | NA | NA | NA | NA | NA | 0 (0) | 0 (0) | 0 (0) | 1 (0) |
| SportRxiv | NA | NA | NA | NA | NA | NA | 0 (0) | 0 (0) | 0 (0) | 2 (0) |
| Wellcome Open Research | NA | NA | NA | NA | NA | 0 (0) | 0 (0) | 1 (0) | 5 (0) | 4 (0) |
Abbreviations: PROSPERO = International Prospective Register of Systematic Reviews; HRB = Health Research Board; ICTRP = International Clinical Trials Registry Platform; OSF = Open Science Framework; SSRN = Social Science Research Network; SciELO = Scientific Electronic Library Online; NA = not available
Values are shown as numbers (percentages).
We described “NA” if the platforms did not exist or did not accept registrations throughout the 10-year period.
Creation years of the platforms were as follows: PROSPERO in 2011, arXiv in 1991, ClinicalTrials.gov in 2000, F1000 Research in 2012, HRB Open Research in 2018, ICTRP in 2005, medRxiv in 2019, OSF Preprints in 2007, OSF Registries in 2012, PeerJ Preprints in 2013, Protocols.io in 2014, PsyArXiv in 2016, SSRN in 1994, SciELO Preprints in 2018, SocArXiv in 2017, SportRxiv in 2017, Wellcome Open Research in 2016.
PeerJ Preprints did not accept new registrations since September 2019. We referred to the creation year of PROSPERO in “Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev 2012;1:2. doi:10.1186/2046-4053-1-2.” We referred to the creation year of ClinicalTrials.gov and ICTRP on their official sites. We referred to the creation year of OSF Registries in https://www.cos.io/blog/new-osf-registries-enhancements-improve-efficiency-and-quality-registrations. We referred to the creation year of Protocols.io in “Teytelman L, Stoliartchouk A, Kindler L, Hurwitz BL (2016) Protocols.io: Virtual Communities for Protocol Development and Discussion. PLoS Biol 14(8): e1002538. doi:10.1371/journal.pbio.1002538.” We referred to the creation year of other platforms in “Kirkham JJ, Penfold NC, Murphy F, et al. Systematic examination of preprint platforms for use in the medical and biomedical sciences setting. BMJ Open 2020;10:e041849. doi:10.1136/bmjopen-2020-041849.”
a Non-PROSPERO included 16 preprint platforms.
Although the number of SR protocols in both PROSPERO and non-PROSPERO platforms increased rapidly in 2020, the proportion of non-PROSPERO protocols was exceedingly small and remained unchanged over the 10 year period. Furthermore, 12% of non-PROSPERO protocols were also registered in PROSPERO. Our findings suggest that finding duplicate studies with similar research questions was unlikely when searching the non-PROSPERO protocols. However, it should be noted that our study was based on data before PRISMA 2020 dissemination, which suggested that systematic reviewers can register SR protocols in other platforms apart from PROSPERO [7]. In contrast, when we restricted our search to COVID-19 studies, the proportion of non-PROSPERO protocols increased from 0% to 3%. Therefore, a future update regarding this study may be needed, especially for emerging diseases. Systematic reviewers should describe the country of the corresponding authors and funding (for SR protocols in non-PROSPERO) because this information was often unclear in non-PROSPERO SR protocols compared with PROSPERO SR protocols.
This study has some limitations. First, we may have underestimated the proportion of non-PROSPERO protocols because we only included non-PROSPERO protocols that included the term “systematic review” in their titles, whereas we included all PROSPERO protocols regardless of the title. Second, the comparability between PROSPERO and non-PROSPERO protocols was not ensured in terms of the research field. Hence, future research focusing on the research field is needed. Third, we did not search for protocol publications in individual peer-review journals. Lastly, identifying SR protocols cannot prevent 79% of redundant SRs published in major journals [9].
The proportion of non-PROSPERO protocols remained unchanged over the 10 year period. Systematic reviewers might not need to search non-PROSPERO protocols for duplicate information regarding similar research questions before the PRISMA 2020 dissemination [7]. Nevertheless, future research is needed because some repositories, such as OSF Registries, have been rapidly increasing their registrations recently.
The authors declare that they have no competing interests.
We would like to thank the organizations for managing PROSPERO, 44 preprint platforms, Protocols.io, ClinicalTrials.gov, ICTRP, and OSF Registries. We also thank Editage (www.editage.jp) for their English language editing services.
The English editing fee was supported by the Scientific Research Works Peer Support Group (not-for-profit organization). The funders played no role in the design of the study and collection, analysis, and interpretation of data, and in writing the manuscript.
