The past decade has seen concerns rise about the robustness and replicability of results across many different domains of science, ranging from basic studies to applied work (
1
, 2
, 3
, 4
, 5
, 6
, 7
, 8
). As one way to address such concerns, the open science movement has promoted the preregistration of hypotheses and analyses for research studies as a means to enhance the quality of scientific results and to increase the likelihood that findings are robust and able to be replicated over time (9
). There are now many platforms and options available for preregistration, ranging from websites that allow researchers to upload and share preregistrations (e.g., the Center for Open Science [https://osf.io/], AsPredicted [https://aspredicted.org/], PROSPERO [www.crd.york.ac.uk/prospero], and ClinicalTrials.gov) to the formal review and publication of registered reports (RRs) (10
,11
), wherein methods and analyses are reviewed prior to data collection and the results are published regardless of the outcome if the reviewed methods are followed. RRs have a long history in psychological research (12
), with a type of RR started at the European Journal for Parapsychology in the 1970s (this journal is no longer in operation); the Lancet initiated articles that included protocols of proposed research in 1997.Recent results suggest that the benefits of preregistration are starting to bear fruit, with evidence that readers trust empirical research findings more when they were preregistered (
13
), that the rigor of the science in RRs is rated more highly (14
), and that preregistration improves the estimation of effect sizes (15
) and helps reduce the publication bias for positive results (16
).Options to publish RRs are becoming increasing frequent in broad disciplinary journals (see www.cos.io/initiatives/registered-reports for a list of journals offering RRs) but are still relatively rare in journals that are focused on mental health research, particularly journals focused on psychiatry research. Biological Psychiatry: Global Open Science is one such outlet offering an option for RRs with the goal of helping to lead the field in this regard. This editorial introduces readers to the RR option, addresses some concerns or misconceptions about preregistration and RRs, and argues for the greater use of preregistration in general and RRs in particular in mental health research for both studies involving new data collection and those involving preexisting data.
One can argue that preregistration already has a strong history in at least some aspects of mental health research, stemming in part from the emergence of requirements to preregister study designs and analyses for clinical trials and a growing emphasis on preregistration of meta-analyses and systematic reviews. However, preregistration has not been sufficiently adopted in mental health research more broadly, and this is particularly true for the pursuit for RRs, which have rarely been used in mental health research. I acknowledge my own limitations in the previous use of preregistration and RRs and thank my students and trainees for their leadership in this regard in my own lab group. I address concerns about preregistration and RRs in psychopathology research in this editorial and argue that greater engagement with preregistration will serve to enhance the quality of research and the robustness and replicability of our results in addition to being well worth the added effort in the long run.
As noted above, there are many different avenues for investigators to preregister the methods and analyses of their proposed research, but there is great variability in the amount of detail that different individuals provide in such preregistrations. These venues do not formally review such preregistrations prior to upload or “publishing,” and thus the degree to which such preregistration will affect the review of the eventually published results is variable. In other words, preregistrations with little detail and specificity are much less likely to generate enhanced confidence than preregistrations with a great deal of detail. For example, for studies with null results or unexpected findings, the prior public release of a highly detailed preregistration that includes prospective power analyses can significantly enhance confidence in the findings. In my experience as an editor, this has led reviewers to have a much more positive view of the informativeness of null results. RRs provide an avenue to ensure that preregistrations are of sufficient detail and quality to lead to that type of confidence.
RRs typically involve two stages. The first stage is a review of a detailed protocol of the study design and/or analyses, and this stage is peer reviewed just as a full manuscript would be. Once “accepted,” that stage 1 report is held until the study/analyses are complete and the author submits the complete manuscript. If the author followed the protocol, the results are published regardless of outcome (see below for additional considerations of new and more exploratory analyses). There are concerns that investigators often have when pursuing RRs in mental health research, each of which can be overcome.
Study Length
Much of the action in preregistration and RRs has been in the domain of psychological research, where many studies can be conducted relatively quickly, and the time from acceptance of a stage 1 RR to stage 2 publication might be fairly short—in the range of 6 months to 1 or 2 years. In contrast, many studies on the causes or treatment of mental illness might take many years to conduct, resulting in longer delays between stages 1 and 2, leading investigators to be concerned that this gap in time precludes the pursuit of RRs. This concern regarding length is understandable but no different than concerns regarding length in clinical trials, which have required preregistration for many years. In some ways, one could argue that lengthy and expensive studies benefit even more from preregistration and the pursuit of RRs, as it makes it much more likely that there will be high-quality results publishable in strong journals regardless of the outcome. In addition, for grant-funded research, many investigators have spent a great deal of time and effort generating clear hypotheses and outstanding methods, and these grant applications can serve as the starting basis for preregistration and even RRs.
Unexpected Hurdles and Exploratory Analyses
A second set of concerns about RRs and preregistration in general is that it is often difficult to know all the necessary statistical choices ahead of time, that unexpected hurdles can arise that necessitate design or analysis changes, and that some questions warrant exploratory analyses. However, many of these concerns are not actually hurdles for preregistration or RRs. First, one can preregister exploratory research (
17
), allowing authors to be clearer about how they will handle multiple comparison correction, replication, etc. Second, additions and changes can be made to designs and analyses after preregistration and even in RRs if authors are crystal clear about why and how such changes are made. The goal is to promote transparency and clearly distinguish between hypothesis-driven and exploratory analyses, both of which are highly valuable, and not to preclude data exploration or new analyses that arise out of unexpected findings. One of the most challenging situations is when authors are unable to meet the minimum sample size that they determined a priori to be necessary for sufficient power. In such situations, authors can choose to pursue publication of the work as a regular research article rather than an RR if they are unable to generate the minimum sample sizes that they prespecified in their RR. Of note, several researchers are likely to be in this position due to COVID-19 disruptions, adding impetus to the need for funding agencies to provide additional support for such research to achieve well-powered designs.Getting Scooped
A third concern sometimes articulated about preregistration and RRs is worry that others will read about the preregistration, do the work first, and “scoop” the author. While there is no way to guarantee that this will never happen, in practice it is likely to be a rare occurrence for several reasons. First, in cases of new data collection, it is unlikely that others have the data necessary to test an author’s specific hypotheses. Second, the preregistration itself serves as evidence of the timing of an investigator’s hypotheses. Third, in the case of RRs, publication is guaranteed no matter how many similar papers are published between stage 1 acceptance and the submission of stage 2 results.
Working With Preexisting Data
A fourth concern is that there is an increasing emphasis on the use of preexisting public release data for analyses and worries that because the data already exist this precludes preregistration or the pursuit of RR (
18
, 19
, 20
). It is true that preregistration is more challenging when all data are already available, and in theory an author could have conducted all analyses before submission of the preregistration. However, in practice, this would only be an issue for authors who knowingly lie about not having conducted analyses before preregistration (something that is hopefully an infrequent event). In addition, there are ways to document time stamps as to when authors acquired access to public release data that can be used to confirm preregistration prior tobefore data access. Perhaps more useful and compelling for RRs is the fact that many large public access databases are releasing data in waves in ways that are highly amenable to preregistration and RRs prior to any possibility of data access. A good example of this is the Adolescent Brain and Cognitive Development Study, which provides annual releases of data as additional participants and waves are acquired. Investigators who plan to use the Adolescent Brain and Cognitive Development Study data could submit preregistrations or RRs related to upcoming but unreleased waves of data whether they propose hypothesis-driven or exploratory analyses, an approach that maximizes the utility of such publicly available data.I am optimistic that RRs and preregistration more generally will become increasingly the norm in all fields of science, and that these practices will serve to further enhance the quality of our science, leading us closer to the search for causes and cures for mental illness. I have addressed some of the concerns that might be precluding individuals in this field from taking advantage of preregistration and the pursuit of RRs and hope that this discussion has alleviated some of these concerns. In addition, I hope that Biological Psychiatry: Global Open Science can contribute to the revolution in terms of open science practices by providing options for RRs that are serving to move science forward in terms of rigor, robustness, and reproducibility.
Acknowledgments and Disclosures
As an editor for the Biological Psychiatry family of journals, DMB receives an editorial stipend from the Society of Biological Psychiatry. She reports no other biomedical financial interests or potential conflicts of interest related to this editorial. Her complete list of disclosures of financial and conflicts of interests are available here: https://www.bpsgos.org/bpsgos-editorial-disclosures.
References
- Non-replication of association for six polymorphisms from meta-analysis of genome-wide association studies of Parkinson’s disease: Large-scale collaborative study.Am J Med Genet B Neuropsychiatr Genet. 2010; 153B: 220-228
- An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017).R Soc Open Sci. 2020; 7: 190806
- Why most discovered true associations are inflated.Epidemiology. 2008; 19: 640-648
- Why science is not necessarily self-correcting.Perspect Psychol Sci. 2012; 7: 645-654
- Reproducible research practices and transparency across the biomedical literature.PLoS Biol. 2016; 14e1002333
- Replication in genome-wide association studies.Stat Sci. 2009; 24: 561-573
- Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017.PLoS Biol. 2018; 16e2006930
- What the replication crisis means for intervention science.Int J Psychophysiol. 2020; 154: 3-5
- Preregistration is hard, and worthwhile.Trends Cogn Sci. 2019; 23: 815-818
- Registered reports: a method to increase the credibility of published results.Soc Psychol. 2014; 45: 137-141
- Mapping the universe of registered reports.Nat Hum Behav. 2018; 2: 793-796
- What’s next for registered reports?.Nature. 2019; 573: 187-189
- The effect of preregistration on trust in empirical research findings: Results of a registered report.R Soc Open Sci. 2020; 7: 181351
- Initial evidence of research quality of registered reports compared with the standard publishing model [published online ahead of print Jun 24].Nat Hum Behav. 2021;
- Preregistration and reproducibility.J Econ Psychol. 2019; 75: 102143
- An excess of positive results: Comparing the standard psychology literature with registered reports.Adv Methods Pract Psychol Sci. 2021; 4: 1-12
- Preregistration of exploratory research: Learning from the golden age of discovery.PLoS Biol. 2020; 18e3000690
- A step-by-step guide on preregistration and effective data sharing for psychopathology research.J Abnorm Psychol. 2019; 128: 517-527
- Preregistration of analyses of preexisting data.Psychol Belg. 2019; 59: 338-352
- A plea for preregistration in personality disorders research: The case of psychopathy.J Pers Disord. 2021; 35: 161-176
Article info
Publication history
Accepted:
July 12,
2021
Received in revised form:
July 8,
2021
Received:
June 30,
2021
Identification
Copyright
© 2021 The Authors. Published by Elsevier Inc on behalf of the Society of Biological Psychiatry.
User license
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) | How you can reuse 
Elsevier's open access license policy
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0)
Permitted
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
Not Permitted
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy
