Pre-Analysis Plans: An Early Stocktaking
Pre-analysis plans (PAPs) have been championed as a solution to the problem of research credibility, but without any evidence that PAPs actually bolster the credibility of research. We analyze a representative sample of 195 PAPs registered on the Evidence in Governance and Politics (EGAP) and American Economic Association (AEA) registration platforms to assess whether PAPs registered in the early days of pre-registration (2011-2016) were sufficiently clear, precise and comprehensive to achieve their objective of preventing fishing and reducing the scope for post-hoc adjustment of research hypotheses. We also analyze a subset of 93 PAPs from projects that resulted in publicly available papers to ascertain how faithfully they adhere to their pre-registered specifications and hypotheses. We find significant variation in the extent to which PAPs registered during this period accomplished the goals they were designed to achieve. We discuss these findings in light of both the costs and benefits of pre-registration, showing how our results speak to the various arguments that have been made in support of and against PAPs. We also highlight the norms and institutions that will need to be strengthened to augment the power of PAPs to improve research credibility and to create incentives for researchers to invest in both producing and policing them.
Perspectives on Politics 21(1): 174–190, 2023.
Journal Data & Replication Blog EGAP Q&A
Citation
@article{k._ofosu2023,
author = {K. Ofosu, George and N. Posner, Daniel},
title = {Pre-Analysis {Plans:} {An} {Early} {Stocktaking}},
journal = {Perspectives on Politics},
volume = {21},
number = {1},
pages = {174-190},
date = {2023-03-01},
url = {https://www.cambridge.org/core/journals/perspectives-on-politics/article/preanalysis-plans-an-early-stocktaking/94E7FAE76001C45A04E8F5E272C773CE},
langid = {en},
abstract = {Pre-analysis plans (PAPs) have been championed as a
solution to the problem of research credibility, but without any
evidence that PAPs actually bolster the credibility of research. We
analyze a representative sample of 195 PAPs registered on the
Evidence in Governance and Politics (EGAP) and American Economic
Association (AEA) registration platforms to assess whether PAPs
registered in the early days of pre-registration (2011-2016) were
sufficiently clear, precise and comprehensive to achieve their
objective of preventing fishing and reducing the scope for post-hoc
adjustment of research hypotheses. We also analyze a subset of 93
PAPs from projects that resulted in publicly available papers to
ascertain how faithfully they adhere to their pre-registered
specifications and hypotheses. We find significant variation in the
extent to which PAPs registered during this period accomplished the
goals they were designed to achieve. We discuss these findings in
light of both the costs and benefits of pre-registration, showing
how our results speak to the various arguments that have been made
in support of and against PAPs. We also highlight the norms and
institutions that will need to be strengthened to augment the power
of PAPs to improve research credibility and to create incentives for
researchers to invest in both producing and policing them.}
}