The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency

Abstract

Study preregistration has become increasingly popular in psychology, but itseffectivenessin restrictingpotentially biasingresearcher degrees of freedomremains unclear. We used an extensive protocol to assessthe strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies. We found that preregistrations often lackmethodological detailsandthat undisclosed deviations from preregistered plans are frequent. Combiningthe strictness and consistency results highlightsthat biases due to researcher degrees of freedom are prevalent and likelyinmanypreregistered studies.More comprehensive registration templates typically yieldedstricter and hence better preregistrations. Wedid not find that effectiveness ofpreregistrationsdifferedover timeor between original and replication studies.Furthermore,we found that operationalizations of variables were generally more effectively preregistered than other study parts. Inconsistencies between preregistrationsand published studies were mainly encountered for data collection procedures, statistical models, and exclusion criteria. Our results indicate that, to unlock the full potential of preregistration, researchers in psychology should aim to writestricter preregistrations,adhere to thesepreregistrations more faithfully, and more transparently report any deviations from the preregistrations. This could be facilitated by training and education to improve preregistration skills, as well as the development of more comprehensive templates.

Publication
MetaArxiV

Related