IDEAS home Printed from https://ideas.repec.org/a/eee/jobhdp/v167y2021icp18-27.html
   My bibliography  Save this article

Pre-registration: Weighing costs and benefits for researchers

Author

Listed:
  • Logg, Jennifer M.
  • Dorison, Charles A.

Abstract

In the past decade, the social and behavioral sciences underwent a methodological revolution, offering practical prescriptions for improving the replicability and reproducibility of research results. One key to reforming science is a simple and scalable practice: pre-registration. Pre-registration constitutes pre-specifying an analysis plan prior to data collection. A growing chorus of articles discusses the prescriptive, field-wide benefits of pre-registration. To increase adoption, however, scientists need to know who currently pre-registers and understand perceived barriers to doing so. Thus, we weigh costs and benefits of pre-registration. Our survey of researchers reveals generational differences in who pre-registers and uncertainty regarding how pre-registration benefits individual researchers. We leverage these data to directly address researchers’ uncertainty by clarifying why pre-registration improves the research process itself. Finally, we discuss how to pre-register and compare available resources. The present work examines the who, why, and how of pre-registration in order to weigh the costs and benefits of pre-registration to researchers and motivate continued adoption.

Suggested Citation

  • Logg, Jennifer M. & Dorison, Charles A., 2021. "Pre-registration: Weighing costs and benefits for researchers," Organizational Behavior and Human Decision Processes, Elsevier, vol. 167(C), pages 18-27.
  • Handle: RePEc:eee:jobhdp:v:167:y:2021:i:c:p:18-27
    DOI: 10.1016/j.obhdp.2021.05.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0749597821000649
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.obhdp.2021.05.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    2. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    3. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    4. Nosek, Brian A. & Ebersole, Charles R. & DeHaven, Alexander Carl & Mellor, David Thomas, 2018. "The Preregistration Revolution," OSF Preprints 2dxu5, Center for Open Science.
    5. Tenney, Elizabeth R. & Costa, Elaine & Allard, Aurélien & Vazire, Simine, 2021. "Open science and reform practices in organizational behavior research over time (2011 to 2019)," Organizational Behavior and Human Decision Processes, Elsevier, vol. 162(C), pages 218-223.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matthews, Michael J. & Anglin, Aaron H. & Drover, Will & Wolfe, Marcus T., 2024. "Just a number? Using artificial intelligence to explore perceived founder age in entrepreneurial fundraising," Journal of Business Venturing, Elsevier, vol. 39(1).
    2. Yip, Jeremy A. & Schweitzer, Maurice E., 2022. "Norms for Behavioral Change (NBC) model: How injunctive norms and enforcement shift descriptive norms in science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 168(C).
    3. Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
    4. Minson, Julia A. & Bendersky, Corinne & de Dreu, Carsten & Halperin, Eran & Schroeder, Juliana, 2023. "Experimental studies of conflict: Challenges, solutions, and advice to junior scholars," Organizational Behavior and Human Decision Processes, Elsevier, vol. 177(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Sarstedt, Marko & Adler, Susanne J., 2023. "An advanced method to streamline p-hacking," Journal of Business Research, Elsevier, vol. 163(C).
    3. Brinkerink, Jasper & De Massis, Alfredo & Kellermanns, Franz, 2022. "One finding is no finding: Toward a replication culture in family business research," Journal of Family Business Strategy, Elsevier, vol. 13(4).
    4. Adler, Susanne Jana & Röseler, Lukas & Schöniger, Martina Katharina, 2023. "A toolbox to evaluate the trustworthiness of published findings," Journal of Business Research, Elsevier, vol. 167(C).
    5. Nosek, Brian A. & Errington, Timothy M., 2019. "What is replication?," MetaArXiv u4g6t, Center for Open Science.
    6. Albert J. Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüß & Michael Razen & Utz Weitzel & David Abad‐Díaz & Menachem (Meni) Abudy , 2024. "Nonstandard Errors," Journal of Finance, American Finance Association, vol. 79(3), pages 2339-2390, June.
      • Albert J. Menkveld & Anna Dreber & Félix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüss & Michael Razen & Utz Weitzel & Gunther Capelle-Blancard, 2021. "Non-Standard Errors," Documents de travail du Centre d'Economie de la Sorbonne 21033, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
      • Menkveld, Albert J. & Dreber, Anna & Holzmeister, Felix & Huber, Juergen & Johannesson, Magnus & Kirchler, Michael & Neusüss, Sebastian & Razen, Michael & Weitzel, Utz & Abad-Díaz, David & Abudy, Mena, 2021. "Non-Standard Errors," Working Papers 2021:17, Lund University, Department of Economics.
      • Albert J. Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neussüs & Michael Razen & Utz Weitzel & Christian Brownlees & Javier Gil-Bazo, 2021. "Non-Standard Errors," Working Papers 1303, Barcelona School of Economics.
      • Menkveld, Albert J. & Dreber, Anna & Holzmeister, Felix & Huber, Jürgen & Johannesson, Magnus & Kirchler, Michael & Neusüss, Sebastian & Razen, Michael & Weitzel, Utz, 2021. "Non-standard errors," IWH Discussion Papers 11/2021, Halle Institute for Economic Research (IWH).
      • Albert J. Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neussüs & Michael Razen & Utz Weitzel & Christian T. Brownlees & Javier Gil-Baz, 2021. "Non-standard errors," Economics Working Papers 1807, Department of Economics and Business, Universitat Pompeu Fabra.
      • Albert J. et al. Menkveld, 2021. "Non-Standard Errors," CESifo Working Paper Series 9453, CESifo.
      • Albert J Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüss & Michael Razen & Utz Weitzel & Gunther Capelle-Blancard & David Abad-Dí, 2021. "Non-Standard Errors," Post-Print halshs-03500882, HAL.
      • Francesco Franzoni & Roxana Mihet & Markus Leippold & Per Ostberg & Olivier Scaillet & Norman Schürhoff & Oksana Bashchenko & Nicola Mano & Michele Pelli, 2022. "Non-Standard Errors," Swiss Finance Institute Research Paper Series 22-09, Swiss Finance Institute.
      • Albert J. Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüss & Michael Razen & Utz Weitzel & Edwin Baidoo & Michael Frömmel & et al, 2021. "Non-Standard Errors," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 21/1032, Ghent University, Faculty of Economics and Business Administration.
      • Menkveld, Albert J. & Dreber, Anna & Holzmeister, Felix & Huber, Juergen & Johannesson, Magnus & Hasse, Jean-Baptiste & e.a.,, 2023. "Non-Standard Errors," LIDAM Reprints LFIN 2023002, Université catholique de Louvain, Louvain Finance (LFIN).
      • Moinas, Sophie & Declerck, Fany & Menkveld, Albert J. & Dreber, Anna, 2023. "Non-Standard Errors," TSE Working Papers 23-1451, Toulouse School of Economics (TSE).
      • Menkveld, Albert J. & Dreber, Anna & Holzmeister, Felix & Huber, Juergen & Johannesson, Magnus & Kirchler, Michael & Neusüß, Sebastian & Razen, Michael & Weitzel, Utz & Abad-Díaz, David & Abudy, Menac, 2024. "Nonstandard errors," LSE Research Online Documents on Economics 123002, London School of Economics and Political Science, LSE Library.
      • Menkveld, A. & Dreber, A. & Holzmeister, F. & Huber, J. & Johannesson, M. & Kirchler, M. & Neusüss, S. & Razen, M. & Neusüss, S. & Neusüss, S., 2021. "Non-Standard Errors," Cambridge Working Papers in Economics 2182, Faculty of Economics, University of Cambridge.
      • Menkveld, Albert J. & Dreber, Anna & Holzmeister, Felix & Huber, Jürgen & Johannesson, Magnus & Kirchler, Michael & Neusüss, Sebastian & Razen, Michael & Weitzel, Utz, 2021. "Non-standard errors," SAFE Working Paper Series 327, Leibniz Institute for Financial Research SAFE.
      • Albert J. Menkveld & Anna Dreber & Felix Holzmeister & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüss & Michael Razen & Utz Weitzel & David Abad-Dí­az & Menachem Abudy & Tobi, 2021. "Non-Standard Errors," Working Papers 2021-31, Faculty of Economics and Statistics, Universität Innsbruck.
      • Ferrara, Gerardo & Jurkatis, Simon, 2021. "Non-standard errors," Bank of England working papers 955, Bank of England.
      • Ciril Bosch-Rosa & Bernhard Kassner, 2023. "Non-Standard Errors," Rationality and Competition Discussion Paper Series 385, CRC TRR 190 Rationality and Competition.
      • Albert J Menkveld & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johannesson & Michael Kirchler & Sebastian Neusüss & Michael Razen & Utz Weitzel & Gunther Capelle-Blancard & David Abad-Dí, 2021. "Non-Standard Errors," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-03500882, HAL.
      • Menkveld, A. & Dreber, A. & Holzmeister, F. & Huber, J. & Johannesson, M. & Kirchler, M. & Neusüss, S. & Razen, M. & Neusüss, S. & Neusüss, S., 2021. "Non-Standard Errors," Janeway Institute Working Papers 2112, Faculty of Economics, University of Cambridge.
      • Wolff, Christian & Menkveld, Albert J. & Dreber, Anna & Holzmeister, Felix & Huber, Juergen & Johannesson, Magnus & Kirchler, Michael & Neusüess, Sebastian & Razen, Michael & Weitzel, Utz, 2021. "Non-Standard Errors," CEPR Discussion Papers 16751, C.E.P.R. Discussion Papers.
    7. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    8. Huber, Christoph & Kirchler, Michael, 2023. "Experiments in finance: A survey of historical trends," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    9. Merl, Robert & Stöckl, Thomas & Palan, Stefan, 2023. "Insider trading regulation and shorting constraints. Evaluating the joint effects of two market interventions," Journal of Banking & Finance, Elsevier, vol. 154(C).
    10. repec:grz:wpsses:2021-08 is not listed on IDEAS
    11. Christopher Allen & David M A Mehler, 2019. "Open science challenges, benefits and tips in early career and beyond," PLOS Biology, Public Library of Science, vol. 17(5), pages 1-14, May.
    12. Anna Dreber & Magnus Johannesson & Yifan Yang, 2024. "Selective reporting of placebo tests in top economics journals," Economic Inquiry, Western Economic Association International, vol. 62(3), pages 921-932, July.
    13. Grabiszewski, Konrad & Horenstein, Alex, 2020. "Effort is not a monotonic function of skills: Results from a global mobile experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 634-652.
    14. Mattia Prosperi & Jiang Bian & Iain E. Buchan & James S. Koopman & Matthew Sperrin & Mo Wang, 2019. "Raiders of the lost HARK: a reproducible inference framework for big data science," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    15. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    16. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArXiv 2b5k4, Center for Open Science.
    17. Sébastien Duchêne & Adrien Nguyen-Huu & Dimitri Dubois & Marc Willinger, 2022. "Risk-return trade-offs in the context of environmental impact: a lab-in-the-field experiment with finance professionals," CEE-M Working Papers hal-03883121, CEE-M, Universtiy of Montpellier, CNRS, INRA, Montpellier SupAgro.
    18. Strømland, Eirik, 2019. "Preregistration and reproducibility," Journal of Economic Psychology, Elsevier, vol. 75(PA).
    19. Filip Melinscak & Dominik R Bach, 2020. "Computational optimization of associative learning experiments," PLOS Computational Biology, Public Library of Science, vol. 16(1), pages 1-23, January.
    20. Granberg, Mark & Andersson, Per A. & Ahmed, Ali, 2020. "Hiring Discrimination Against Transgender People: Evidence from a Field Experiment," Labour Economics, Elsevier, vol. 65(C).
    21. Cantone, Giulio Giacomo, 2023. "The multiversal methodology as a remedy of the replication crisis," MetaArXiv kuhmz, Center for Open Science.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jobhdp:v:167:y:2021:i:c:p:18-27. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/obhdp .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.