IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/ryvud.html
   My bibliography  Save this paper

Open Science & Development Engineering: Evidence to Inform Improved Replication Models

Author

Listed:
  • Galiani, Sebastian
  • Gertler, Paul
  • Romero, Mauricio

    (ITAM)

Abstract

Replication is a critical component of scientific credibility. Replication increases our confidence in the reliability of knowledge generated by original research. Yet, replication is the exception rather than the rule in economics. Few replications are completed and even fewer are published. Indeed, in the last 6 years, only 11 replication studies were published in top 11 (empirical) Economics Journals. In this paper, we examine why so little replication is done and propose changes to the incentives to replicate in order to solve these problems. Our study focuses on code replication, which seeks to replicate the results in the original paper uses the same data as the original study. Specifically, these studies seek to replicate exactly the same analyses performed by the authors. The objective is to verify that the analysis code is correct and confirm that there are no coding errors. This is usually done in a two-step process. The first step is to reconstruct the sample and variables used in the analysis from the raw data. The second step is to confirm that the analysis code (i.e., the code that fits a statistical model to the data) reproduces the reported results. By definition, the results reported in the original paper can then be replicated if this two-step procedure is successful. The threat of code replication provides an incentive for authors to put more effort into writing the code to avoid errors and incentive not to purposely misreport results. We analyze the effectiveness of code replication in the context a model that has three characteristics: 1. Unbiasedness: there is no “overturn bias” i.e., the model does not create incentives to find or claim mistakes in the original analysis. 2. Fair: all papers have some probability of being replicated, the sample of papers replicated is representative, and the likelihood that a paper is replicated is independent of author identity, topic, and results. 3. Cost: the model should provide the right incentives at low cost to be efficient. These characteristics are necessary to establish a creditable threat of valid replication that authors take seriously enough to modify behavior. Replication needs to be low cost for researchers to undertake it, fair so that studies face some positive probability of being replicated, and unbiased so that the original authors have reason to participate and the profession believe the replication results. We believe the current model for code replication does not have many of the desired characteristics and fails to provide the proper incentives to authors. We first show that there are low incentives for researchers to perform replication studies and that there is substantial “overturn bias” among editors. This is reflected in the replication studies published in economics. Since 2011 only 11 replication studies published in top journals, has been published, all of which overturned the results from the original paper. We also show poor author compliance with journal policies that require post acceptance posting of data and code, thereby raising the cost of replication. All of this means that there is a very low probability of a paper being replicated, overturn bias lowers the confidence in the replication results, and there is little incentive for authors to facilitate replication, and that the current model of replication fails to provide confidence in the integrity of published results. We outline a simple proposal to improve replication. The core of the proposal is to have journals perform the replication exercise post-acceptance but pre-publication. Specifically, authors submit their data and code after a conditional acceptance. Journals then verify that the code and data reproduce the results in the paper. For a random sample of papers the journal attempts to re-construct the code from scratch or search the code for errors. This can be an iterative process until authors and editors are able to reach agreement. If the results change, the editors can choose to re-review the paper. This simple procedure has three desirable properties. First, it is unbiased since there are no overturn bias incentives for the parties involved (editors/researchers). Second, it is fair because all papers have an equal probability of being replicated. Third, it is low-cost: there is little cost associated with having a research associate perform “push button exercises”, authors have strong incentives to cooperate pre-publication, and there are no adversarial feelings. Such a mechanism would create a strong incentive not to misreport findings and to ensure that code is free of errors.

Suggested Citation

  • Galiani, Sebastian & Gertler, Paul & Romero, Mauricio, 2017. "Open Science & Development Engineering: Evidence to Inform Improved Replication Models," MetaArXiv ryvud, Center for Open Science.
  • Handle: RePEc:osf:metaar:ryvud
    DOI: 10.31219/osf.io/ryvud
    as

    Download full text from publisher

    File URL: https://osf.io/download/58e38e619ad5a10230f49040/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/ryvud?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    2. Charles Ayoubi & Boris Thurm, 2023. "Knowledge diffusion and morality: Why do we freely share valuable information with Strangers?," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 32(1), pages 75-99, January.
    3. Kim, Youngseek & Adler, Melissa, 2015. "Social scientists’ data sharing behaviors: Investigating the roles of individual motivations, institutional pressures, and data repositories," International Journal of Information Management, Elsevier, vol. 35(4), pages 408-418.
    4. Stephanie B Linek & Benedikt Fecher & Sascha Friesike & Marcel Hebing, 2017. "Data sharing as social dilemma: Influence of the researcher’s personality," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-24, August.
    5. Claire M Mason & Paul J Box & Shanae M Burns, 2020. "Research data sharing in the Australian national science agency: Understanding the relative importance of organisational, disciplinary and domain-specific influences," PLOS ONE, Public Library of Science, vol. 15(8), pages 1-17, August.
    6. Benedikt Fecher & Mathis Fräßdorf & Gert G. Wagner, 2016. "Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful," Discussion Papers of DIW Berlin 1572, DIW Berlin, German Institute for Economic Research.
    7. Benedikt Fecher & Gert G. Wagner, 2016. "Open Access, Innovation, and Research Infrastructure," Publications, MDPI, vol. 4(2), pages 1-8, June.
    8. Yulin Yu & Daniel M. Romero, 2024. "Does the Use of Unusual Combinations of Datasets Contribute to Greater Scientific Impact?," Papers 2402.05024, arXiv.org, revised Sep 2024.
    9. Anneke Zuiderwijk, 2024. "Researchers’ Willingness and Ability to Openly Share Their Research Data: A Survey of COVID-19 Pandemic-Related Factors," SAGE Open, , vol. 14(1), pages 21582440241, March.
    10. Maja Jokić & Andrea Mervar & Stjepan Mateljan, 2018. "Scientific potential of European fully open access journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1373-1394, March.
    11. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    12. Garg, Prashant & Fetzer, Thiemo, 2024. "Causal Claims in Economics," OSF Preprints u4vgs, Center for Open Science.
    13. Younes, George Abi & Ayoubi, Charles & Ballester, Omar & Cristelli, Gabriele & de Rassenfosse, Gaetan & Foray, Dominique & Gaule, Patrick & van den Heuvel, Matthias & Webster, Beth & Zhou, Ling, 2020. "COVID-19: Insights from Innovation Economists (with French executive summary)," SocArXiv 65pgr, Center for Open Science.
    14. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 23(1), pages 49-79.
    15. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    16. Frank Mueller-Langer & Benedikt Fecher & Dietmar Harhoff & Gert G. Wagner, 2017. "The Economics of Replication," Discussion Papers of DIW Berlin 1640, DIW Berlin, German Institute for Economic Research.
    17. Courtney Butler & Brett Currier & Kira Lillard, 2021. "Safeguarding Research: A Review of Economics Journals’ Preservation Policies for Published Code and Data Files," Research Working Paper RWP 21-14, Federal Reserve Bank of Kansas City.
    18. Benedikt Fecher & Sascha Friesike & Marcel Hebing & Stephanie Linek & Armin Sauermann, 2015. "A Reputation Economy: Results from an Empirical Survey on Academic Data Sharing," RatSWD Working Papers 246, German Data Forum (RatSWD).
    19. Omar Díaz & Gabriela Riquelme & Gibrán Rivera, 2021. "Sharing Research Data: An Analysis of the Interest of Social Scientists in the Context of a Mexican University," Social Sciences, MDPI, vol. 10(10), pages 1-19, October.
    20. Lis-Gutiérrez, Jenny-Paola, 2015. "Gestión de la propiedad intelectual en museos [Management of intellectual property in museums]," MPRA Paper 68098, University Library of Munich, Germany.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:ryvud. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.