IDEAS home Printed from https://ideas.repec.org/p/fip/fedlwp/93317.html
   My bibliography  Save this paper

Paving the Road for Replications: Experimental Results from an Online Research Bibliography

Author

Abstract

Are users of a bibliographic database interested in learning about replications? Can they be induced to learn? To answer these questions, we performed an experiment at the online research bibliography, RePEc (Research Papers in Economics). RePEc is the main research bibliography for pre-prints and published papers in economics. Using randomized stratification, we allocated 324 replications and their corresponding original studies to clusters. We then drew from those clusters to construct treatment and control groups. Brightly colored tabs were added to the relevant webpages to alert visitors to the existence of a replication study. We then monitored traffic over three phases lasting several months: a) no treatment, b) treatment on one group, c) treatment on both groups. Our estimates indicate that this intervention generated an average click-through-rate (CTR) of 1.6%, resulting in a 13% increase in the visits to replication webpages, though only the former estimate was statistically significant.

Suggested Citation

  • Tom Coupé & W. Robert Reed & Christian Zimmermann, 2021. "Paving the Road for Replications: Experimental Results from an Online Research Bibliography," Working Papers 2021-013, Federal Reserve Bank of St. Louis, revised 24 Feb 2022.
    • Handle: RePEc:fip:fedlwp:93317
    DOI: 10.20955/wp.2021.013
    as

    Download full text from publisher

    File URL: https://s3.amazonaws.com/real.stlouisfed.org/wp/2021/2021-013.pdf
    File Function: Full Text
    Download Restriction: no
    File URL: https://libkey.io/10.20955/wp.2021.013?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2017. "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?," American Economic Review, American Economic Association, vol. 107(5), pages 46-51, May.
    2. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    3. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    4. C. Glenn Begley & Lee M. Ellis, 2012. "Raise standards for preclinical cancer research," Nature, Nature, vol. 483(7391), pages 531-533, March.
    5. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    6. Judit Bar-Ilan & Gali Halevi, 2018. "Temporal characteristics of retracted articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1771-1783, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tom Coupé & W. Robert Reed & Christian Zimmerman, 2021. "Paving the Road for Replications: Experimental Results from an Online Research Repository," Working Papers in Economics 21/09, University of Canterbury, Department of Economics and Finance.
    2. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    3. Hernández Alemán, Anastasia & León, Carmelo J., 2018. "La Réplica en el Análisis Económico Aplicado/Replication in Applied Economic Analysis," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 36, pages 317-332, Enero.
    4. Christophe Hurlin & Christophe Pérignon, 2020. "Reproducibility Certification in Economics Research," Working Papers hal-02896404, HAL.
    5. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    6. Sylvérie Herbert & Hautahi Kingi & Flavio Stanchi & Lars Vilhubern, 2021. "The Reproducibility of Economics Research: A Case Study," Working papers 853, Banque de France.
    7. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    8. Bull, Charles & Courty, Pascal & Doyon, Maurice & Rondeau, Daniel, 2019. "Failure of the Becker–DeGroot–Marschak mechanism in inexperienced subjects: New tests of the game form misconception hypothesis," Journal of Economic Behavior & Organization, Elsevier, vol. 159(C), pages 235-253.
    9. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    10. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    11. Benedikt Fecher & Mathis Fräßdorf & Gert G. Wagner, 2016. "Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful," Discussion Papers of DIW Berlin 1572, DIW Berlin, German Institute for Economic Research.
    12. Christophe Pérignon & Olivier Akmansoy & Christophe Hurlin & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johanneson & Michael Kirchler & Albert Menkveld & Michael Razen & Utz Weitzel, 2022. "Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance," Working Papers hal-03810013, HAL.
    13. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    14. Kiri, Bralind & Lacetera, Nicola & Zirulia, Lorenzo, 2018. "Above a swamp: A theory of high-quality scientific production," Research Policy, Elsevier, vol. 47(5), pages 827-839.
    15. Baltussen, Guido & Swinkels, Laurens & Van Vliet, Pim, 2021. "Global factor premiums," Journal of Financial Economics, Elsevier, vol. 142(3), pages 1128-1154.
    16. Fecher, Benedikt & Fräßdorf, Mathis & Hebing, Marcel & Wagner, Gert G., 2017. "Replikationen, Reputation und gute wissenschaftliche Praxis," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 68(2-3), pages 154-158.
    17. Emma McManus & David Turner & Tracey Sach, 2019. "Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics," PharmacoEconomics, Springer, vol. 37(11), pages 1371-1381, November.
    18. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    19. Brade, Raphael & Himmler, Oliver & Jäckle, Robert, 2018. "Normatively Framed Relative Performance Feedback – Field Experiment and Replication," MPRA Paper 88830, University Library of Munich, Germany.
    20. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.

    More about this item

    Keywords

    replications; RePEc; Experiment; Online Research Bibliography; Webpages; Click-throughs;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • Z0 - Other Special Topics - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedlwp:93317. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Anna Oates (email available below). General contact details of provider: https://edirc.repec.org/data/frbslus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.