How Best to Quantify Replication Success? A Simulation Study on the Comparison of Replication Success Metrics
Author
Abstract
Suggested Citation
DOI: 10.31219/osf.io/wvdjf
Download full text from publisher
References listed on IDEAS
- Leonhard Held, 2020. "A new standard for the analysis and design of replication studies," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 431-448, February.
- John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
- Robbie C M van Aert & Marcel A L M van Assen, 2017. "Bayesian evaluation of effect size after replicating an original study," PLOS ONE, Public Library of Science, vol. 12(4), pages 1-23, April.
- Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018.
"Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015,"
Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
- Camerer, Colin F. & Dreber, Anna & Holzmeister, Felix & Ho, Teck-Hua & Huber, Jürgen & Johannesson, Magnus & Kirchler, Michael & Nave, Gideon & Nosek, Brian A. & Pfeiffer, Thomas & Altmejd, Adam & But, 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Munich Reprints in Economics 62818, University of Munich, Department of Economics.
- Camerer, Colin & Dreber, Anna & Holzmeister, Felix & Ho, Teck Hua & Huber, Juergen & Johannesson, Magnus & Kirchler, Michael & Nave, Gideon & Nosek, Brian A. & Pfeiffer, Thomas, 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," SocArXiv 4hmb6, Center for Open Science.
- Leonard P Freedman & Iain M Cockburn & Timothy S Simcoe, 2015. "The Economics of Reproducibility in Preclinical Research," PLOS Biology, Public Library of Science, vol. 13(6), pages 1-9, June.
- Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
- Zacharias Maniadis & Fabio Tufano & John A. List, 2014.
"One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects,"
American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
- Zacharias Maniadis & Fabio Tufano & John A List, 2013. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," Discussion Papers 2013-07, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
- Zacharias Maniadis & Fabio Tufano & John List, 2013. "One Swallow Does not Make a Summer: New Evidence on Anchoring Effects," Levine's Working Paper Archive 786969000000000824, David K. Levine.
- Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
Citations
Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
Cited by:
- Leon C Reteig & Lionel A Newman & K Richard Ridderinkhof & Heleen A Slagter, 2022. "Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-23, January.
Most related items
These are the items that most often cite the same works as this one and are cited by the same works as this one.- Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018.
"Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015,"
Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
- Camerer, Colin & Dreber, Anna & Holzmeister, Felix & Ho, Teck Hua & Huber, Juergen & Johannesson, Magnus & Kirchler, Michael & Nave, Gideon & Nosek, Brian A. & Pfeiffer, Thomas, 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," SocArXiv 4hmb6, Center for Open Science.
- Camerer, Colin F. & Dreber, Anna & Holzmeister, Felix & Ho, Teck-Hua & Huber, Jürgen & Johannesson, Magnus & Kirchler, Michael & Nave, Gideon & Nosek, Brian A. & Pfeiffer, Thomas & Altmejd, Adam & But, 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Munich Reprints in Economics 62818, University of Munich, Department of Economics.
- Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019.
"The Dozen Things Experimental Economists Should Do (More of),"
Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
- Eszter Czibor & David Jimenez-Gomez & John List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Artefactual Field Experiments 00648, The Field Experiments Website.
- Eszter Czibor & David Jimenez-Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," NBER Working Papers 25451, National Bureau of Economic Research, Inc.
- Williams, Cole Randall, 2019. "How redefining statistical significance can worsen the replication crisis," Economics Letters, Elsevier, vol. 181(C), pages 65-69.
- Strømland, Eirik & Torsvik, Gaute, 2019. "Intuitive Prosociality: Heterogeneous Treatment Effects or False Positive?," OSF Preprints hrx2y, Center for Open Science.
- Leonhard Held, 2020. "A new standard for the analysis and design of replication studies," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 431-448, February.
- Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
- Isaiah Andrews & Maximilian Kasy, 2019.
"Identification of and Correction for Publication Bias,"
American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
- Isaiah Andrews & Maximilian Kasy, 2017. "Identification of and correction for publication bias," Papers 1711.10527, arXiv.org.
- Kasy, Maximilian & Andrews, Isaiah, 2018. "Identification of and correction for publication bias," MetaArXiv 49yst, Center for Open Science.
- Isaiah Andrews & Maximilian Kasy, 2017. "Identification of and Correction for Publication Bias," NBER Working Papers 23298, National Bureau of Economic Research, Inc.
- Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018.
"How to run an experimental auction: A review of recent advances,"
Working Papers
2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
- Canavari, Maurizio & Drichoutis, Andreas C. & Lusk, Jayson L. & Nayga, Rodolfo, 2018. "How to run an experimental auction: A review of recent advances," MPRA Paper 89715, University Library of Munich, Germany.
- Bull, Charles & Courty, Pascal & Doyon, Maurice & Rondeau, Daniel, 2019.
"Failure of the Becker–DeGroot–Marschak mechanism in inexperienced subjects: New tests of the game form misconception hypothesis,"
Journal of Economic Behavior & Organization, Elsevier, vol. 159(C), pages 235-253.
- Courty, Pascal & Bull, Charlie & Rondeau, Daniel & Doyon, Maurice, 2017. "Failure of the Becker-Degroot-Marschak Mechanism in Inexperienced Subjects: New Tests of the Game Form Misconception Hypothesis," CEPR Discussion Papers 12479, C.E.P.R. Discussion Papers.
- Oliver Braganza, 2020. "A simple model suggesting economically rational sample-size choice drives irreproducibility," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-19, March.
- Adam Altmejd & Anna Dreber & Eskil Forsell & Juergen Huber & Taisuke Imai & Magnus Johannesson & Michael Kirchler & Gideon Nave & Colin Camerer, 2019.
"Predicting the replicability of social science lab experiments,"
PLOS ONE, Public Library of Science, vol. 14(12), pages 1-18, December.
- Altmejd, Adam & Dreber, Anna & Forsell, Eskil & Huber, Jürgen & Imai, Taisuke & Johannesson, Magnus & Kirchler, Michael & Nave, Gideon & Camerer, Colin, 2019. "Predicting the replicability of social science lab experiments," Munich Reprints in Economics 78212, University of Munich, Department of Economics.
- John A. List & Azeem M. Shaikh & Yang Xu, 2019.
"Multiple hypothesis testing in experimental economics,"
Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
- John List & Azeem Shaikh & Yang Xu, 2016. "Multiple Hypothesis Testing in Experimental Economics," Artefactual Field Experiments 00402, The Field Experiments Website.
- John A. List & Azeem M. Shaikh & Yang Xu, 2016. "Multiple Hypothesis Testing in Experimental Economics," NBER Working Papers 21875, National Bureau of Economic Research, Inc.
- Anna Dreber & Magnus Johannesson & Yifan Yang, 2024.
"Selective reporting of placebo tests in top economics journals,"
Economic Inquiry, Western Economic Association International, vol. 62(3), pages 921-932, July.
- Dreber, Anna & Johannesson, Magnus & Yang, Yifan, 2023. "Selective Reporting of Placebo Tests in Top Economics Journals," I4R Discussion Paper Series 31, The Institute for Replication (I4R).
- Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019.
"Replication studies in economics—How many and which papers are chosen for replication, and why?,"
Research Policy, Elsevier, vol. 48(1), pages 62-83.
- Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
- Frank Mueller-Langer & Benedikt Fecher & Dietmar Harhoff & Gert G. Wagner, 2018. "Replication Studies in Economics: How Many and Which Papers Are Chosen for Replication, and Why?," JRC Working Papers on Digital Economy 2018-01, Joint Research Centre.
- Buffat, Justin & Praxmarer, Matthias & Sutter, Matthias, 2023. "The intrinsic value of decision rights: A replication and an extension to team decision making," Journal of Economic Behavior & Organization, Elsevier, vol. 209(C), pages 560-571.
- Jeff Miller & Rolf Ulrich, 2019. "The quest for an optimal alpha," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-13, January.
- Grabiszewski, Konrad & Horenstein, Alex, 2020. "Effort is not a monotonic function of skills: Results from a global mobile experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 634-652.
- Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
- Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
- Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArXiv 2b5k4, Center for Open Science.
Corrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:wvdjf. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .
Please note that corrections may take a couple of weeks to filter through the various RePEc services.