IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/75461.html
   My bibliography  Save this paper

Evaluating replicability of laboratory experiments in Economics

Author

Listed:
  • Camerer, Colin
  • Dreber, Anna
  • Forsell, Eskil
  • Ho, Teck-Hua
  • Huber, Jurgen
  • Johannesson, Magnus
  • Kirchler, Michael
  • Almenberg, Johan
  • Altmejd, Adam
  • Chan, Taizan
  • Heikensten, Emma
  • Holzmeister, Felix
  • Imai, Taisuke
  • Isaksson, Siri
  • Nave, Gideon
  • Pfeiffer, Thomas
  • Razen, Michael
  • Wu, Hang

Abstract

The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.

Suggested Citation

  • Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:75461
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/75461/1/MPRA_paper_75461.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Manski, Charles F., 2006. "Interpreting the predictions of prediction markets," Economics Letters, Elsevier, vol. 91(3), pages 425-429, June.
    2. Berg, Joyce & Forsythe, Robert & Nelson, Forrest & Rietz, Thomas, 2008. "Results from a Dozen Years of Election Futures Markets Research," Handbook of Experimental Economics Results, in: Charles R. Plott & Vernon L. Smith (ed.), Handbook of Experimental Economics Results, edition 1, volume 1, chapter 80, pages 742-751, Elsevier.
    3. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    4. Johannes Abeler & Armin Falk & Lorenz Goette & David Huffman, 2011. "Reference Points and Effort Provision," American Economic Review, American Economic Association, vol. 101(2), pages 470-492, April.
    5. Bjorn Bartling & Ernst Fehr & Klaus M. Schmidt, 2012. "Screening, Competition, and Job Design: Economic Origins of Good Jobs," American Economic Review, American Economic Association, vol. 102(2), pages 834-864, April.
    6. Christian Franz Horn & Bjoern Sven Ivens & Michael Ohneberg & Alexander Brem, 2014. "Ideas Markets: Prediction Markets – A literature review 2014," Journal of Prediction Markets, University of Buckingham Press, vol. 8(2), pages 89-126.
    7. Shimon Kogan & Anthony M. Kwasnica & Roberto A. Weber, 2011. "Coordination in the Presence of Asset Markets," American Economic Review, American Economic Association, vol. 101(2), pages 927-947, April.
    8. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    9. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    10. Geoffroy de Clippel & Kfir Eliaz & Brian Knight, 2014. "On the Selection of Arbitrators," American Economic Review, American Economic Association, vol. 104(11), pages 3434-3458, November.
    11. Wolfers, Justin & Zitzewitz, Eric, 2006. "Interpreting Prediction Market Prices as Probabilities," IZA Discussion Papers 2092, Institute of Labor Economics (IZA).
    12. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    13. Attila Ambrus & Ben Greiner, 2012. "Imperfect Public Monitoring with Costly Punishment: An Experimental Study," American Economic Review, American Economic Association, vol. 102(7), pages 3317-3332, December.
    14. Drew Fudenberg & David G. Rand & Anna Dreber, 2012. "Slow to Anger and Fast to Forgive: Cooperation in an Uncertain World," American Economic Review, American Economic Association, vol. 102(2), pages 720-749, April.
    15. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    16. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    17. Gelman, Andrew & Stern, Hal, 2006. "The Difference Between," The American Statistician, American Statistical Association, vol. 60, pages 328-331, November.
    18. Ernst Fehr & Holger Herz & Tom Wilkening, 2013. "The Lure of Authority: Motivation and Incentive Effects of Power," American Economic Review, American Economic Association, vol. 103(4), pages 1325-1359, June.
    19. David Gill & Victoria Prowse, 2012. "A Structural Analysis of Disappointment Aversion in a Real Effort Competition," American Economic Review, American Economic Association, vol. 102(1), pages 469-503, February.
    20. Nisvan Erkal & Lata Gangadharan & Nikos Nikiforakis, 2011. "Relative Earnings and Giving in a Real-Effort Experiment," American Economic Review, American Economic Association, vol. 101(7), pages 3330-3348, December.
    21. Uwe Dulleck & Rudolf Kerschbamer & Matthias Sutter, 2011. "The Economics of Credence Goods: An Experiment on the Role of Liability, Verifiability, Reputation, and Competition," American Economic Review, American Economic Association, vol. 101(2), pages 526-555, April.
    22. Judd B. Kessler & Alvin E. Roth, 2012. "Organ Allocation Policy and the Decision to Donate," American Economic Review, American Economic Association, vol. 102(5), pages 2018-2047, August.
    23. Michael Kirchler & Jurgen Huber & Thomas Stockl, 2012. "Thar She Bursts: Reducing Confusion Reduces Bubbles," American Economic Review, American Economic Association, vol. 102(2), pages 865-883, April.
    24. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    25. Brian Merlob & Charles R. Plott & Yuanjun Zhang, 2012. "The CMS Auction: Experimental Studies of a Median-Bid Procurement Auction with Nonbinding Bids," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(2), pages 793-827.
    26. Johan Almenberg & Ken Kittlitz & Thomas Pfeiffer, 2009. "An Experiment on Prediction Markets in Science," PLOS ONE, Public Library of Science, vol. 4(12), pages 1-7, December.
    27. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    28. C. Glenn Begley & Lee M. Ellis, 2012. "Raise standards for preclinical cancer research," Nature, Nature, vol. 483(7391), pages 531-533, March.
    29. Catherine C. Eckel & Ragan Petrie, 2011. "Face Value," American Economic Review, American Economic Association, vol. 101(4), pages 1497-1513, June.
    30. John Ifcher & Homa Zarghamee, 2011. "Happiness and Time Preference: The Effect of Positive Affect in a Random-Assignment Experiment," American Economic Review, American Economic Association, vol. 101(7), pages 3109-3129, December.
    31. Leonard P Freedman & Iain M Cockburn & Timothy S Simcoe, 2015. "The Economics of Reproducibility in Preclinical Research," PLOS Biology, Public Library of Science, vol. 13(6), pages 1-9, June.
    32. Robin Hanson, 2007. "Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 3-15, February.
    33. Ryan W. Buell & Taly Reich & Michael I. Norton, 2014. ""Last-Place Aversion": Evidence and Redistributive Implications," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(1), pages 105-149.
    34. Steffen Huck & Andrew J. Seltzer & Brian Wallace, 2011. "Deferred Compensation in Multiperiod Labor Contracts: An Experimental Test of Lazear's Model," American Economic Review, American Economic Association, vol. 101(2), pages 819-843, April.
    35. Daniel Friedman & Ryan Oprea, 2012. "A Continuous Dilemma," American Economic Review, American Economic Association, vol. 102(1), pages 337-363, February.
    36. Keith M. Marzilli Ericson & Andreas Fuster, 2011. "Expectations as Endowments: Evidence on Reference-Dependent Preferences from Exchange and Valuation Experiments," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1879-1907.
    37. Georgios Tziralis & Ilias Tatsiopoulos, 2007. "Prediction Markets: An Extended Literature Review," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 75-91, February.
    38. Roy Chen & Yan Chen, 2011. "The Potential of Social Identity for Equilibrium Selection," American Economic Review, American Economic Association, vol. 101(6), pages 2562-2589, October.
    39. Roth, Alvin E, 1994. "Lets Keep the Con out of Experimental Econ.: A Methodological Note," Empirical Economics, Springer, vol. 19(2), pages 279-289.
    40. John Duffy & Daniela Puzzello, 2014. "Gift Exchange versus Monetary Exchange: Theory and Evidence," American Economic Review, American Economic Association, vol. 104(6), pages 1735-1776, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Snowberg, Erik & Wolfers, Justin & Zitzewitz, Eric, 2013. "Prediction Markets for Economic Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 657-687, Elsevier.
    2. Mikuláš Gangur & Miroslav Plevný, 2014. "Tools for Consumer Rights Protection in the Prediction of Electronic Virtual Market and Technological Changes," The AMFITEATRU ECONOMIC journal, Academy of Economic Studies - Bucharest, Romania, vol. 16(36), pages 578-578, May.
    3. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    4. Stefan Palan & Jürgen Huber & Larissa Senninger, 2020. "Aggregation mechanisms for crowd predictions," Experimental Economics, Springer;Economic Science Association, vol. 23(3), pages 788-814, September.
    5. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    6. Hou, Kewei & Xue, Chen & Zhang, Lu, 2017. "Replicating Anomalies," Working Paper Series 2017-10, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    7. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    8. Joyce E. Berg & John Geweke & Thomas A. Rietz, 2010. "Memoirs of an indifferent trader: Estimating forecast distributions from prediction markets," Quantitative Economics, Econometric Society, vol. 1(1), pages 163-186, July.
    9. Khan, Urmee & Lieli, Robert P., 2018. "Information flow between prediction markets, polls and media: Evidence from the 2008 presidential primaries," International Journal of Forecasting, Elsevier, vol. 34(4), pages 696-710.
    10. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    11. Coulomb, Renaud & Sangnier, Marc, 2014. "The impact of political majorities on firm value: Do electoral promises or friendship connections matter?," Journal of Public Economics, Elsevier, vol. 115(C), pages 158-170.
    12. Justin Wolfers & Eric Zitzewitz, 2006. "Prediction Markets in Theory and Practice," NBER Working Papers 12083, National Bureau of Economic Research, Inc.
    13. Qianwei Ying & Tahir Yousaf & Qurat ul Ain & Yasmeen Akhtar & Muhammad Shahid Rasheed, 2019. "Stock Investment and Excess Returns: A Critical Review in the Light of the Efficient Market Hypothesis," JRFM, MDPI, vol. 12(2), pages 1-22, June.
    14. Gill, David & Prowse, Victoria, 2019. "Measuring costly effort using the slider task," Journal of Behavioral and Experimental Finance, Elsevier, vol. 21(C), pages 1-9.
    15. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    16. Simon Gächter & Lingbo Huang & Martin Sefton, 2018. "Disappointment Aversion And Social Comparisons In A Real‐Effort Competition," Economic Inquiry, Western Economic Association International, vol. 56(3), pages 1512-1525, July.
    17. Ackfeld, Viola & Ockenfels, Axel, 2021. "Do people intervene to make others behave prosocially?," Games and Economic Behavior, Elsevier, vol. 128(C), pages 58-72.
    18. Dertwinkel-Kalt, Markus & Köster, Mats, 2017. "Salient compromises in the newsvendor game," Journal of Economic Behavior & Organization, Elsevier, vol. 141(C), pages 301-315.
    19. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    20. Gall, Thomas & Hu, Xiaocheng & Vlassopoulos, Michael, 2016. "Dynamic Incentive Effects of Team Formation: Experimental Evidence," IZA Discussion Papers 10393, Institute of Labor Economics (IZA).

    More about this item

    Keywords

    replication; economics; experiment;
    All these keywords.

    JEL classification:

    • A12 - General Economics and Teaching - - General Economics - - - Relation of Economics to Other Disciplines
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:75461. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.