IDEAS home Printed from https://ideas.repec.org/p/zbw/rwirep/1005.html
   My bibliography  Save this paper

Is economics self-correcting? Replications in the American Economic Review

Author

Listed:
  • Ankel-Peters, Jörg
  • Fiala, Nathan
  • Neubauer, Florian

Abstract

Replication and constructive controversy are essential for scientific progress. This paper reviews the impact of all replications published as comments in the American Economic Review between 2010 and 2020. We investigate the citation rates of comments and whether a comment affects its original paper's citation rates. We find that most comments are barely cited, and they have no impact on the original papers' subsequent citations. This finding holds for original papers for which the comment diagnoses a substantive problem. We conclude from these citation patterns that replications do not update the economics literature. In an online opinion survey, we elicited viewpoints of both comment authors and original authors and find that in most cases, there is no consensus regarding the replication's success and to what extent the original paper's contribution sustains. This resonates with the conventional wisdom that robustness and replicability are hard to define in economics.

Suggested Citation

  • Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Is economics self-correcting? Replications in the American Economic Review," Ruhr Economic Papers 1005, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    • Handle: RePEc:zbw:rwirep:1005
    DOI: 10.4419/96973171
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/269235/1/1838894071.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.4419/96973171?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    2. Daron Acemoglu & Simon Johnson & James A. Robinson & Pierre Yared, 2008. "Income and Democracy," American Economic Review, American Economic Association, vol. 98(3), pages 808-842, June.
    3. Jason Long & Joseph Ferrie, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850," American Economic Review, American Economic Association, vol. 103(4), pages 1109-1137, June.
    4. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    5. Lucas C. Coffman & Muriel Niederle & Alistair J. Wilson, 2017. "A Proposal to Organize and Promote Replications," American Economic Review, American Economic Association, vol. 107(5), pages 41-45, May.
    6. Derek Lemoine & Ivan Rudik, 2017. "Steering the Climate System: Using Inertia to Lower the Cost of Policy," American Economic Review, American Economic Association, vol. 107(10), pages 2947-2957, October.
    7. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    8. H. M. Collins, 1991. "The Meaning of Replication and the Science of Economics," History of Political Economy, Duke University Press, vol. 23(1), pages 123-142, Spring.
    9. Jason Long & Joseph Ferrie, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850: Reply," American Economic Review, American Economic Association, vol. 103(5), pages 2041-2049, August.
    10. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
    11. Michael Hout & Avery M. Guest, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850: Comment," American Economic Review, American Economic Association, vol. 103(5), pages 2021-2040, August.
    12. Whittington, Leslie A & Alm, James & Peters, H Elizabeth, 1990. "Fertility and the Personal Exemption: Implicit Pronatalist Policy in the United States," American Economic Review, American Economic Association, vol. 80(3), pages 545-556, June.
    13. Richard Crump & Gopi Shah Goda & Kevin J. Mumford, 2011. "Fertility and the Personal Exemption: Comment," American Economic Review, American Economic Association, vol. 101(4), pages 1616-1628, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-Term Effects of the Targeting the Ultra-Poor Program - A Reproducibility and Replicability Assessment of Banerjee et al. (2021)," I4R Discussion Paper Series 142, The Institute for Replication (I4R).
    2. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    3. Tom Coupé & Thomas Logchies & W. Robert Reed, 2024. "Do Replications Really Receive Fewer Citations?," Working Papers in Economics 24/15, University of Canterbury, Department of Economics and Finance.
    4. Ankel-Peters, Jörg & Schmidt, Christoph M., 2023. "Rural electrification, the credibility revolution, and the limits of evidence-based policy," Ruhr Economic Papers 1051, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. Thibaut Arpinon & Marianne Lefebvre, 2024. "Registered Reports and Associated Benefits for Agricultural Economics," Post-Print hal-04635986, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    2. Natkhov, T. & Polishchuk, L., 2017. "Political Economy of Institutions and Development: The Importance of Being Inclusive. Reflection on "Why Nations Fail" by D. Acemoglu and J. Robinson. Part I. Institutions and Economic Devel," Journal of the New Economic Association, New Economic Association, vol. 34(2), pages 12-38.
    3. Casey, Gregory & Klemp, Marc, 2021. "Historical instruments and contemporary endogenous regressors," Journal of Development Economics, Elsevier, vol. 149(C).
    4. Ran Abramitzky, 2015. "Economics and the Modern Economic Historian," NBER Working Papers 21636, National Bureau of Economic Research, Inc.
    5. Casey, Gregory & Klemp, Marc, 2016. "Instrumental Variables in the Long Run," MPRA Paper 68696, University Library of Munich, Germany.
    6. Ang, James B., 2013. "Institutions and the long-run impact of early development," Journal of Development Economics, Elsevier, vol. 105(C), pages 1-18.
    7. Markus Jäntti & Stephen P. Jenkins, 2013. "Income Mobility," SOEPpapers on Multidisciplinary Panel Data Research 607, DIW Berlin, The German Socio-Economic Panel (SOEP).
    8. David S. Johnson & Catherine Massey & Amy O’Hara, 2015. "The Opportunities and Challenges of Using Administrative Data Linkages to Evaluate Mobility," The ANNALS of the American Academy of Political and Social Science, , vol. 657(1), pages 247-264, January.
    9. Peter Lloyd & Cassey Lee, 2018. "A Review Of The Recent Literature On The Institutional Economics Analysis Of The Long†Run Performance Of Nations," Journal of Economic Surveys, Wiley Blackwell, vol. 32(1), pages 1-22, February.
    10. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    11. Dribe, Martin & Eriksson, Björn & Scalone, Francesco, 2019. "Migration, marriage and social mobility: Women in Sweden 1880–1900," Explorations in Economic History, Elsevier, vol. 71(C), pages 93-111.
    12. Zachary Ward, 2023. "Intergenerational Mobility in American History: Accounting for Race and Measurement Error," American Economic Review, American Economic Association, vol. 113(12), pages 3213-3248, December.
    13. Fedderke, Johannes & Klitgaard, Robert, 2013. "How Much Do Rights Matter?," World Development, Elsevier, vol. 51(C), pages 187-206.
    14. Saima Nawaz & M. Idrees Khawaja, 2016. "Fiscal Policy, Institutions And Growth: New Insights," The Singapore Economic Review (SER), World Scientific Publishing Co. Pte. Ltd., vol. 64(05), pages 1251-1278, December.
    15. Sen, Kunal & Pritchett, Lant & Kar, Sabyasachi & Raihan, Selim, 2016. "Democracy Versus Dictatorship? The Political Determinants of Growth Episodes," Working Paper Series rwp17-009, Harvard University, John F. Kennedy School of Government.
    16. Nina Boberg-Fazlić & Paul Sharp, 2018. "North and south: long-run social mobility in England and attitudes toward welfare," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 12(2), pages 251-276, May.
    17. Peter Lloyd & Cassey Lee, 2016. "A Review of the Recent Literature on the Institutional Economics Analysis of the Long-Run Performance of Nations," Department of Economics - Working Papers Series 2019, The University of Melbourne.
    18. Ran Abramitzky & Roy Mill & Santiago Pérez, 2020. "Linking individuals across historical sources: A fully automated approach," Historical Methods: A Journal of Quantitative and Interdisciplinary History, Taylor & Francis Journals, vol. 53(2), pages 94-111, April.
    19. Otto Brøns-Petersen & Søren Havn Gjedsted, 2021. "Climate change and institutional change: what is the relative importance for economic performance?," Environmental Economics and Policy Studies, Springer;Society for Environmental Economics and Policy Studies - SEEPS, vol. 23(2), pages 333-360, April.
    20. Adrian Adermon & Mikael Lindahl & Daniel Waldenström, 2018. "Intergenerational Wealth Mobility and the Role of Inheritance: Evidence from Multiple Generations," Economic Journal, Royal Economic Society, vol. 128(612), pages 482-513, July.

    More about this item

    Keywords

    Replication; citations; meta-science;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:rwirep:1005. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/rwiesde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.