IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/22-16.html
   My bibliography  Save this paper

Do Negative Replications Affect Citations?

Author

Abstract

This study examines the effect of negative replications on the citation rates of replicated studies. We study a set of 204 replicated studies in economics and compare their citation performance with an initial sample of 112,000 potential controls taken from Scopus. From this initial pool, we match each replicated study with multiple controls based on having comparable citation histories. Our main finding is that there is no evidence that studies that receive negative replications suffer a penalty in the form of fewer citations. We also find that replicated studies receive somewhat more citations than their matched control studies, though here the causal interpretation is more suspect.

Suggested Citation

  • Tom Coupé & W. Robert Reed, 2022. "Do Negative Replications Affect Citations?," Working Papers in Economics 22/16, University of Canterbury, Department of Economics and Finance.
    • Handle: RePEc:cbt:econwp:22/16
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/2216.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Anton Nakov, 2010. "Jackknife instrumental variables estimation: replication and extension of angrist, imbens and krueger (1999)," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1063-1066.
    2. Pierre Azoulay & Toby Stuart & Yanbo Wang, 2014. "Matthew: Effect or Fable?," Management Science, INFORMS, vol. 60(1), pages 92-109, January.
    3. Rees, Daniel I. & Sabia, Joseph J., 2010. "Body weight and smoking initiation: Evidence from Add Health," Journal of Health Economics, Elsevier, vol. 29(5), pages 774-777, September.
    4. Cawley, John & Markowitz, Sara & Tauras, John, 2004. "Lighting up and slimming down: the effects of body weight and cigarette prices on adolescent smoking initiation," Journal of Health Economics, Elsevier, vol. 23(2), pages 293-311, March.
    5. Paul J. Devereux & Robert A. Hart, 2010. "Forced to be Rich? Returns to Compulsory Schooling in Britain," Economic Journal, Royal Economic Society, vol. 120(549), pages 1345-1364, December.
    6. Hoenig J. M. & Heisey D. M., 2001. "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis," The American Statistician, American Statistical Association, vol. 55, pages 19-24, February.
    7. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    8. Philip Oreopoulos, 2006. "Estimating Average and Local Average Treatment Effects of Education when Compulsory Schooling Laws Really Matter," American Economic Review, American Economic Association, vol. 96(1), pages 152-175, March.
    9. Badi H Baltagi, 2010. "Narrow Replication of Serlenga and Shin (2007) gravity models of intra-EU trade: application of the CCEP-HT estimation in heterogeneous panels with unobserved common time-specific factors," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(3), pages 505-506.
    10. Angrist, J D & Imbens, G W & Krueger, A B, 1999. "Jackknife Instrumental Variables Estimation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 14(1), pages 57-67, Jan.-Feb..
    11. DeSimone, Jeff, 2007. "Fraternity membership and binge drinking," Journal of Health Economics, Elsevier, vol. 26(5), pages 950-967, September.
    12. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    13. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
    14. Jodi Schneider & Di Ye & Alison M. Hill & Ashley S. Whitehorn, 2020. "Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2877-2913, December.
    15. Judit Bar-Ilan & Gali Halevi, 2018. "Temporal characteristics of retracted articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1771-1783, September.
    16. Yongcheol Shin & Laura Serlenga, 2007. "Gravity models of intra-EU trade: application of the CCEP-HT estimation in heterogeneous panels with unobserved common time-specific factors," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 22(2), pages 361-381.
    17. Amar Hamoudi, 2010. "Exploring the Causal Machinery behind Sex Ratios at Birth: Does Hepatitis B Play a Role?," Economic Development and Cultural Change, University of Chicago Press, vol. 59(1), pages 1-21, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tom Coupé & W. Robert Reed, 2023. "Do Replications Make a Difference?," Working Papers in Economics 23/10, University of Canterbury, Department of Economics and Finance.
    2. L. Pieroni & L. Salmasi, 2015. "Does Cigarette Smoking Affect Body Weight? Causal Estimates from the Clean Indoor Air Law Discontinuity," Economica, London School of Economics and Political Science, vol. 82(328), pages 671-704, October.
    3. Coupé, Tom & Reed, W. Robert & Zimmermann, Christian, 2023. "Getting seen: Results from an online experiment to draw more attention to replications," Research Policy, Elsevier, vol. 52(8).
    4. Clémentine Garrouste & Mathilde Godard, 2016. "The lasting health impact of leaving school in a bad economy : Britons in the 1970s recession," Post-Print hal-01408637, HAL.
    5. Jonneke Bolhaar & Sander Gerritsen & Sonny Kuijpers & Karen van der Wiel, 2019. "Experimenting with dropout prevention policies," CPB Discussion Paper 400, CPB Netherlands Bureau for Economic Policy Analysis.
    6. Frederique Bordignon, 2020. "Self-correction of science: a comparative study of negative citations and post-publication peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1225-1239, August.
    7. Stephen Machin & Olivier Marie & Sunčica Vujić, 2011. "The Crime Reducing Effect of Education," Economic Journal, Royal Economic Society, vol. 121(552), pages 463-484, May.
    8. Daniel A. Kamhöfer & Hendrik Schmitz, 2013. "Analyzing Zero Returns to Education in Germany – Heterogeneous Eff ects and Skill Formation," Ruhr Economic Papers 0446, Rheinisch-Westfälisches Institut für Wirtschaftsforschung, Ruhr-Universität Bochum, Universität Dortmund, Universität Duisburg-Essen.
    9. Seeun Jung, 2015. "Does education affect risk aversion? Evidence from the British education reform," Applied Economics, Taylor & Francis Journals, vol. 47(28), pages 2924-2938, June.
    10. Setti Rais Ali & Paul Dourgnon & Lise Rochaix, 2018. "Social Capital or Education: What Matters Most to Cut Time to Diagnosis?," Working Papers halshs-01703170, HAL.
    11. Tony Beatton & Michael P. Kidd & Matteo Sandi, 2020. "School indiscipline and crime," CEP Discussion Papers dp1727, Centre for Economic Performance, LSE.
    12. Liwiński, Jacek, 2018. "The Impact of Compulsory Education on Employment and Earnings in a Transition Economy," GLO Discussion Paper Series 193, Global Labor Organization (GLO).
    13. Paul J. Devereux & Robert A. Hart, 2010. "Forced to be Rich? Returns to Compulsory Schooling in Britain," Economic Journal, Royal Economic Society, vol. 120(549), pages 1345-1364, December.
    14. Johnston, David W. & Lordan, Grace & Shields, Michael A. & Suziedelyte, Agne, 2015. "Education and health knowledge: Evidence from UK compulsory schooling reform," Social Science & Medicine, Elsevier, vol. 127(C), pages 92-100.
    15. Robert A. HartBy & Mirko Moro & J. Elizabeth Roberts, 2017. "Who gained from the introduction of free universal secondary education in England and Wales?," Oxford Economic Papers, Oxford University Press, vol. 69(3), pages 707-733.
    16. Salim Moussa, 2022. "The propagation of error: retracted articles in marketing and their citations," Italian Journal of Marketing, Springer, vol. 2022(1), pages 11-36, March.
    17. Nattavudh Powdthavee & Warn N. Lekfuangfu & Mark Wooden, 2013. "The Marginal Income Effect of Education on Happiness: Estimating the Direct and Indirect Effects of Compulsory Schooling on Well-Being in Australia," Melbourne Institute Working Paper Series wp2013n16, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    18. Stanislav Avdeev, 2020. "Zero Returns To Higher Education: Evidence From A Natural Experiment," HSE Working papers WP BRP 236/EC/2020, National Research University Higher School of Economics.
    19. Martin Fischer & Martin Karlsson & Therese Nilsson & Nina Schwarz, 2020. "The Long-Term Effects of Long Terms – Compulsory Schooling Reforms in Sweden," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 2776-2823.
    20. Fiorini, Mario & Katrien Stevens, 2014. "Assessing the Monotonicity Assumption in IV and fuzzy RD designs," Working Papers 2014-13, University of Sydney, School of Economics.

    More about this item

    Keywords

    Replications; Citations; Matching; Meta-science; Self-correcting science;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:22/16. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.