IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/22-02.html
   My bibliography  Save this paper

Do Negative Replications Affect Citations?

Author

Abstract

This study examines the effect of negative replications on the citation rates of replicated studies. We study a set of 204 replicated studies in economics and compare their citation performance with an initial sample of 112,000 potential controls taken from Scopus. From this initial pool, we match each replicated study with multiple controls based on having comparable citation histories. We have two main findings. First, studies that are replicated receive somewhat more citations than their matched control studies. Second, there is no evidence that studies that receive negative replications suffer a penalty in the form of fewer citations.

Suggested Citation

  • Tom Coupé & W. Robert Reed, 2022. "Do Negative Replications Affect Citations?," Working Papers in Economics 22/02, University of Canterbury, Department of Economics and Finance.
    • Handle: RePEc:cbt:econwp:22/02
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/2202.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Anton Nakov, 2010. "Jackknife instrumental variables estimation: replication and extension of angrist, imbens and krueger (1999)," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1063-1066.
    2. Philip Oreopoulos, 2006. "Estimating Average and Local Average Treatment Effects of Education when Compulsory Schooling Laws Really Matter," American Economic Review, American Economic Association, vol. 96(1), pages 152-175, March.
    3. DeSimone, Jeff, 2007. "Fraternity membership and binge drinking," Journal of Health Economics, Elsevier, vol. 26(5), pages 950-967, September.
    4. Badi H Baltagi, 2010. "Narrow Replication of Serlenga and Shin (2007) gravity models of intra-EU trade: application of the CCEP-HT estimation in heterogeneous panels with unobserved common time-specific factors," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(3), pages 505-506.
    5. Pierre Azoulay & Toby Stuart & Yanbo Wang, 2014. "Matthew: Effect or Fable?," Management Science, INFORMS, vol. 60(1), pages 92-109, January.
    6. Paul J. Devereux & Robert A. Hart, 2010. "Forced to be Rich? Returns to Compulsory Schooling in Britain," Economic Journal, Royal Economic Society, vol. 120(549), pages 1345-1364, December.
    7. Rees, Daniel I. & Sabia, Joseph J., 2010. "Body weight and smoking initiation: Evidence from Add Health," Journal of Health Economics, Elsevier, vol. 29(5), pages 774-777, September.
    8. Cawley, John & Markowitz, Sara & Tauras, John, 2004. "Lighting up and slimming down: the effects of body weight and cigarette prices on adolescent smoking initiation," Journal of Health Economics, Elsevier, vol. 23(2), pages 293-311, March.
    9. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
    10. Jodi Schneider & Di Ye & Alison M. Hill & Ashley S. Whitehorn, 2020. "Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2877-2913, December.
    11. Angrist, J D & Imbens, G W & Krueger, A B, 1999. "Jackknife Instrumental Variables Estimation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 14(1), pages 57-67, Jan.-Feb..
    12. Hoenig J. M. & Heisey D. M., 2001. "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis," The American Statistician, American Statistical Association, vol. 55, pages 19-24, February.
    13. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    14. Judit Bar-Ilan & Gali Halevi, 2018. "Temporal characteristics of retracted articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1771-1783, September.
    15. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    16. Yongcheol Shin & Laura Serlenga, 2007. "Gravity models of intra-EU trade: application of the CCEP-HT estimation in heterogeneous panels with unobserved common time-specific factors," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 22(2), pages 361-381.
    17. Amar Hamoudi, 2010. "Exploring the Causal Machinery behind Sex Ratios at Birth: Does Hepatitis B Play a Role?," Economic Development and Cultural Change, University of Chicago Press, vol. 59(1), pages 1-21, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tom Coupé & W. Robert Reed, 2023. "Do Replications Make a Difference?," Working Papers in Economics 23/10, University of Canterbury, Department of Economics and Finance.
    2. L. Pieroni & L. Salmasi, 2015. "Does Cigarette Smoking Affect Body Weight? Causal Estimates from the Clean Indoor Air Law Discontinuity," Economica, London School of Economics and Political Science, vol. 82(328), pages 671-704, October.
    3. Tom Coupé & W. Robert Reed & Christian Zimmerman, 2021. "Paving the Road for Replications: Experimental Results from an Online Research Repository," Working Papers in Economics 21/09, University of Canterbury, Department of Economics and Finance.
    4. Clémentine Garrouste & Mathilde Godard, 2016. "The lasting health impact of leaving school in a bad economy : Britons in the 1970s recession," Post-Print hal-01408637, HAL.
    5. Liwiński, Jacek, 2018. "The Impact of Compulsory Schooling on Earnings. Evidence from the 1999 Education Reform in Poland," GLO Discussion Paper Series 253, Global Labor Organization (GLO).
    6. Stephen Machin & Olivier Marie & Sunčica Vujić, 2011. "The Crime Reducing Effect of Education," Economic Journal, Royal Economic Society, vol. 121(552), pages 463-484, May.
    7. Jonneke Bolhaar & Sander Gerritsen & Sonny Kuijpers & Karen van der Wiel, 2019. "Experimenting with dropout prevention policies," CPB Discussion Paper 400, CPB Netherlands Bureau for Economic Policy Analysis.
    8. Robert A. HartBy & Mirko Moro & J. Elizabeth Roberts, 2017. "Who gained from the introduction of free universal secondary education in England and Wales?," Oxford Economic Papers, Oxford University Press, vol. 69(3), pages 707-733.
    9. Gradstein, Mark & Justman, Moshe, 2009. "Understanding Compulsory Schooling Legislation: A Formal Model and Implications for Empirical Analysis," IZA Discussion Papers 4420, Institute of Labor Economics (IZA).
    10. Seeun Jung, 2015. "Does education affect risk aversion? Evidence from the British education reform," Applied Economics, Taylor & Francis Journals, vol. 47(28), pages 2924-2938, June.
    11. Martin Fischer & Martin Karlsson & Therese Nilsson & Nina Schwarz, 2020. "The Long-Term Effects of Long Terms – Compulsory Schooling Reforms in Sweden," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 2776-2823.
    12. Frederique Bordignon, 2020. "Self-correction of science: a comparative study of negative citations and post-publication peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1225-1239, August.
    13. Silles, Mary A., 2011. "The intergenerational effects of parental schooling on the cognitive and non-cognitive development of children," Economics of Education Review, Elsevier, vol. 30(2), pages 258-268, April.
    14. Daniel A. Kamhöfer & Hendrik Schmitz, 2013. "Analyzing Zero Returns to Education in Germany – Heterogeneous Eff ects and Skill Formation," Ruhr Economic Papers 0446, Rheinisch-Westfälisches Institut für Wirtschaftsforschung, Ruhr-Universität Bochum, Universität Dortmund, Universität Duisburg-Essen.
    15. Setti Rais Ali & Paul Dourgnon & Lise Rochaix, 2018. "Social Capital or Education: What Matters Most to Cut Time to Diagnosis?," Working Papers halshs-01703170, HAL.
    16. Daniel A. Kamhöfer & Hendrik Schmitz, 2013. "Analyzing Zero Returns to Education in Germany: Heterogeneous Effects and Skill Formation," SOEPpapers on Multidisciplinary Panel Data Research 598, DIW Berlin, The German Socio-Economic Panel (SOEP).
    17. John Cawley & Stephanie von Hinke Kessler Scholder, 2013. "The Demand for Cigarettes as Derived from the Demand for Weight Control," NBER Working Papers 18805, National Bureau of Economic Research, Inc.
    18. Weili Ding, 2020. "Laboratory experiments can pre-design to address power and selection issues," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 125-138, December.
    19. Tony Beatton & Michael P. Kidd & Matteo Sandi, 2020. "School indiscipline and crime," CEP Discussion Papers dp1727, Centre for Economic Performance, LSE.
    20. Nattavudh Powdthavee, 2010. "Does Education Reduce the Risk of Hypertension? Estimating the Biomarker Effect of Compulsory Schooling in England," Journal of Human Capital, University of Chicago Press, vol. 4(2), pages 173-202.

    More about this item

    Keywords

    Replications; Citations; Matching; Meta-science; Self-correcting science;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:22/02. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.