IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp3665.html
   My bibliography  Save this paper

Can We Test for Bias in Scientific Peer-Review?

Author

Listed:
  • Oswald, Andrew J.

    (University of Warwick)

Abstract

Science rests upon the reliability of peer review. This paper suggests a way to test for bias. It is able to avoid the fallacy – one seen in the popular press and the research literature – that to measure discrimination it is sufficient to study averages within two populations. The paper’s contribution is primarily methodological, but I apply it, as an illustration, to data from the field of economics. No scientific bias or favoritism is found (although the Journal of Political Economy discriminates against its own Chicago authors). The test’s methodology is applicable in most scholarly disciplines.

Suggested Citation

  • Oswald, Andrew J., 2008. "Can We Test for Bias in Scientific Peer-Review?," IZA Discussion Papers 3665, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp3665
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp3665.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Tom Coupé & Victor Ginsburgh & Abdul Noury, 2010. "Are leading papers of better quality? Evidence from a natural experiment," Oxford Economic Papers, Oxford University Press, vol. 62(1), pages 1-11, January.
    2. Shamena Anwar & Hanming Fang, 2006. "An Alternative Test of Racial Prejudice in Motor Vehicle Searches: Theory and Evidence," American Economic Review, American Economic Association, vol. 96(1), pages 127-151, March.
    3. Glenn Ellison, 2011. "Is Peer Review In Decline?," Economic Inquiry, Western Economic Association International, vol. 49(3), pages 635-657, July.
    4. Eric W. K. Tsang & Bruno S. Frey, 2006. "The as-is journal review process: Let authors own their ideas," CREMA Working Paper Series 2006-09, Center for Research in Economics, Management and the Arts (CREMA).
    5. Timothy Clark & Mike Wright, 2007. "Reviewing Journal Rankings and Revisiting Peer Reviews: Editorial Perspectives," Journal of Management Studies, Wiley Blackwell, vol. 44(4), pages 612-621, June.
    6. van Praag, Mirjam C. & van Praag, Bernard M. S., 2007. "The Benefits of Being Economics Professor A (and not Z)," IZA Discussion Papers 2673, Institute of Labor Economics (IZA).
    7. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    8. M. Luwel, 1999. "Is the science citation index US-biased?," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(3), pages 549-562, November.
    9. Frey, Bruno S, 2003. "Publishing as Prostitution?--Choosing between One's Own Ideas and Academic Success," Public Choice, Springer, vol. 116(1-2), pages 205-223, July.
    10. David N. Laband & Robert D. Tollison & Gokhan Karahan, 2002. "Quality Control in Economics," Kyklos, Wiley Blackwell, vol. 55(3), pages 315-334, August.
    11. repec:eee:labchp:v:1:y:1986:i:c:p:693-785 is not listed on IDEAS
    12. Amanda H Goodall, 2005. "Should Research Universities be Led by Top Researchers? Part 1: Are they?," HEW 0506003, University Library of Munich, Germany.
    13. J. Peter Neary & James A. Mirrlees & Jean Tirole, 2003. "Evaluating Economics Research in Europe: An Introduction," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1239-1249, December.
    14. Cecilia Rouse & Claudia Goldin, 2000. "Orchestrating Impartiality: The Impact of "Blind" Auditions on Female Musicians," American Economic Review, American Economic Association, vol. 90(4), pages 715-741, September.
    15. Liran Einav & Leeat Yariv, 2006. "What's in a Surname? The Effects of Surname Initials on Academic Success," Journal of Economic Perspectives, American Economic Association, vol. 20(1), pages 175-187, Winter.
    16. Daniel S. Hamermesh & Peter Schmidt, 2003. "The Determinants of Econometric Society Fellows Elections," Econometrica, Econometric Society, vol. 71(1), pages 399-407, January.
    17. C. Mirjam Van Praag & Bernard M.S. Van Praag, 2008. "The Benefits of Being Economics Professor A (rather than Z)," Economica, London School of Economics and Political Science, vol. 75(300), pages 782-796, November.
    18. Daniel S. Hamermesh, 2002. "International Labor Economics," Journal of Labor Economics, University of Chicago Press, vol. 20(4), pages 709-732, October.
    19. Charles Oppenheim, 2007. "Using the h‐index to rank influential British researchers in information science and librarianship," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(2), pages 297-301, January.
    20. JS Armstrong, 2004. "Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation," General Economics and Teaching 0412027, University Library of Munich, Germany.
    21. Stephen Wu, 2007. "Recent publishing trends at the AER, JPE and QJE," Applied Economics Letters, Taylor & Francis Journals, vol. 14(1), pages 59-63.
    22. Justin Wolfers, 2006. "Diagnosing Discrimination: Stock Returns and CEO Gender," Journal of the European Economic Association, MIT Press, vol. 4(2-3), pages 531-541, 04-05.
    23. William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
    24. repec:bla:kyklos:v:55:y:2002:i:3:p:315-34 is not listed on IDEAS
    25. Scott Smart & Joel Waldfogel, 1996. "A Citation-Based Test for Discrimination at Economics and Finance Journals," NBER Working Papers 5460, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. A poor test for scientific bias in peer review
      by Economic Logician in Economic Logic on 2009-03-25 19:30:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ana Rute Cardoso & Paulo Guimarães & Klaus F. Zimmermann, 2010. "Comparing the early research performance of PhD graduates in labor economics in Europe and the USA," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 621-637, September.
    2. Bradford Demarest & Guo Freeman & Cassidy R. Sugimoto, 2014. "The reviewer in the mirror: examining gendered and ethnicized notions of reciprocity in peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 717-735, October.
    3. Marco Seeber & Alberto Bacchelli, 2017. "Does single blind peer review hinder newcomers?," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 567-585, October.
    4. Brogaard, Jonathan & Engelberg, Joseph & Parsons, Christopher A., 2014. "Networks and productivity: Causal evidence from editor rotations," Journal of Financial Economics, Elsevier, vol. 111(1), pages 251-270.
    5. Vicente Safón, 2013. "What do global university rankings really measure? The search for the X factor and the X entity," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 223-244, November.
    6. Lawson, Nicholas, 2023. "What citation tests really tell us about bias in academic publishing," European Economic Review, Elsevier, vol. 158(C).
    7. Philip R P Coelho & James E McClure & Peter J Reilly, 2014. "An Investigation of Editorial Favoritism in the AER," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 40(2), pages 274-281, March.
    8. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    9. Vicente Safón, 2019. "Inter-ranking reputational effects: an analysis of the Academic Ranking of World Universities (ARWU) and the Times Higher Education World University Rankings (THE) reputational relationship," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 897-915, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Oswald, Andrew J., 2009. "World-Leading Research and its Measurement," The Warwick Economics Research Paper Series (TWERPS) 887, University of Warwick, Department of Economics.
    2. Andrew J. Oswald, 2010. "A suggested method for the measurement of world-leading research (illustrated with data on economics)," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 99-113, July.
    3. Oswald, Andrew J., 2015. "The Objective Measurement of World-Leading Research," IZA Discussion Papers 8829, Institute of Labor Economics (IZA).
    4. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    5. Jens Prüfer & David Zetland, 2010. "An auction market for journal articles," Public Choice, Springer, vol. 145(3), pages 379-403, December.
    6. Margit Osterloh & Bruno S. Frey, 2009. "Research Governance in Academia: Are there Alternatives to Academic Rankings?," CREMA Working Paper Series 2009-17, Center for Research in Economics, Management and the Arts (CREMA).
    7. Ali, Showkat & Carden, Giles & Culling, Benjamin & Hunter, Rosalind & Oswald, Andrew J & Owen, Nicola & Ralsmark, Hilda & Snodgrass, Natalie, 2007. "Elite Scientists and the Global Brain Drain," The Warwick Economics Research Paper Series (TWERPS) 825, University of Warwick, Department of Economics.
    8. Tom Coupé & Victor Ginsburgh & Abdul Noury, 2010. "Are leading papers of better quality? Evidence from a natural experiment," Oxford Economic Papers, Oxford University Press, vol. 62(1), pages 1-11, January.
    9. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    10. Berlemann, Michael & Haucap, Justus, 2012. "Which factors drive the decision to boycott and opt out of research rankings? A note," DICE Discussion Papers 72, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    11. Jan Ours & Frederic Vermeulen, 2007. "Ranking Dutch Economists," De Economist, Springer, vol. 155(4), pages 469-487, December.
    12. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    13. Michael Berlemann & Justus Haucap, 2012. "Which Factors Drive the Decision to Boycott and Opt Out of Research Rankings?," CESifo Working Paper Series 3997, CESifo.
    14. Marcella Corsi & Carlo D'Ippoliti & Federico Lucidi, 2010. "Pluralism at Risk? Heterodox Economic Approaches and the Evaluation of Economic Research in Italy," American Journal of Economics and Sociology, Wiley Blackwell, vol. 69(5), pages 1495-1529, November.
    15. Ma, Chao & Li, Yiwei & Guo, Feng & Si, Kao, 2019. "The citation trap: Papers published at year-end receive systematically fewer citations," Journal of Economic Behavior & Organization, Elsevier, vol. 166(C), pages 667-687.
    16. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.
    17. Justus Haucap & Johannes Muck, 2015. "What drives the relevance and reputation of economics journals? An update from a survey among economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 849-877, June.
    18. Dolado, Juan J. & Felgueroso, Florentino & Almunia, Miguel, 2005. "Do Men and Women-Economists Choose the Same Research Fields? Evidence from Top-50 Departments," IZA Discussion Papers 1859, Institute of Labor Economics (IZA).
    19. Matthias Weber, 2016. "The Effects of Listing Authors in Alphabetical Order: A survey of the Empirical Evidence," Bank of Lithuania Occasional Paper Series 12, Bank of Lithuania.
    20. John Tressler & David L. Anderson, 2012. "Citations as a Measure of the Research Outputs of New Zealand's Economics Departments: The Problem of 'Long and Variable Lags'," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 19(1), pages 17-40.

    More about this item

    Keywords

    discrimination; citations; science; peer-review system;
    All these keywords.

    JEL classification:

    • H8 - Public Economics - - Miscellaneous Issues

    NEP fields

    This paper has been announced in the following NEP Reports:

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. Economic Logic blog

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp3665. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.