IDEAS home Printed from https://ideas.repec.org/p/urv/wpaper/2072-267264.html
   My bibliography  Save this paper

Judgement and Ranking: Living with Hidden Bias

Author

Listed:
  • Osório, António (António Miguel)

Abstract

The complexity and subjectivity of the judgement task conceals the existence of biases that undermines the quality of the process. This paper presents a weighted aggregation function that attempts to reduce the influence of biased judgements on the final score. We also discuss a set of desirable properties. The proposed weighted aggregation function is able to correct the "nationalism bias" found by Emerson et al. (2009) in the 2000 Olympic Games diving competition and suggest the possibility of a "reputation bias". Our results can be applied to judgement sports and other activities that require the aggregation of several personal evaluations. Keywords: Weighted aggregation function; Judgement by Grading; Nationalism Bias; Reputation Bias; Bias Correction; Olympic Games. JEL classification: D72, D81, D03.

Suggested Citation

  • Osório, António (António Miguel), 2016. "Judgement and Ranking: Living with Hidden Bias," Working Papers 2072/267264, Universitat Rovira i Virgili, Department of Economics.
    • Handle: RePEc:urv:wpaper:2072/267264
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/2072/267264
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Nolan Miller & Paul Resnick & Richard Zeckhauser, 2005. "Eliciting Informative Feedback: The Peer-Prediction Method," Management Science, INFORMS, vol. 51(9), pages 1359-1373, September.
    2. Jungmin Lee, 2008. "Outlier Aversion in Subjective Evaluation," Journal of Sports Economics, , vol. 9(2), pages 141-159, April.
    3. Victor A. Ginsburgh & Jan C. van Ours, 2003. "Expert Opinion and Compensation: Evidence from a Musical Competition," American Economic Review, American Economic Association, vol. 93(1), pages 289-296, March.
    4. Michel Grabisch & Jean-Luc Marichal & Radko Mesiar & Endre Pap, 2011. "Aggregation functions: Means," Post-Print hal-00539028, HAL.
    5. repec:dau:papers:123456789/15122 is not listed on IDEAS
    6. Pavel Yu. Chebotarev & Elena Shamis, 1998. "Characterizations of scoring methodsfor preference aggregation," Annals of Operations Research, Springer, vol. 80(0), pages 299-332, January.
    7. Satterthwaite, Mark Allen, 1975. "Strategy-proofness and Arrow's conditions: Existence and correspondence theorems for voting procedures and social welfare functions," Journal of Economic Theory, Elsevier, vol. 10(2), pages 187-217, April.
    8. John T. Buchanan & Erez J. Henig & Mordecai I. Henig, 1998. "Objectivity and subjectivity in thedecision making process," Annals of Operations Research, Springer, vol. 80(0), pages 333-345, January.
    9. Eric Zitzewitz, 2006. "Nationalism in Winter Sports Judging and Its Lessons for Organizational Decision Making," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 15(1), pages 67-99, March.
    10. Brams, Steven J. & Fishburn, Peter C., 1978. "Approval Voting," American Political Science Review, Cambridge University Press, vol. 72(3), pages 831-847, September.
    11. Michel Balinski & Rida Laraki, 2011. "Election by Majority Judgment: Experimental Evidence," Studies in Public Choice, in: Bernard Dolez & Bernard Grofman & Annie Laurent (ed.), In Situ and Laboratory Experiments on Electoral Law Reform, chapter 0, pages 13-54, Springer.
    12. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    13. Smith, John H, 1973. "Aggregation of Preferences with Variable Electorate," Econometrica, Econometric Society, vol. 41(6), pages 1027-1041, November.
    14. Emerson, John W. & Seltzer, Miki & Lin, David, 2009. "Assessing Judging Bias: An Example From the 2000 Olympic Games," The American Statistician, American Statistical Association, vol. 63(2), pages 124-131.
    15. Kenneth J. Arrow, 1950. "A Difficulty in the Concept of Social Welfare," Journal of Political Economy, University of Chicago Press, vol. 58(4), pages 328-328.
    16. Michel Grabisch & Jean-Luc Marichal & Radko Mesiar & Endre Pap, 2011. "Aggregation functions: construction methods, conjunctive, disjunctive and mixed classes," Post-Print hal-00539032, HAL.
    17. Fishburn, Peter C., 1978. "Axioms for approval voting: Direct proof," Journal of Economic Theory, Elsevier, vol. 19(1), pages 180-185, October.
    18. Gibbard, Allan, 1973. "Manipulation of Voting Schemes: A General Result," Econometrica, Econometric Society, vol. 41(4), pages 587-601, July.
    19. Hodgson, Robert T., 2008. "An Examination of Judge Reliability at a major U.S. Wine Competition," Journal of Wine Economics, Cambridge University Press, vol. 3(2), pages 105-113, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. António Osório, 2017. "Judgement and ranking: living with hidden bias," Annals of Operations Research, Springer, vol. 253(1), pages 501-518, June.
    2. António Osório, 2020. "Performance Evaluation: Subjectivity, Bias and Judgment Style in Sport," Group Decision and Negotiation, Springer, vol. 29(4), pages 655-678, August.
    3. Wolitzky, Alexander, 2009. "Fully sincere voting," Games and Economic Behavior, Elsevier, vol. 67(2), pages 720-735, November.
    4. Erdamar, Bora & Sanver, M. Remzi & Sato, Shin, 2017. "Evaluationwise strategy-proofness," Games and Economic Behavior, Elsevier, vol. 106(C), pages 227-238.
    5. Peter Kurrild-Klitgaard, 2014. "Empirical social choice: an introduction," Public Choice, Springer, vol. 158(3), pages 297-310, March.
    6. Roger B. Myerson, 1991. "Proportional Representation," Discussion Papers 928, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
    7. Csató, László, 2017. "European qualifiers to the 2018 FIFA World Cup can be manipulated," MPRA Paper 82652, University Library of Munich, Germany.
    8. Green-Armytage, James, 2011. "Strategic voting and nomination," MPRA Paper 32200, University Library of Munich, Germany.
    9. Carlos Alós-Ferrer & Ðura-Georg Granić, 2012. "Two field experiments on Approval Voting in Germany," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 39(1), pages 171-205, June.
    10. Dellis, Arnaud, 2009. "Would letting people vote for multiple candidates yield policy moderation?," Journal of Economic Theory, Elsevier, vol. 144(2), pages 772-801, March.
    11. Michel Balinski & Rida Laraki, 2014. "Judge: Don't Vote !," Operations Research, INFORMS, vol. 62(3), pages 483-511, June.
    12. L'aszl'o Csat'o, 2019. "Journal ranking should depend on the level of aggregation," Papers 1904.06300, arXiv.org, revised Sep 2019.
    13. Peter Fishburn & Steven Brams, 1984. "Manipulability of voting by sincere truncation of preferences," Public Choice, Springer, vol. 44(3), pages 397-410, January.
    14. Felix Brandt & Patrick Lederer & René Romen, 2024. "Relaxed notions of Condorcet-consistency and efficiency for strategyproof social decision schemes," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 63(1), pages 19-55, August.
    15. Bock, Hans-Hermann & Day, William H. E. & McMorris, F. R., 1998. "Consensus rules for committee elections," Mathematical Social Sciences, Elsevier, vol. 35(3), pages 219-232, May.
    16. Samet, Dov & Schmeidler, David, 2003. "Between liberalism and democracy," Journal of Economic Theory, Elsevier, vol. 110(2), pages 213-233, June.
    17. Donaldson, Jason & Piacentino, Giorgia & Malenko, Nadya, 2017. "Deadlock on the Board," CEPR Discussion Papers 12503, C.E.P.R. Discussion Papers.
    18. Freixas, Josep & Parker, Cameron, 2015. "Manipulation in games with multiple levels of output," Journal of Mathematical Economics, Elsevier, vol. 61(C), pages 144-151.
    19. Burak Can & Peter Csoka & Emre Ergin, 2017. "How to choose a non-manipulable delegation?," CERS-IE WORKING PAPERS 1713, Institute of Economics, Centre for Economic and Regional Studies.
    20. Stensholt, Eivind, 2019. "MMP-elections and the assembly size," Discussion Papers 2019/15, Norwegian School of Economics, Department of Business and Management Science.

    More about this item

    Keywords

    Decisió; Presa de; Microeconomia -- Aspectes psicològics; 33 - Economia;
    All these keywords.

    JEL classification:

    • D72 - Microeconomics - - Analysis of Collective Decision-Making - - - Political Processes: Rent-seeking, Lobbying, Elections, Legislatures, and Voting Behavior
    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty
    • D03 - Microeconomics - - General - - - Behavioral Microeconomics: Underlying Principles

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:urv:wpaper:2072/267264. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ariadna Casals (email available below). General contact details of provider: https://edirc.repec.org/data/deurves.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.