IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0012642.html
   My bibliography  Save this article

The Calculus of Committee Composition

Author

Listed:
  • Eric Libby
  • Leon Glass

Abstract

Modern institutions face the recurring dilemma of designing accurate evaluation procedures in settings as diverse as academic selection committees, social policies, elections, and figure skating competitions. In particular, it is essential to determine both the number of evaluators and the method for combining their judgments. Previous work has focused on the latter issue, uncovering paradoxes that underscore the inherent difficulties. Yet the number of judges is an important consideration that is intimately connected with the methodology and the success of the evaluation. We address the question of the number of judges through a cost analysis that incorporates the accuracy of the evaluation method, the cost per judge, and the cost of an error in decision. We associate the optimal number of judges with the lowest cost and determine the optimal number of judges in several different scenarios. Through analytical and numerical studies, we show how the optimal number depends on the evaluation rule, the accuracy of the judges, the (cost per judge)/(cost per error) ratio. Paradoxically, we find that for a panel of judges of equal accuracy, the optimal panel size may be greater for judges with higher accuracy than for judges with lower accuracy. The development of any evaluation procedure requires knowledge about the accuracy of evaluation methods, the costs of judges, and the costs of errors. By determining the optimal number of judges, we highlight important connections between these quantities and uncover a paradox that we show to be a general feature of evaluation procedures. Ultimately, our work provides policy-makers with a simple and novel method to optimize evaluation procedures.

Suggested Citation

  • Eric Libby & Leon Glass, 2010. "The Calculus of Committee Composition," PLOS ONE, Public Library of Science, vol. 5(9), pages 1-8, September.
  • Handle: RePEc:plo:pone00:0012642
    DOI: 10.1371/journal.pone.0012642
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0012642
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0012642&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0012642?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Cozzens, Susan E., 1997. "The knowledge pool: Measurement challenges in evaluating fundamental research programs," Evaluation and Program Planning, Elsevier, vol. 20(1), pages 77-89, February.
    2. Bornmann, Lutz & Daniel, Hans-Dieter, 2009. "Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition," Journal of Informetrics, Elsevier, vol. 3(4), pages 348-352.
    3. Ordeshook,Peter C., 1986. "Game Theory and Political Theory," Cambridge Books, Cambridge University Press, number 9780521315937, October.
    4. Lutz Bornmann & Gerlind Wallon & Anna Ledin, 2008. "Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes," PLOS ONE, Public Library of Science, vol. 3(10), pages 1-11, October.
    5. Richard P. Larrick & Jack B. Soll, 2006. "Erratum--Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(2), pages 309-310, February.
    6. Saari,Donald G., 2001. "Decisions and Elections," Cambridge Books, Cambridge University Press, number 9780521808163, October.
    7. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    8. Drora Karotkin & Jacob Paroush, 2003. "Optimum committee size: Quality-versus-quantity dilemma," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 20(3), pages 429-441, June.
    9. Sawoong Kang, 2004. "Essays on Social Learning and Optimal Committee Size," Levine's Working Paper Archive 618897000000000892, David K. Levine.
    10. Willy Aspinall, 2010. "A route to more tractable expert advice," Nature, Nature, vol. 463(7279), pages 294-295, January.
    11. Saari,Donald G., 2001. "Decisions and Elections," Cambridge Books, Cambridge University Press, number 9780521004046, October.
    12. David Kaplan & Nicola Lacetera & Celia Kaplan, 2008. "Sample Size and Precision in NIH Peer Review," PLOS ONE, Public Library of Science, vol. 3(7), pages 1-3, July.
    13. Scott Feld & Bernard Grofman, 1984. "The accuracy of group majority decisions in groups with added members," Public Choice, Springer, vol. 42(3), pages 273-285, January.
    14. Richard P. Larrick & Jack B. Soll, 2006. "Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(1), pages 111-127, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gustaf Arrhenius & Klas Markstrom, 2024. "More, better or different? Trade-offs between group size and competence development in jury theorems," Papers 2404.09523, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Atanasov, Pavel & Witkowski, Jens & Ungar, Lyle & Mellers, Barbara & Tetlock, Philip, 2020. "Small steps to accuracy: Incremental belief updaters are better forecasters," Organizational Behavior and Human Decision Processes, Elsevier, vol. 160(C), pages 19-35.
    2. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    3. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    4. Johannes Müller-Trede & Shoham Choshen-Hillel & Meir Barneron & Ilan Yaniv, 2018. "The Wisdom of Crowds in Matters of Taste," Management Science, INFORMS, vol. 64(4), pages 1779-1803, April.
    5. Tomás Lejarraga & Johannes Müller-Trede, 2017. "When Experience Meets Description: How Dyads Integrate Experiential and Descriptive Information in Risky Decisions," Management Science, INFORMS, vol. 63(6), pages 1953-1971, June.
    6. Ville A. Satopää & Marat Salikhov & Philip E. Tetlock & Barbara Mellers, 2021. "Bias, Information, Noise: The BIN Model of Forecasting," Management Science, INFORMS, vol. 67(12), pages 7599-7618, December.
    7. Christian Ganser & Marc Keuschnigg, 2018. "Social Influence Strengthens Crowd Wisdom Under Voting," Advances in Complex Systems (ACS), World Scientific Publishing Co. Pte. Ltd., vol. 21(06n07), pages 1-23, September.
    8. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    9. Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.
    10. Michel Balinski & Rida Laraki, 2022. "Majority Judgment vs. Approval Voting," Operations Research, INFORMS, vol. 70(3), pages 1296-1316, May.
    11. Donald Saari, 2006. "Which is better: the Condorcet or Borda winner?," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 26(1), pages 107-129, January.
    12. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    13. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    14. Aki Lehtinen, 2007. "The Borda rule is also intended for dishonest men," Public Choice, Springer, vol. 133(1), pages 73-90, October.
    15. Gino, Francesca, 2008. "Do we listen to advice just because we paid for it? The impact of advice cost on its use," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 234-245, November.
    16. Herrade Igersheim, 2005. "Extending Xu's results to Arrow''s Impossibility Theorem," Economics Bulletin, AccessEcon, vol. 4(13), pages 1-6.
    17. repec:cup:judgdm:v:8:y:2013:i:2:p:91-105 is not listed on IDEAS
    18. Phanish Puranam, 2021. "Human–AI collaborative decision-making as an organization design problem," Journal of Organization Design, Springer;Organizational Design Community, vol. 10(2), pages 75-80, June.
    19. Alison Wood Brooks & Francesca Gino & Maurice E. Schweitzer, 2015. "Smart People Ask for (My) Advice: Seeking Advice Boosts Perceptions of Competence," Management Science, INFORMS, vol. 61(6), pages 1421-1435, June.
    20. Chiara Franzoni & Paula Stephan & Reinhilde Veugelers, 2022. "Funding Risky Research," Entrepreneurship and Innovation Policy and the Economy, University of Chicago Press, vol. 1(1), pages 103-133.
    21. Conal Duddy & Ashley Piggins & William Zwicker, 2016. "Aggregation of binary evaluations: a Borda-like approach," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 46(2), pages 301-333, February.
    22. Mirko Kremer & Enno Siemsen & Douglas J. Thomas, 2016. "The Sum and Its Parts: Judgmental Hierarchical Forecasting," Management Science, INFORMS, vol. 62(9), pages 2745-2764, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0012642. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.