IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2211.08494.html
   My bibliography  Save this paper

Who Reviews The Reviewers? A Multi-Level Jury Problem

Author

Listed:
  • Ben Abramowitz
  • Omer Lev
  • Nicholas Mattei

Abstract

We consider the problem of determining a binary ground truth using advice from a group of independent reviewers (experts) who express their guess about a ground truth correctly with some independent probability (competence). In this setting, when all reviewers are competent (competence greater than one-half), the Condorcet Jury Theorem tells us that adding more reviewers increases the overall accuracy, and if all competences are known, then there exists an optimal weighting of the reviewers. However, in practical settings, reviewers may be noisy or incompetent, i.e., competence below half, and the number of experts may be small, so the asymptotic Condorcet Jury Theorem is not practically relevant. In such cases we explore appointing one or more chairs (judges) who determine the weight of each reviewer for aggregation, creating multiple levels. However, these chairs may be unable to correctly identify the competence of the reviewers they oversee, and therefore unable to compute the optimal weighting. We give conditions when a set of chairs is able to weight the reviewers optimally, and depending on the competence distribution of the agents, give results about when it is better to have more chairs or more reviewers. Through numerical simulations we show that in some cases it is better to have more chairs, but in many cases it is better to have more reviewers.

Suggested Citation

  • Ben Abramowitz & Omer Lev & Nicholas Mattei, 2022. "Who Reviews The Reviewers? A Multi-Level Jury Problem," Papers 2211.08494, arXiv.org, revised Dec 2023.
    • Handle: RePEc:arx:papers:2211.08494
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2211.08494
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ruth Ben-Yashar & Jacob Paroush, 2000. "A nonasymptotic Condorcet jury theorem," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 17(2), pages 189-199.
    2. Eyal Baharad & Jacob Goldberger & Moshe Koppel & Shmuel Nitzan, 2012. "Beyond Condorcet: optimal aggregation rules using voting records," Theory and Decision, Springer, vol. 72(1), pages 113-130, January.
    3. Roy Baharad & Shmuel Nitzan & Erel Segal-Halevi, 2022. "One person, one weight: when is weighted voting democratic?," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 59(2), pages 467-493, August.
    4. Malik Magdon-Ismail & Lirong Xia, 2018. "A Mathematical Model for Optimal Decisions in a Representative Democracy," Papers 1807.06157, arXiv.org.
    5. Scott Feld & Bernard Grofman, 1984. "The accuracy of group majority decisions in groups with added members," Public Choice, Springer, vol. 42(3), pages 273-285, January.
    6. Lloyd Shapley & Bernard Grofman, 1984. "Optimizing group judgmental accuracy in the presence of interdependencies," Public Choice, Springer, vol. 43(3), pages 329-343, January.
    7. Arnold Cédrick SOH VOUTSA, 2020. "Approval Voting & Majority Judgment in Weighted Representative Democracy," THEMA Working Papers 2020-15, THEMA (THéorie Economique, Modélisation et Applications), Université de Cergy-Pontoise.
    8. Daniel Berend & Jacob Paroush, 1998. "When is Condorcet's Jury Theorem valid?," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 15(4), pages 481-488.
    9. Nitzan, Shmuel & Paroush, Jacob, 1982. "Optimal Decision Rules in Uncertain Dichotomous Choice Situations," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 23(2), pages 289-297, June.
    10. Pivato, Marcus & Soh, Arnold, 2020. "Weighted representative democracy," Journal of Mathematical Economics, Elsevier, vol. 88(C), pages 52-63.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben Abramowitz & Nicholas Mattei, 2022. "Towards Group Learning: Distributed Weighting of Experts," Papers 2206.02566, arXiv.org.
    2. Ruth Ben-Yashar & Shmuel Nitzan, 2017. "Are two better than one? A note," Public Choice, Springer, vol. 171(3), pages 323-329, June.
    3. Ruth Ben-Yashar & Shmuel Nitzan, 2017. "Is diversity in capabilities desirable when adding decision makers?," Theory and Decision, Springer, vol. 82(3), pages 395-402, March.
    4. Baharad, Eyal & Ben-Yashar, Ruth & Patal, Tal, 2020. "On the merit of non-specialization in the context of majority voting," Journal of Mathematical Economics, Elsevier, vol. 87(C), pages 128-133.
    5. Eyal Baharad & Ruth Ben-Yashar & Shmuel Nitzan, 2020. "Variable Competence and Collective Performance: Unanimity Versus Simple Majority Rule," Group Decision and Negotiation, Springer, vol. 29(1), pages 157-167, February.
    6. Ruth Ben-Yashar & Shmuel Nitzan, 2014. "On the significance of the prior of a correct decision in committees," Theory and Decision, Springer, vol. 76(3), pages 317-327, March.
    7. Sapir, Luba, 2005. "Generalized means of jurors' competencies and marginal changes of jury's size," Mathematical Social Sciences, Elsevier, vol. 50(1), pages 83-101, July.
    8. Ruth Ben-Yashar & Shmuel Nitzan, 2017. "Is diversity in capabilities desirable when adding decision makers?," Theory and Decision, Springer, vol. 82(3), pages 395-402, March.
    9. Ruth Ben-Yashar, 2014. "The generalized homogeneity assumption and the Condorcet jury theorem," Theory and Decision, Springer, vol. 77(2), pages 237-241, August.
    10. Ruth Ben-Yashar, 2023. "An application of simple majority rule to a group with an even number of voters," Theory and Decision, Springer, vol. 94(1), pages 83-95, January.
    11. Eyal Baharad & Jacob Goldberger & Moshe Koppel & Shmuel Nitzan, 2012. "Beyond Condorcet: optimal aggregation rules using voting records," Theory and Decision, Springer, vol. 72(1), pages 113-130, January.
    12. Bezalel Peleg & Shmuel Zamir, 2012. "Extending the Condorcet Jury Theorem to a general dependent jury," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 39(1), pages 91-125, June.
    13. Shmuel Nitzan & Tomoya Tajika, 2022. "Inequality of decision-makers’ power and marginal contribution," Theory and Decision, Springer, vol. 92(2), pages 275-292, March.
    14. Ruth Ben-Yashar & Miriam Krausz & Shmuel Nitzan, 2018. "The effect of democratic decision-making on investment in reputation," Public Choice, Springer, vol. 177(1), pages 155-164, October.
    15. Malik Magdon-Ismail & Lirong Xia, 2018. "A Mathematical Model for Optimal Decisions in a Representative Democracy," Papers 1807.06157, arXiv.org.
    16. Ruth Ben-Yashar & Mor Zahavi, 2011. "The Condorcet jury theorem and extension of the franchise with rationally ignorant voters," Public Choice, Springer, vol. 148(3), pages 435-443, September.
    17. Romaniega Sancho, Álvaro, 2022. "On the probability of the Condorcet Jury Theorem or the Miracle of Aggregation," Mathematical Social Sciences, Elsevier, vol. 119(C), pages 41-55.
    18. Ruth Ben-Yashar & Miriam Krausz & Shmuel Nitzan, 2018. "The effect of democratic decision-making on investment in reputation," Public Choice, Springer, vol. 177(1), pages 155-164, October.
    19. Takuya Sekiguchi, 2023. "Voting Records as Assessors of Premises Behind Collective Decisions," Group Decision and Negotiation, Springer, vol. 32(2), pages 257-275, April.
    20. Dhillon, Amrita & Kotsialou, Grammateia & Xefteris, Dimitris, 2021. "Information Aggregation with Delegation of Votes," SocArXiv ubk7p, Center for Open Science.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2211.08494. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.