IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0096431.html
   My bibliography  Save this article

A Cognitive Model for Aggregating People's Rankings

Author

Listed:
  • Michael D Lee
  • Mark Steyvers
  • Brent Miller

Abstract

We develop a cognitive modeling approach, motivated by classic theories of knowledge representation and judgment from psychology, for combining people's rankings of items. The model makes simple assumptions about how individual differences in knowledge lead to observed ranking data in behavioral tasks. We implement the cognitive model as a Bayesian graphical model, and use computational sampling to infer an aggregate ranking and measures of the individual expertise. Applications of the model to 23 data sets, dealing with general knowledge and prediction tasks, show that the model performs well in producing an aggregate ranking that is often close to the ground truth and, as in the “wisdom of the crowd” effect, usually performs better than most of individuals. We also present some evidence that the model outperforms the traditional statistical Borda count method, and that the model is able to infer people's relative expertise surprisingly well without knowing the ground truth. We discuss the advantages of the cognitive modeling approach to combining ranking data, and in wisdom of the crowd research generally, as well as highlighting a number of potential directions for future model development.

Suggested Citation

  • Michael D Lee & Mark Steyvers & Brent Miller, 2014. "A Cognitive Model for Aggregating People's Rankings," PLOS ONE, Public Library of Science, vol. 9(5), pages 1-9, May.
    • Handle: RePEc:plo:pone00:0096431
    DOI: 10.1371/journal.pone.0096431
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0096431
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0096431&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0096431?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. George Karabatsos & William Batchelder, 2003. "Markov chain estimation for test theory without an answer key," Psychometrika, Springer;The Psychometric Society, vol. 68(3), pages 373-389, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tang, Ming & Liao, Huchang, 2024. "Group efficiency and individual fairness tradeoff in making wise decisions," Omega, Elsevier, vol. 124(C).
    2. Joanna Jaroszewicz & Anna Majewska, 2021. "Group Spatial Preferences of Residential Locations—Simplified Method Based on Crowdsourced Spatial Data and MCDA," Sustainability, MDPI, vol. 13(9), pages 1-24, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Royce Anders & William Batchelder, 2015. "Cultural Consensus Theory for the Ordinal Data Case," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 151-181, March.
    2. Dora Matzke & Conor Dolan & William Batchelder & Eric-Jan Wagenmakers, 2015. "Bayesian Estimation of Multinomial Processing Tree Models with Heterogeneity in Participants and Items," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 205-235, March.
    3. Zita Oravecz & Royce Anders & William Batchelder, 2015. "Hierarchical Bayesian Modeling for Test Theory Without an Answer Key," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 341-364, June.
    4. Karl Klauer, 2010. "Hierarchical Multinomial Processing Tree Models: A Latent-Trait Approach," Psychometrika, Springer;The Psychometric Society, vol. 75(1), pages 70-98, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0096431. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.