IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v35y2010i2p194-214.html
   My bibliography  Save this article

A Bayesian Approach to Ranking and Rater Evaluation

Author

Listed:
  • Jing Cao
  • S. Lynne Stokes
  • Song Zhang

    (University of Texas Southwestern Medical Center)

Abstract

We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater’s score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater’s performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.

Suggested Citation

  • Jing Cao & S. Lynne Stokes & Song Zhang, 2010. "A Bayesian Approach to Ranking and Rater Evaluation," Journal of Educational and Behavioral Statistics, , vol. 35(2), pages 194-214, April.
    • Handle: RePEc:sae:jedbes:v:35:y:2010:i:2:p:194-214
    DOI: 10.3102/1076998609353116
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998609353116
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998609353116?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. David Thissen & Lynne Steinberg, 1986. "A taxonomy of item response models," Psychometrika, Springer;The Psychometric Society, vol. 51(4), pages 567-577, December.
    2. David Andrich, 1978. "A rating formulation for ordered response categories," Psychometrika, Springer;The Psychometric Society, vol. 43(4), pages 561-573, December.
    3. Wei Shen & Thomas A. Louis, 1998. "Triple‐goal estimates in two‐stage hierarchical models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 60(2), pages 455-471.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hua-Hua Chang, 1996. "The asymptotic posterior normality of the latent trait for polytomous IRT models," Psychometrika, Springer;The Psychometric Society, vol. 61(3), pages 445-463, September.
    2. Dylan Molenaar, 2015. "Heteroscedastic Latent Trait Models for Dichotomous Data," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 625-644, September.
    3. Gerhard Tutz, 2022. "Item Response Thresholds Models: A General Class of Models for Varying Types of Items," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1238-1269, December.
    4. Singh, Jagdip, 2004. "Tackling measurement problems with Item Response Theory: Principles, characteristics, and assessment, with an illustrative example," Journal of Business Research, Elsevier, vol. 57(2), pages 184-208, February.
    5. David Magis, 2015. "A Note on the Equivalence Between Observed and Expected Information Functions With Polytomous IRT Models," Journal of Educational and Behavioral Statistics, , vol. 40(1), pages 96-105, February.
    6. David Magis, 2015. "A Note on Weighted Likelihood and Jeffreys Modal Estimation of Proficiency Levels in Polytomous Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 200-204, March.
    7. Timothy R. Johnson & Daniel M. Bolt, 2010. "On the Use of Factor-Analytic Multinomial Logit Item Response Models to Account for Individual Differences in Response Style," Journal of Educational and Behavioral Statistics, , vol. 35(1), pages 92-114, February.
    8. Bas Hemker & L. Andries van der Ark & Klaas Sijtsma, 2001. "On measurement properties of continuation ratio models," Psychometrika, Springer;The Psychometric Society, vol. 66(4), pages 487-506, December.
    9. van der Ark, L.A., 1999. "A reference card for the relationships between IRT models for ordered polytomous items and some relevant properties," WORC Paper 99.10.02, Tilburg University, Work and Organization Research Centre.
    10. Hua-Hua Chang & John Mazzeo, 1994. "The unique correspondence of the item response function and item category response functions in polytomously scored item response models," Psychometrika, Springer;The Psychometric Society, vol. 59(3), pages 391-404, September.
    11. David Magis & Norman Verhelst, 2017. "On the Finiteness of the Weighted Likelihood Estimator of Ability," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 637-647, September.
    12. Eun-Young Park & Soojung Chae, 2020. "Rasch Analysis of the Korean Parenting Stress Index Short Form (K-PSI-SF) in Mothers of Children with Cerebral Palsy," IJERPH, MDPI, vol. 17(19), pages 1-11, September.
    13. P. A. Ferrari & S. Salini, 2008. "Measuring Service Quality: The Opinion of Europeans about Utilities," Working Papers 2008.36, Fondazione Eni Enrico Mattei.
    14. Chang, Hsin-Li & Yang, Cheng-Hua, 2008. "Explore airlines’ brand niches through measuring passengers’ repurchase motivation—an application of Rasch measurement," Journal of Air Transport Management, Elsevier, vol. 14(3), pages 105-112.
    15. Ivana Bassi & Matteo Carzedda & Enrico Gori & Luca Iseppi, 2022. "Rasch analysis of consumer attitudes towards the mountain product label," Agricultural and Food Economics, Springer;Italian Society of Agricultural Economics (SIDEA), vol. 10(1), pages 1-25, December.
    16. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.
    17. Antonio Caronni & Marina Ramella & Pietro Arcuri & Claudia Salatino & Lucia Pigini & Maurizio Saruggia & Chiara Folini & Stefano Scarano & Rosa Maria Converti, 2023. "The Rasch Analysis Shows Poor Construct Validity and Low Reliability of the Quebec User Evaluation of Satisfaction with Assistive Technology 2.0 (QUEST 2.0) Questionnaire," IJERPH, MDPI, vol. 20(2), pages 1-19, January.
    18. Wanke, Peter Fernandes & Chiappetta Jabbour, Charbel José & Moreira Antunes, Jorge Junio & Lopes de Sousa Jabbour, Ana Beatriz & Roubaud, David & Sobreiro, Vinicius Amorim & Santibanez Gonzalez‬, Erne, 2021. "An original information entropy-based quantitative evaluation model for low-carbon operations in an emerging market," International Journal of Production Economics, Elsevier, vol. 234(C).
    19. Bas Hemker & Klaas Sijtsma & Ivo Molenaar & Brian Junker, 1996. "Polytomous IRT models and monotone likelihood ratio of the total score," Psychometrika, Springer;The Psychometric Society, vol. 61(4), pages 679-693, December.
    20. Curt Hagquist & Raili Välimaa & Nina Simonsen & Sakari Suominen, 2017. "Differential Item Functioning in Trend Analyses of Adolescent Mental Health – Illustrative Examples Using HBSC-Data from Finland," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 10(3), pages 673-691, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:35:y:2010:i:2:p:194-214. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.