IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v80y2015i3p645-664.html
   My bibliography  Save this article

Standard Error of Ability Estimates and the Classification Accuracy and Consistency of Binary Decisions

Author

Listed:
  • Ying Cheng
  • Cheng Liu
  • John Behrens

Abstract

While estimation bias is a primary concern in psychological and educational measurement, the standard error is of equal importance in linking key aspects of the assessment structure, especially when the assessment goal concerns the classification of individuals into categories (e.g., master/non-mastery). In this paper, we show analytically how standard error of ability estimates affects expected classification accuracy and consistency when the decision is binary. When standard error decreases, the conditional classification accuracy and consistency increase. Given an examinee population and a cut score, smaller standard error over the entire latent trait continuum guarantees higher overall expected classification accuracy and consistency. We were also able to show the interrelationship between standard error, the expected classification consistency, and reliability. Utilizing the relationship between standard error and expected classification accuracy and consistency, we derive the upper bounds of the overall expected classification accuracy and consistency of a fixed-length computerized adaptive test. The lower bound of the expected classification accuracy and consistency is also derived given a number of stopping rules of variable-length computerized adaptive testing. Implications of these analytical results on operational tests are discussed. Copyright The Psychometric Society 2015

Suggested Citation

  • Ying Cheng & Cheng Liu & John Behrens, 2015. "Standard Error of Ability Estimates and the Classification Accuracy and Consistency of Binary Decisions," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 645-664, September.
    • Handle: RePEc:spr:psycho:v:80:y:2015:i:3:p:645-664
    DOI: 10.1007/s11336-014-9407-z
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s11336-014-9407-z
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11336-014-9407-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Carol Woods & David Thissen, 2006. "Item Response Theory with Estimation of the Latent Population Distribution Using Spline-Based Densities," Psychometrika, Springer;The Psychometric Society, vol. 71(2), pages 281-301, June.
    2. Thomas Warm, 1989. "Weighted likelihood estimation of ability in item response theory," Psychometrika, Springer;The Psychometric Society, vol. 54(3), pages 427-450, September.
    3. Lee Cronbach, 1951. "Coefficient alpha and the internal structure of tests," Psychometrika, Springer;The Psychometric Society, vol. 16(3), pages 297-334, September.
    4. Ying Cheng & Ke-Hai Yuan, 2010. "The Impact of Fallible Item Parameter Estimates on Latent Trait Recovery," Psychometrika, Springer;The Psychometric Society, vol. 75(2), pages 280-291, June.
    5. Frederic Lord, 1983. "Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability," Psychometrika, Springer;The Psychometric Society, vol. 48(2), pages 233-245, June.
    6. Hua-Hua Chang & William Stout, 1993. "The asymptotic posterior normality of the latent trait in an IRT model," Psychometrika, Springer;The Psychometric Society, vol. 58(1), pages 37-52, March.
    7. Carol M. Woods & David Thissen, 2006. "Item Response Theory with Estimation of the Latent Population Distribution Using Spline-Based Densities," Psychometrika, Springer;The Psychometric Society, vol. 71(2), pages 281-301, June.
    8. Ogasawara, Haruhiko, 2013. "Asymptotic properties of the Bayes and pseudo Bayes estimators of ability in item response theory," Journal of Multivariate Analysis, Elsevier, vol. 114(C), pages 359-377.
    9. Nageswari Rajaratnam & Lee Cronbach & Goldine Gleser, 1965. "Generalizability of stratified-parallel tests," Psychometrika, Springer;The Psychometric Society, vol. 30(1), pages 39-56, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jeanne A. Teresi & Katja Ocepek-Welikson & John A. Toner & Marjorie Kleinman & Mildred Ramirez & Joseph P. Eimicke & Barry J. Gurland & Albert Siu, 2017. "Methodological Issues in Measuring Subjective Well-Being and Quality-of-Life: Applications to Assessment of Affect in Older, Chronically and Cognitively Impaired, Ethnically Diverse Groups Using the F," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 12(2), pages 251-288, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sandip Sinharay, 2015. "The Asymptotic Distribution of Ability Estimates," Journal of Educational and Behavioral Statistics, , vol. 40(5), pages 511-528, October.
    2. Yang Liu & Ji Seung Yang, 2018. "Bootstrap-Calibrated Interval Estimates for Latent Variable Scores in Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 333-354, June.
    3. Ogasawara, Haruhiko, 2013. "Asymptotic cumulants of ability estimators using fallible item parameters," Journal of Multivariate Analysis, Elsevier, vol. 119(C), pages 144-162.
    4. Xiang Liu & James Yang & Hui Soo Chae & Gary Natriello, 2020. "Power Divergence Family of Statistics for Person Parameters in IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 502-525, June.
    5. Yang Liu & Jan Hannig & Abhishek Pal Majumder, 2019. "Second-Order Probability Matching Priors for the Person Parameter in Unidimensional IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 701-718, September.
    6. Martin Biehler & Heinz Holling & Philipp Doebler, 2015. "Saddlepoint Approximations of the Distribution of the Person Parameter in the Two Parameter Logistic Model," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 665-688, September.
    7. J. R. Lockwood & Katherine E. Castellano & Benjamin R. Shear, 2018. "Flexible Bayesian Models for Inferences From Coarsened, Group-Level Achievement Data," Journal of Educational and Behavioral Statistics, , vol. 43(6), pages 663-692, December.
    8. Oberrauch, Luis & Kaiser, Tim, 2020. "Economic competence in early secondary school: Evidence from a large-scale assessment in Germany," International Review of Economics Education, Elsevier, vol. 35(C).
    9. David Magis, 2016. "Efficient Standard Error Formulas of Ability Estimators with Dichotomous Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 184-200, March.
    10. Chun Wang, 2015. "On Latent Trait Estimation in Multidimensional Compensatory Item Response Models," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 428-449, June.
    11. Shaobo Jin & Fan Yang-Wallentin, 2017. "Asymptotic Robustness Study of the Polychoric Correlation Estimation," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 67-85, March.
    12. Yang Liu, 2020. "A Riemannian Optimization Algorithm for Joint Maximum Likelihood Estimation of High-Dimensional Exploratory Item Factor Analysis," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 439-468, June.
    13. John E. Hunter & Gerald M. Gillmore, 1974. "A Memory Search Model of Reliability," Sociological Methods & Research, , vol. 2(3), pages 281-311, February.
    14. Seonghoon Kim, 2012. "A Note on the Reliability Coefficients for Item Response Model-Based Ability Estimates," Psychometrika, Springer;The Psychometric Society, vol. 77(1), pages 153-162, January.
    15. Ping Chen & Chun Wang, 2016. "A New Online Calibration Method for Multidimensional Computerized Adaptive Testing," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 674-701, September.
    16. Li Cai, 2010. "A Two-Tier Full-Information Item Factor Analysis Model with Applications," Psychometrika, Springer;The Psychometric Society, vol. 75(4), pages 581-612, December.
    17. Piero Veronese & Eugenio Melilli, 2021. "Confidence Distribution for the Ability Parameter of the Rasch Model," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 131-166, March.
    18. Salim Moussa, 2016. "A Comment on the Estimation of the Reliability of Multidimensional Marketing Constructs: A Store Personality Scale Application," Global Business Review, International Management Institute, vol. 17(5), pages 1125-1144, October.
    19. Klaas Sijtsma & Ivo Molenaar, 1987. "Reliability of test scores in nonparametric item response theory," Psychometrika, Springer;The Psychometric Society, vol. 52(1), pages 79-97, March.
    20. Georg Gittler & Gerhard Fischer, 2011. "IRT-Based Measurement of Short-Term Changes of Ability, With an Application to Assessing the “Mozart Effectâ€," Journal of Educational and Behavioral Statistics, , vol. 36(1), pages 33-75, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:80:y:2015:i:3:p:645-664. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.