IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v35y2010i3p280-306.html
   My bibliography  Save this article

Estimating Classification Accuracy for Complex Decision Rules Based on Multiple Scores

Author

Listed:
  • Karen M. Douglas
  • Robert J. Mislevy

Abstract

Important decisions about students are made by combining multiple measures using complex decision rules. Although methods for characterizing the accuracy of decisions based on a single measure have been suggested by numerous researchers, such methods are not useful for estimating the accuracy of decisions based on multiple measures. This study presents a simulation method useful for estimating classification accuracy for any measurement model and complex decision rule. The utility of the method is demonstrated by application to actual data from General Educational Development (GED) test takers. Findings in this study illustrate how multiple scores that are combined can have a large impact on the accuracy of the resulting decision. Furthermore, choice of measure of agreement is of central importance in describing decision accuracy.

Suggested Citation

  • Karen M. Douglas & Robert J. Mislevy, 2010. "Estimating Classification Accuracy for Complex Decision Rules Based on Multiple Scores," Journal of Educational and Behavioral Statistics, , vol. 35(3), pages 280-306, June.
    • Handle: RePEc:sae:jedbes:v:35:y:2010:i:3:p:280-306
    DOI: 10.3102/1076998609346969
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998609346969
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998609346969?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. C. Vale & Vincent Maurelli, 1983. "Simulating multivariate nonnormal distributions," Psychometrika, Springer;The Psychometric Society, vol. 48(3), pages 465-471, September.
    2. Todd Headrick & Shlomo Sawilowsky, 1999. "Simulating correlated multivariate nonnormal distributions: Extending the fleishman power method," Psychometrika, Springer;The Psychometric Society, vol. 64(2), pages 251-251, June.
    3. Todd Headrick & Shlomo Sawilowsky, 1999. "Simulating correlated multivariate nonnormal distributions: Extending the fleishman power method," Psychometrika, Springer;The Psychometric Society, vol. 64(1), pages 25-35, March.
    4. Frederic Lord, 1969. "Estimating true-score distributions in psychological testing (an empirical bayes estimation problem)," Psychometrika, Springer;The Psychometric Society, vol. 34(3), pages 259-299, September.
    5. Robert Mislevy, 1984. "Estimating latent distributions," Psychometrika, Springer;The Psychometric Society, vol. 49(3), pages 359-381, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wheadon, Chris, 2014. "Classification Accuracy and Consistency under Item Response Theory Models Using the Package classify," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 56(i10).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Max Auerswald & Morten Moshagen, 2015. "Generating Correlated, Non-normally Distributed Data Using a Non-linear Structural Model," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 920-937, December.
    2. Mohan D. Pant & Todd C. Headrick, 2017. "Simulating Uniform- and Triangular- Based Double Power Method Distributions," Journal of Statistical and Econometric Methods, SCIENPRESS Ltd, vol. 6(1), pages 1-1.
    3. Al-Subaihi, Ali A., 2004. "Simulating Correlated Multivariate Pseudorandom Numbers," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 9(i04).
    4. M Hashem Pesaran & Takashi Yamagata, 2012. "Testing CAPM with a Large Number of Assets," Discussion Papers 12/05, Department of Economics, University of York.
    5. Headrick, Todd C. & Mugdadi, Abdel, 2006. "On simulating multivariate non-normal distributions from the generalized lambda distribution," Computational Statistics & Data Analysis, Elsevier, vol. 50(11), pages 3343-3353, July.
    6. Ke-Hai Yuan & Peter Bentler, 2002. "On robusiness of the normal-theory based asymptotic distributions of three reliability coefficient estimates," Psychometrika, Springer;The Psychometric Society, vol. 67(2), pages 251-259, June.
    7. repec:jss:jstsof:09:i04 is not listed on IDEAS
    8. M Hashem Pesaran & Takashi Yamagata, 2024. "Testing for Alpha in Linear Factor Pricing Models with a Large Number of Securities," Journal of Financial Econometrics, Oxford University Press, vol. 22(2), pages 407-460.
    9. Pesaran, M. H. & Yamagata, T., 2012. "Testing CAPM with a Large Number of Assets (Updated 28th March 2012)," Cambridge Working Papers in Economics 1210, Faculty of Economics, University of Cambridge.
    10. Headrick, Todd C., 2002. "Fast fifth-order polynomial transforms for generating univariate and multivariate nonnormal distributions," Computational Statistics & Data Analysis, Elsevier, vol. 40(4), pages 685-711, October.
    11. Gary van Vuuren & Riaan de Jongh, 2017. "A comparison of risk aggregation estimates using copulas and Fleishman distributions," Applied Economics, Taylor & Francis Journals, vol. 49(17), pages 1715-1731, April.
    12. Yen Lee & David Kaplan, 2018. "Generating Multivariate Ordinal Data via Entropy Principles," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 156-181, March.
    13. Beasley, T. Mark & Zumbo, Bruno D., 2003. "Comparison of aligned Friedman rank and parametric methods for testing interactions in split-plot designs," Computational Statistics & Data Analysis, Elsevier, vol. 42(4), pages 569-593, April.
    14. Ping Chen & Chun Wang, 2021. "Using EM Algorithm for Finite Mixtures and Reformed Supplemented EM for MIRT Calibration," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 299-326, March.
    15. repec:jss:jstsof:19:i03 is not listed on IDEAS
    16. Wendy Yen, 1987. "A comparison of the efficiency and accuracy of BILOG and LOGIST," Psychometrika, Springer;The Psychometric Society, vol. 52(2), pages 275-291, June.
    17. Mishra, SK, 2004. "On generating correlated random variables with a given valid or invalid Correlation matrix," MPRA Paper 1782, University Library of Munich, Germany.
    18. Jeffrey J. Hoogland & Anne Boomsma, 1998. "Robustness Studies in Covariance Structure Modeling," Sociological Methods & Research, , vol. 26(3), pages 329-367, February.
    19. Melissa Gladstone & Gillian Lancaster & Gareth McCray & Vanessa Cavallera & Claudia R. L. Alves & Limbika Maliwichi & Muneera A. Rasheed & Tarun Dua & Magdalena Janus & Patricia Kariger, 2021. "Validation of the Infant and Young Child Development (IYCD) Indicators in Three Countries: Brazil, Malawi and Pakistan," IJERPH, MDPI, vol. 18(11), pages 1-19, June.
    20. Emanuela Raffinetti & Pier Alda Ferrari, 2021. "A dependence measure flow tree through Monte Carlo simulations," Quality & Quantity: International Journal of Methodology, Springer, vol. 55(2), pages 467-496, April.
    21. Andreas Oranje & Andrew Kolstad, 2019. "Research on Psychometric Modeling, Analysis, and Reporting of the National Assessment of Educational Progress," Journal of Educational and Behavioral Statistics, , vol. 44(6), pages 648-670, December.
    22. Hiroyuki Kasahara & Katsumi Shimotsu, 2014. "Non-parametric identification and estimation of the number of components in multivariate mixtures," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 97-111, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:35:y:2010:i:3:p:280-306. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.