IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v35y2010i3p280-306.html
   My bibliography  Save this article

Estimating Classification Accuracy for Complex Decision Rules Based on Multiple Scores

Author

Listed:
  • Karen M. Douglas
  • Robert J. Mislevy

Abstract

Important decisions about students are made by combining multiple measures using complex decision rules. Although methods for characterizing the accuracy of decisions based on a single measure have been suggested by numerous researchers, such methods are not useful for estimating the accuracy of decisions based on multiple measures. This study presents a simulation method useful for estimating classification accuracy for any measurement model and complex decision rule. The utility of the method is demonstrated by application to actual data from General Educational Development (GED) test takers. Findings in this study illustrate how multiple scores that are combined can have a large impact on the accuracy of the resulting decision. Furthermore, choice of measure of agreement is of central importance in describing decision accuracy.

Suggested Citation

  • Karen M. Douglas & Robert J. Mislevy, 2010. "Estimating Classification Accuracy for Complex Decision Rules Based on Multiple Scores," Journal of Educational and Behavioral Statistics, , vol. 35(3), pages 280-306, June.
    • Handle: RePEc:sae:jedbes:v:35:y:2010:i:3:p:280-306
    DOI: 10.3102/1076998609346969
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998609346969
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998609346969?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. C. Vale & Vincent Maurelli, 1983. "Simulating multivariate nonnormal distributions," Psychometrika, Springer;The Psychometric Society, vol. 48(3), pages 465-471, September.
    2. Todd Headrick & Shlomo Sawilowsky, 1999. "Simulating correlated multivariate nonnormal distributions: Extending the fleishman power method," Psychometrika, Springer;The Psychometric Society, vol. 64(2), pages 251-251, June.
    3. Todd Headrick & Shlomo Sawilowsky, 1999. "Simulating correlated multivariate nonnormal distributions: Extending the fleishman power method," Psychometrika, Springer;The Psychometric Society, vol. 64(1), pages 25-35, March.
    4. Frederic Lord, 1969. "Estimating true-score distributions in psychological testing (an empirical bayes estimation problem)," Psychometrika, Springer;The Psychometric Society, vol. 34(3), pages 259-299, September.
    5. Robert Mislevy, 1984. "Estimating latent distributions," Psychometrika, Springer;The Psychometric Society, vol. 49(3), pages 359-381, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wheadon, Chris, 2014. "Classification Accuracy and Consistency under Item Response Theory Models Using the Package classify," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 56(i10).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pesaran, M. Hashem & Yamagata, Takashi, 2012. "Testing CAPM with a Large Number of Assets," IZA Discussion Papers 6469, Institute of Labor Economics (IZA).
    2. Max Auerswald & Morten Moshagen, 2015. "Generating Correlated, Non-normally Distributed Data Using a Non-linear Structural Model," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 920-937, December.
    3. Mohan D. Pant & Todd C. Headrick, 2017. "Simulating Uniform- and Triangular- Based Double Power Method Distributions," Journal of Statistical and Econometric Methods, SCIENPRESS Ltd, vol. 6(1), pages 1-1.
    4. Al-Subaihi, Ali A., 2004. "Simulating Correlated Multivariate Pseudorandom Numbers," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 9(i04).
    5. M Hashem Pesaran & Takashi Yamagata, 2024. "Testing for Alpha in Linear Factor Pricing Models with a Large Number of Securities," Journal of Financial Econometrics, Oxford University Press, vol. 22(2), pages 407-460.
    6. Headrick, Todd C., 2002. "Fast fifth-order polynomial transforms for generating univariate and multivariate nonnormal distributions," Computational Statistics & Data Analysis, Elsevier, vol. 40(4), pages 685-711, October.
    7. Headrick, Todd C. & Mugdadi, Abdel, 2006. "On simulating multivariate non-normal distributions from the generalized lambda distribution," Computational Statistics & Data Analysis, Elsevier, vol. 50(11), pages 3343-3353, July.
    8. Gary van Vuuren & Riaan de Jongh, 2017. "A comparison of risk aggregation estimates using copulas and Fleishman distributions," Applied Economics, Taylor & Francis Journals, vol. 49(17), pages 1715-1731, April.
    9. Ke-Hai Yuan & Peter Bentler, 2002. "On robusiness of the normal-theory based asymptotic distributions of three reliability coefficient estimates," Psychometrika, Springer;The Psychometric Society, vol. 67(2), pages 251-259, June.
    10. repec:jss:jstsof:09:i04 is not listed on IDEAS
    11. Pesaran, M. H. & Yamagata, T., 2012. "Testing CAPM with a Large Number of Assets (Updated 28th March 2012)," Cambridge Working Papers in Economics 1210, Faculty of Economics, University of Cambridge.
    12. repec:jss:jstsof:19:i03 is not listed on IDEAS
    13. Yen Lee & David Kaplan, 2018. "Generating Multivariate Ordinal Data via Entropy Principles," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 156-181, March.
    14. Beasley, T. Mark & Zumbo, Bruno D., 2003. "Comparison of aligned Friedman rank and parametric methods for testing interactions in split-plot designs," Computational Statistics & Data Analysis, Elsevier, vol. 42(4), pages 569-593, April.
    15. Wendy Yen, 1987. "A comparison of the efficiency and accuracy of BILOG and LOGIST," Psychometrika, Springer;The Psychometric Society, vol. 52(2), pages 275-291, June.
    16. Ping Chen & Chun Wang, 2021. "Using EM Algorithm for Finite Mixtures and Reformed Supplemented EM for MIRT Calibration," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 299-326, March.
    17. Mishra, SK, 2004. "On generating correlated random variables with a given valid or invalid Correlation matrix," MPRA Paper 1782, University Library of Munich, Germany.
    18. Jeffrey J. Hoogland & Anne Boomsma, 1998. "Robustness Studies in Covariance Structure Modeling," Sociological Methods & Research, , vol. 26(3), pages 329-367, February.
    19. Headrick, Todd C. & Sheng, Yanyan & Hodis, Flaviu-Adrian, 2007. "Numerical Computing and Graphics for the Power Method Transformation Using Mathematica," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 19(i03).
    20. Melissa Gladstone & Gillian Lancaster & Gareth McCray & Vanessa Cavallera & Claudia R. L. Alves & Limbika Maliwichi & Muneera A. Rasheed & Tarun Dua & Magdalena Janus & Patricia Kariger, 2021. "Validation of the Infant and Young Child Development (IYCD) Indicators in Three Countries: Brazil, Malawi and Pakistan," IJERPH, MDPI, vol. 18(11), pages 1-19, June.
    21. Mahul, Olivier, 2002. "Hedging Price Risk in the Presence of Crop Yield and Revenue Insurance," 2002 International Congress, August 28-31, 2002, Zaragoza, Spain 24881, European Association of Agricultural Economists.
    22. Shaobo Jin & Fan Yang-Wallentin, 2017. "Asymptotic Robustness Study of the Polychoric Correlation Estimation," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 67-85, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:35:y:2010:i:3:p:280-306. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.