IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i9p1493-d806515.html
   My bibliography  Save this article

The Utility of Receiver Operating Characteristic Curve in Educational Assessment: Performance Prediction

Author

Listed:
  • Hyunsuk Han

    (Department of Counseling Psychology, Kyungil University, Gyeongsan 38428, Korea)

Abstract

When examinees are classified into groups based on scores from educational assessment, two indices are widely used to gauge the psychometric quality of the classifications: accuracy and consistency. The two indices take correct classifications into consideration while overlooking incorrect ones, where unbalanced class distribution threatens the validity of results from the accuracy and consistency indices. The single values produced from the two indices also fail to address the inconsistent accuracy of the classifier across different cut score locations. The current study proposed the concept of classification quality, which utilizes the receiver operating characteristics (ROC) graph to comprehensively evaluate the performance of classifiers. The ROC graph illustrates the tradeoff between benefits (true positive rate) and costs (false positive rate) in classification. In this article, a simulation study was conducted to demonstrate how to generate and interpret ROC graphs in educational assessment and the benefits of using ROC graphs to interpret classification quality. The results show that ROC graphs provide an efficient approach to (a) visualize the fluctuating performance of scoring classifiers, (b) address the unbalanced class distribution issue inherent in the accuracy and consistency indices, and (c) produce more accurate estimation of the classification results.

Suggested Citation

  • Hyunsuk Han, 2022. "The Utility of Receiver Operating Characteristic Curve in Educational Assessment: Performance Prediction," Mathematics, MDPI, vol. 10(9), pages 1-11, April.
    • Handle: RePEc:gam:jmathe:v:10:y:2022:i:9:p:1493-:d:806515
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/9/1493/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/10/9/1493/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jun Zhang & Shane Mueller, 2005. "A note on ROC analysis and non-parametric estimate of sensitivity," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 203-212, March.
    2. Allen Fleishman, 1978. "A method for simulating non-normal distributions," Psychometrika, Springer;The Psychometric Society, vol. 43(4), pages 521-532, December.
    3. Huynh Huynh, 1976. "Statistical consideration of mastery scores," Psychometrika, Springer;The Psychometric Society, vol. 41(1), pages 65-78, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Doole, Graeme J. & Romera, Alvaro J. & Leslie, Jennifer E. & Chapman, David F. & Pinxterhuis, Ina (J.B.). & Kemp, Peter D., 2021. "Economic assessment of plantain (Plantago lanceolata) uptake in the New Zealand dairy sector," Agricultural Systems, Elsevier, vol. 187(C).
    2. Headrick, Todd C. & Sheng, Yanyan & Hodis, Flaviu-Adrian, 2007. "Numerical Computing and Graphics for the Power Method Transformation Using Mathematica," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 19(i03).
    3. Pesaran, M. Hashem & Yamagata, Takashi, 2012. "Testing CAPM with a Large Number of Assets," IZA Discussion Papers 6469, Institute of Labor Economics (IZA).
    4. Schinckus, Christophe, 2015. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 35(C), pages 104-110.
    5. Max Auerswald & Morten Moshagen, 2015. "Generating Correlated, Non-normally Distributed Data Using a Non-linear Structural Model," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 920-937, December.
    6. Weihua Fan & Gregory R. Hancock, 2012. "Robust Means Modeling," Journal of Educational and Behavioral Statistics, , vol. 37(1), pages 137-156, February.
    7. Mohan D. Pant & Todd C. Headrick, 2017. "Simulating Uniform- and Triangular- Based Double Power Method Distributions," Journal of Statistical and Econometric Methods, SCIENPRESS Ltd, vol. 6(1), pages 1-1.
    8. Mahul, Olivier, 2002. "Hedging Price Risk in the Presence of Crop Yield and Revenue Insurance," 2002 International Congress, August 28-31, 2002, Zaragoza, Spain 24881, European Association of Agricultural Economists.
    9. Löhndorf, Nils, 2016. "An empirical analysis of scenario generation methods for stochastic optimization," European Journal of Operational Research, Elsevier, vol. 255(1), pages 121-132.
    10. Huynh Huynh, 1980. "A nonrandomized minimax solution for passing scores in the binomial error model," Psychometrika, Springer;The Psychometric Society, vol. 45(2), pages 167-182, June.
    11. Shaobo Jin & Fan Yang-Wallentin, 2017. "Asymptotic Robustness Study of the Polychoric Correlation Estimation," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 67-85, March.
    12. Schinckus, Christophe, 2018. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 45(C), pages 1-6.
    13. Foss, Tron & Jöreskog, Karl G. & Olsson, Ulf H., 2011. "Testing structural equation models: The effect of kurtosis," Computational Statistics & Data Analysis, Elsevier, vol. 55(7), pages 2263-2275, July.
    14. Huynh Huynh, 1982. "A bayesian procedure for mastery decisions based on multivariate normal test data," Psychometrika, Springer;The Psychometric Society, vol. 47(3), pages 309-319, September.
    15. Pelagatti, Matteo M. & Sen, Pranab K., 2013. "Rank tests for short memory stationarity," Journal of Econometrics, Elsevier, vol. 172(1), pages 90-105.
    16. Nicolas Depraetere & Martina Vandebroek, 2014. "Order selection in finite mixtures of linear regressions," Statistical Papers, Springer, vol. 55(3), pages 871-911, August.
    17. Emanuela Raffinetti & Pier Alda Ferrari, 2021. "A dependence measure flow tree through Monte Carlo simulations," Quality & Quantity: International Journal of Methodology, Springer, vol. 55(2), pages 467-496, April.
    18. Al-Subaihi, Ali A., 2004. "Simulating Correlated Multivariate Pseudorandom Numbers," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 9(i04).
    19. Ringle, Christian M. & Götz, Oliver & Wetzels, Martin & Wilson, Bradley, 2009. "On the Use of Formative Measurement Specifications in Structural Equation Modeling: A Monte Carlo Simulation Study to Compare Covariance-Based and Partial Least Squares Model Estimation Methodologies," MPRA Paper 15390, University Library of Munich, Germany.
    20. Africa Borges del Rosal & Concepción San Luis & Alfonso Sánchez-Bruno, 2003. "Dominance Statistics: A Simulation Study on the d Statistic," Quality & Quantity: International Journal of Methodology, Springer, vol. 37(3), pages 303-316, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:9:p:1493-:d:806515. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.