IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v77y2014icp300-312.html
   My bibliography  Save this article

Making classifier performance comparisons when ROC curves intersect

Author

Listed:
  • Gigliarano, Chiara
  • Figini, Silvia
  • Muliere, Pietro

Abstract

The ROC curve is one of the most common statistical tools useful to assess classifier performance. The selection of the best classifier when ROC curves intersect is quite challenging. A novel approach for model comparisons when ROC curves show intersections is proposed. In particular, the relationship between ROC orderings and stochastic dominance is investigated in a theoretical framework and a general class of indicators is proposed which is coherent with dominance criteria also when ROC curves cross. Furthermore, a simulation study and a real application to credit risk data are proposed to illustrate the use of the new methodological approach.

Suggested Citation

  • Gigliarano, Chiara & Figini, Silvia & Muliere, Pietro, 2014. "Making classifier performance comparisons when ROC curves intersect," Computational Statistics & Data Analysis, Elsevier, vol. 77(C), pages 300-312.
    • Handle: RePEc:eee:csdana:v:77:y:2014:i:c:p:300-312
    DOI: 10.1016/j.csda.2014.03.008
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947314000851
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2014.03.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhang, Biao, 2006. "A semiparametric hypothesis testing procedure for the ROC curve area under a density ratio model," Computational Statistics & Data Analysis, Elsevier, vol. 50(7), pages 1855-1876, April.
    2. Fishburn, Peter C., 1980. "Continua of stochastic dominance relations for unbounded probability distributions," Journal of Mathematical Economics, Elsevier, vol. 7(3), pages 271-285, December.
    3. Garry F. Barrett & Stephen G. Donald, 2003. "Consistent Tests for Stochastic Dominance," Econometrica, Econometric Society, vol. 71(1), pages 71-104, January.
    4. Yousef, Waleed A., 2013. "Assessing classifiers in terms of the partial area under the ROC curve," Computational Statistics & Data Analysis, Elsevier, vol. 64(C), pages 51-70.
    5. Anderson, Gordon, 1996. "Nonparametric Tests of Stochastic Dominance in Income Distributions," Econometrica, Econometric Society, vol. 64(5), pages 1183-1193, September.
    6. Lori E. Dodd & Margaret S. Pepe, 2003. "Partial AUC Estimation and Regression," Biometrics, The International Biometric Society, vol. 59(3), pages 614-623, September.
    7. David J. Hand, 2012. "Assessing the Performance of Classification Methods," International Statistical Review, International Statistical Institute, vol. 80(3), pages 400-414, December.
    8. Donna Katzman McClish, 1989. "Analyzing a Portion of the ROC Curve," Medical Decision Making, , vol. 9(3), pages 190-195, August.
    9. Lee, Sauchi Stephen, 2000. "Noisy replication in skewed binary classification," Computational Statistics & Data Analysis, Elsevier, vol. 34(2), pages 165-191, August.
    10. Anthony F. Shorrocks & James E. Foster, 1987. "Transfer Sensitive Inequality Measures," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 54(3), pages 485-497.
    11. Fishburn, Peter C., 1976. "Continua of stochastic dominance relations for bounded probability distributions," Journal of Mathematical Economics, Elsevier, vol. 3(3), pages 295-311, December.
    12. Russell Davidson & Jean-Yves Duclos, 2000. "Statistical Inference for Stochastic Dominance and for the Measurement of Poverty and Inequality," Econometrica, Econometric Society, vol. 68(6), pages 1435-1464, November.
    13. Muliere, Pietro & Scarsini, Marco, 1989. "A note on stochastic dominance and inequality measures," Journal of Economic Theory, Elsevier, vol. 49(2), pages 314-323, December.
    14. Thomas, Lyn C., 2009. "Consumer Credit Models: Pricing, Profit and Portfolios," OUP Catalogue, Oxford University Press, number 9780199232130.
    15. Atkinson, Anthony B., 1970. "On the measurement of inequality," Journal of Economic Theory, Elsevier, vol. 2(3), pages 244-263, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Abbas Keramati & Hajar Ghaneei & Seyed Mohammad Mirmohammadi, 2016. "Developing a prediction model for customer churn from electronic banking services using data mining," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 2(1), pages 1-13, December.
    2. Elena Ballante & Silvia Figini & Pierpaolo Uberti, 2022. "A new approach in model selection for ordinal target variables," Computational Statistics, Springer, vol. 37(1), pages 43-56, March.
    3. Verme, Paolo & Gigliarano, Chiara, 2019. "Optimal targeting under budget constraints in a humanitarian context," World Development, Elsevier, vol. 119(C), pages 224-233.
    4. Silvia Figini & Roberto Savona & Marika Vezzoli, 2016. "Corporate Default Prediction Model Averaging: A Normative Linear Pooling Approach," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 23(1-2), pages 6-20, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Aaberge, Rolf & Havnes, Tarjei & Mogstad, Magne, 2013. "A Theory for Ranking Distribution Functions," IZA Discussion Papers 7738, Institute of Labor Economics (IZA).
    2. Rolf Aaberge & Tarjei Havnes & Magne Mogstad, 2021. "Ranking intersecting distribution functions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(6), pages 639-662, September.
    3. Francesco Andreoli, 2018. "Robust Inference for Inverse Stochastic Dominance," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(1), pages 146-159, January.
    4. Hongyi Jiang & Zhenting Sun & Shiyun Hu, 2023. "A Nonparametric Test of $m$th-degree Inverse Stochastic Dominance," Papers 2306.12271, arXiv.org, revised Jul 2023.
    5. Oliver Linton & Esfandiar Maasoumi & Yoon-Jae Wang, 2002. "Consistent testing for stochastic dominance: a subsampling approach," CeMMAP working papers 03/02, Institute for Fiscal Studies.
    6. Härdle, Wolfgang Karl & Schulz, Rainer & Xie, Taojun, 2019. "Cooling Measures and Housing Wealth: Evidence from Singapore," IRTG 1792 Discussion Papers 2019-001, Humboldt University of Berlin, International Research Training Group 1792 "High Dimensional Nonstationary Time Series".
    7. Maasoumi, Esfandiar & Almas Heshmati, 2003. "Evaluating Dominance Ranking of PSID Incomes by various Household Attributes," Departmental Working Papers 0509, Southern Methodist University, Department of Economics.
    8. Raymond H. Chan & Ephraim Clark & Xu Guo & Wing-Keung Wong, 2020. "New development on the third-order stochastic dominance for risk-averse and risk-seeking investors with application in risk management," Risk Management, Palgrave Macmillan, vol. 22(2), pages 108-132, June.
    9. David Lander & David Gunawan & William Griffiths & Duangkamon Chotikapanich, 2020. "Bayesian assessment of Lorenz and stochastic dominance," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 53(2), pages 767-799, May.
    10. Wolfgang Karl Hardle & Rainer Schulz & Taojun Sie, 2021. "Cooling Measures and Housing Wealth: Evidence from Singapore," Papers 2108.11915, arXiv.org.
    11. David Lander & David Gunawan & William E. Griffiths & Duangkamon Chotikapanich, 2016. "Bayesian Assessment of Lorenz and Stochastic Dominance Using a Mixture of Gamma Densities," Department of Economics - Working Papers Series 2023, The University of Melbourne.
    12. Pinar, Mehmet & Stengos, Thanasis & Topaloglou, Nikolas, 2020. "On the construction of a feasible range of multidimensional poverty under benchmark weight uncertainty," European Journal of Operational Research, Elsevier, vol. 281(2), pages 415-427.
    13. Francesco Andreoli & Claudio Zoli, 2020. "From unidimensional to multidimensional inequality: a review," METRON, Springer;Sapienza Università di Roma, vol. 78(1), pages 5-42, April.
    14. Frank A. Cowell & Emmanuel Flachaire, 2014. "Statistical Methods for Distributional Analysis," Working Papers halshs-01115996, HAL.
    15. James E. Foster & Joel Greer & Erik Thorbecke, 2010. "The Foster-Greer-Thorbecke (FGT) Poverty Measures: Twenty-Five Years Later," Working Papers 2010-14, The George Washington University, Institute for International Economic Policy.
    16. Iosif Pinelis, 2013. "An optimal three-way stable and monotonic spectrum of bounds on quantiles: a spectrum of coherent measures of financial risk and economic inequality," Papers 1310.6025, arXiv.org.
    17. Rothe, Christoph, 2010. "Nonparametric estimation of distributional policy effects," Journal of Econometrics, Elsevier, vol. 155(1), pages 56-70, March.
    18. Charles Beach, 2023. "Quantile Tool Box Measures for Empirical Analysis and for Testing Distributional Comparisons in Direct Distribution-Free Fashion," Working Paper 1508, Economics Department, Queen's University.
    19. Kuan Xu & Gordon Fisher, 2006. "Myopic loss aversion and margin of safety: the risk of value investing," Quantitative Finance, Taylor & Francis Journals, vol. 6(6), pages 481-494.
    20. Francesco Andreoli, 2013. "Inference for Inverse Stochastic Dominance," Working Papers 295, ECINEQ, Society for the Study of Economic Inequality.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:77:y:2014:i:c:p:300-312. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.