IDEAS home Printed from https://ideas.repec.org/a/bla/jorssb/v84y2022i2p524-546.html
   My bibliography  Save this article

Prediction and outlier detection in classification problems

Author

Listed:
  • Leying Guan
  • Robert Tibshirani

Abstract

We consider the multi‐class classification problem when the training data and the out‐of‐sample test data may have different distributions and propose a method called BCOPS (balanced and conformal optimized prediction sets). BCOPS constructs a prediction set C(x) as a subset of class labels, possibly empty. It tries to optimize the out‐of‐sample performance, aiming to include the correct class and to detect outliers x as often as possible. BCOPS returns no prediction (corresponding to C(x) equal to the empty set) if it infers x to be an outlier. The proposed method combines supervised learning algorithms with conformal prediction to minimize a misclassification loss averaged over the out‐of‐sample distribution. The constructed prediction sets have a finite sample coverage guarantee without distributional assumptions. We also propose a method to estimate the outlier detection rate of a given procedure. We prove asymptotic consistency and optimality of our proposals under suitable assumptions and illustrate our methods on real data examples.

Suggested Citation

  • Leying Guan & Robert Tibshirani, 2022. "Prediction and outlier detection in classification problems," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 524-546, April.
  • Handle: RePEc:bla:jorssb:v:84:y:2022:i:2:p:524-546
    DOI: 10.1111/rssb.12443
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssb.12443
    Download Restriction: no

    File URL: https://libkey.io/10.1111/rssb.12443?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mauricio Sadinle & Jing Lei & Larry Wasserman, 2019. "Least Ambiguous Set-Valued Classifiers With Bounded Error Levels," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(525), pages 223-234, January.
    2. Jing Lei & James Robins & Larry Wasserman, 2013. "Distribution-Free Prediction Sets," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(501), pages 278-287, March.
    3. Cadre, BenoI^t, 2006. "Kernel estimation of density level sets," Journal of Multivariate Analysis, Elsevier, vol. 97(4), pages 999-1023, April.
    4. Jing Lei, 2014. "Classification with confidence," Biometrika, Biometrika Trust, vol. 101(4), pages 755-769.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Leying Guan, 2023. "Localized conformal prediction: a generalized inference framework for conformal prediction," Biometrika, Biometrika Trust, vol. 110(1), pages 33-50.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wei Liu & Frank Bretz & Natchalee Srimaneekarn & Jianan Peng & Anthony J. Hayter, 2019. "Confidence Sets for Statistical Classification," Stats, MDPI, vol. 2(3), pages 1-15, June.
    2. Dau, Hai Dang & Laloë, Thomas & Servien, Rémi, 2020. "Exact asymptotic limit for kernel estimation of regression level sets," Statistics & Probability Letters, Elsevier, vol. 161(C).
    3. Christopher R. Genovese & Marco Perone-Pacifico & Isabella Verdinelli & Larry Wasserman, 2016. "Non-parametric inference for density modes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(1), pages 99-126, January.
    4. Victor Chernozhukov & Kaspar Wuthrich & Yinchu Zhu, 2019. "Distributional conformal prediction," Papers 1909.07889, arXiv.org, revised Aug 2021.
    5. Chiwoo Park & Jianhua Z. Huang & Yu Ding, 2010. "A Computable Plug-In Estimator of Minimum Volume Sets for Novelty Detection," Operations Research, INFORMS, vol. 58(5), pages 1469-1480, October.
    6. Philippe Goulet Coulombe & Mikael Frenette & Karin Klieber, 2023. "From Reactive to Proactive Volatility Modeling with Hemisphere Neural Networks," Working Papers 23-04, Chair in macroeconomics and forecasting, University of Quebec in Montreal's School of Management, revised Nov 2023.
    7. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2021. "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1849-1864, October.
    8. Mammen, Enno & Martínez Miranda, María Dolores & Nielsen, Jens Perch, 2015. "In-sample forecasting applied to reserving and mesothelioma mortality," Insurance: Mathematics and Economics, Elsevier, vol. 61(C), pages 76-86.
    9. Matteo Fontana & Gianluca Zeni & Simone Vantini, 2020. "Conformal Prediction: a Unified Review of Theory and New Challenges," Papers 2005.07972, arXiv.org, revised Jul 2022.
    10. Yen-Chi Chen & Christopher R. Genovese & Larry Wasserman, 2017. "Density Level Sets: Asymptotics, Inference, and Visualization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1684-1696, October.
    11. João A. Bastos, 2023. "Conformal prediction of option prices," Working Papers REM 2023/0304, ISEG - Lisbon School of Economics and Management, REM, Universidade de Lisboa.
    12. João A. Bastos & Jeanne Paquette, 2024. "On the uncertainty of real estate price predictions," Working Papers REM 2024/0314, ISEG - Lisbon School of Economics and Management, REM, Universidade de Lisboa.
    13. Ben O’Neill, 2022. "Smallest covering regions and highest density regions for discrete distributions," Computational Statistics, Springer, vol. 37(3), pages 1229-1254, July.
    14. Philippe Goulet Coulombe & Mikael Frenette & Karin Klieber, 2023. "From Reactive to Proactive Volatility Modeling with Hemisphere Neural Networks," Papers 2311.16333, arXiv.org, revised Apr 2024.
    15. Hu, Jianming & Luo, Qingxi & Tang, Jingwei & Heng, Jiani & Deng, Yuwen, 2022. "Conformalized temporal convolutional quantile regression networks for wind power interval forecasting," Energy, Elsevier, vol. 248(C).
    16. Diquigiovanni, Jacopo & Fontana, Matteo & Vantini, Simone, 2022. "Conformal prediction bands for multivariate functional data," Journal of Multivariate Analysis, Elsevier, vol. 189(C).
    17. Xiaochun Meng & James W. Taylor & Souhaib Ben Taieb & Siran Li, 2020. "Scores for Multivariate Distributions and Level Sets," Papers 2002.09578, arXiv.org, revised Jun 2023.
    18. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2018. "Exact and robust conformal inference methods for predictive machine learning with dependent data," CeMMAP working papers CWP16/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    19. Elena Di Bernardino & Thomas Laloë & Véronique Maume-Deschamps & Clémentine Prieur, 2013. "Plug-in estimation of level sets in a non-compact setting with applications in multivariate risk theory," Post-Print hal-00580624, HAL.
    20. Lihua Lei & Emmanuel J. Candès, 2021. "Conformal inference of counterfactuals and individual treatment effects," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(5), pages 911-938, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssb:v:84:y:2022:i:2:p:524-546. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.