IDEAS home Printed from https://ideas.repec.org/a/bla/jorssb/v70y2008i1p159-173.html
   My bibliography  Save this article

Theoretical measures of relative performance of classifiers for high dimensional data with small sample sizes

Author

Listed:
  • Peter Hall
  • Yvonne Pittelkow
  • Malay Ghosh

Abstract

We suggest a technique, related to the concept of ‘detection boundary’ that was developed by Ingster and by Donoho and Jin, for comparing the theoretical performance of classifiers constructed from small training samples of very large vectors. The resulting ‘classification boundaries’ are obtained for a variety of distance‐based methods, including the support vector machine, distance‐weighted discrimination and kth‐nearest‐neighbour classifiers, for thresholded forms of those methods, and for techniques based on Donoho and Jin's higher criticism approach to signal detection. Assessed in these terms, standard distance‐based methods are shown to be capable only of detecting differences between populations when those differences can be estimated consistently. However, the thresholded forms of distance‐based classifiers can do better, and in particular can correctly classify data even when differences between distributions are only detectable, not estimable. Other methods, including higher criticism classifiers, can on occasion perform better still, but they tend to be more limited in scope, requiring substantially more information about the marginal distributions. Moreover, as tail weight becomes heavier the classification boundaries of methods designed for particular distribution types can converge to, and achieve, the boundary for thresholded nearest neighbour approaches. For example, although higher criticism has a lower classification boundary, and in this sense performs better, in the case of normal data, the boundaries are identical for exponentially distributed data when both sample sizes equal 1.

Suggested Citation

  • Peter Hall & Yvonne Pittelkow & Malay Ghosh, 2008. "Theoretical measures of relative performance of classifiers for high dimensional data with small sample sizes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 70(1), pages 159-173, February.
  • Handle: RePEc:bla:jorssb:v:70:y:2008:i:1:p:159-173
    DOI: 10.1111/j.1467-9868.2007.00631.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1467-9868.2007.00631.x
    Download Restriction: no

    File URL: https://libkey.io/10.1111/j.1467-9868.2007.00631.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Peter Hall & J. S. Marron & Amnon Neeman, 2005. "Geometric representation of high dimension, low sample size data," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(3), pages 427-444, June.
    2. Politis, D. N. & Romano, Joseph P. & Wolf, Michael, 1997. "Subsampling for heteroskedastic time series," Journal of Econometrics, Elsevier, vol. 81(2), pages 281-317, December.
    3. Efron, Bradley, 2004. "Large-Scale Simultaneous Hypothesis Testing: The Choice of a Null Hypothesis," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 96-104, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Makoto Aoshima & Kazuyoshi Yata, 2019. "Distance-based classifier by data transformation for high-dimension, strongly spiked eigenvalue models," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 71(3), pages 473-503, June.
    2. Makoto Aoshima & Kazuyoshi Yata, 2014. "A distance-based, misclassification rate adjusted classifier for multiclass, high-dimensional data," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 66(5), pages 983-1010, October.
    3. Yugo Nakayama & Kazuyoshi Yata & Makoto Aoshima, 2020. "Bias-corrected support vector machine with Gaussian kernel in high-dimension, low-sample-size settings," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 72(5), pages 1257-1286, October.
    4. Rauf Ahmad, M. & Pavlenko, Tatjana, 2018. "A U-classifier for high-dimensional data under non-normality," Journal of Multivariate Analysis, Elsevier, vol. 167(C), pages 269-283.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Geert Bekaert & Robert J. Hodrick, 2001. "Expectations Hypotheses Tests," Journal of Finance, American Finance Association, vol. 56(4), pages 1357-1394, August.
    2. Paulo M. D. C. Parente & Richard J. Smith, 2021. "Quasi‐maximum likelihood and the kernel block bootstrap for nonlinear dynamic models," Journal of Time Series Analysis, Wiley Blackwell, vol. 42(4), pages 377-405, July.
    3. Pounds Stanley B. & Gao Cuilan L. & Zhang Hui, 2012. "Empirical Bayesian Selection of Hypothesis Testing Procedures for Analysis of Sequence Count Expression Data," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 11(5), pages 1-32, October.
    4. Yata, Kazuyoshi & Aoshima, Makoto, 2013. "PCA consistency for the power spiked model in high-dimensional settings," Journal of Multivariate Analysis, Elsevier, vol. 122(C), pages 334-354.
    5. Jung, Sungkyu & Sen, Arusharka & Marron, J.S., 2012. "Boundary behavior in High Dimension, Low Sample Size asymptotics of PCA," Journal of Multivariate Analysis, Elsevier, vol. 109(C), pages 190-203.
    6. Shigeyuki Matsui & Hisashi Noma, 2011. "Estimating Effect Sizes of Differentially Expressed Genes for Power and Sample-Size Assessments in Microarray Experiments," Biometrics, The International Biometric Society, vol. 67(4), pages 1225-1235, December.
    7. Dichtl, Hubert & Drobetz, Wolfgang, 2011. "Portfolio insurance and prospect theory investors: Popularity and optimal design of capital protected financial products," Journal of Banking & Finance, Elsevier, vol. 35(7), pages 1683-1697, July.
    8. Wang, Shao-Hsuan & Huang, Su-Yun, 2022. "Perturbation theory for cross data matrix-based PCA," Journal of Multivariate Analysis, Elsevier, vol. 190(C).
    9. Kazuyoshi Yata & Makoto Aoshima, 2012. "Inference on High-Dimensional Mean Vectors with Fewer Observations Than the Dimension," Methodology and Computing in Applied Probability, Springer, vol. 14(3), pages 459-476, September.
    10. Won, Joong-Ho & Lim, Johan & Yu, Donghyeon & Kim, Byung Soo & Kim, Kyunga, 2014. "Monotone false discovery rate," Statistics & Probability Letters, Elsevier, vol. 87(C), pages 86-93.
    11. Saha, Enakshi & Sarkar, Soham & Ghosh, Anil K., 2017. "Some high-dimensional one-sample tests based on functions of interpoint distances," Journal of Multivariate Analysis, Elsevier, vol. 161(C), pages 83-95.
    12. van Wieringen, Wessel N. & Stam, Koen A. & Peeters, Carel F.W. & van de Wiel, Mark A., 2020. "Updating of the Gaussian graphical model through targeted penalized estimation," Journal of Multivariate Analysis, Elsevier, vol. 178(C).
    13. Ian W. McKeague & Min Qian, 2015. "An Adaptive Resampling Test for Detecting the Presence of Significant Predictors," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1422-1433, December.
    14. Angela Schörgendorfer & Adam J. Branscum & Timothy E. Hanson, 2013. "A Bayesian Goodness of Fit Test and Semiparametric Generalization of Logistic Regression with Measurement Data," Biometrics, The International Biometric Society, vol. 69(2), pages 508-519, June.
    15. Romano, Joseph P. & Wolf, Michael, 1998. "Subsampling confidence intervals for the autoregressive root," DES - Working Papers. Statistics and Econometrics. WS 6268, Universidad Carlos III de Madrid. Departamento de Estadística.
    16. Iván Blanco, Juan Ignacio Peña, and Rosa Rodriguez, 2018. "Modelling Electricity Swaps with Stochastic Forward Premium Models," The Energy Journal, International Association for Energy Economics, vol. 0(Number 2).
    17. Jean-Marie DUFOUR & Lynda KHALAF & Marcel VOIA, 2013. "Finite-Sample Resampling-Based Combined Hypothesis Tests, with Applications to Serial Correlation and Predictability," Cahiers de recherche 13-2013, Centre interuniversitaire de recherche en économie quantitative, CIREQ.
    18. Yugo Nakayama & Kazuyoshi Yata & Makoto Aoshima, 2020. "Bias-corrected support vector machine with Gaussian kernel in high-dimension, low-sample-size settings," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 72(5), pages 1257-1286, October.
    19. Bar, Haim & Wells, Martin T., 2023. "On graphical models and convex geometry," Computational Statistics & Data Analysis, Elsevier, vol. 187(C).
    20. Marie Briere & Léopold Simar & Ariane Szafarz & Anne Vanhems, 2023. "Sensitivity to measurement errors of the distance to the efficient frontier," Working Papers CEB 23-004, ULB -- Universite Libre de Bruxelles.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssb:v:70:y:2008:i:1:p:159-173. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.