IDEAS home Printed from https://ideas.repec.org/a/bpj/sagmbi/v3y2004i1n4.html
   My bibliography  Save this article

Asymptotic Optimality of Likelihood-Based Cross-Validation

Author

Listed:
  • van der Laan Mark J.

    (Division of Biostatistics, School of Public Health, University of California, Berkeley)

  • Dudoit Sandrine

    (Division of Biostatistics, School of Public Health, University of California, Berkeley)

  • Keles Sunduz

    (Division of Biostatistics, School of Public Health, University of California, Berkeley)

Abstract

Likelihood-based cross-validation is a statistical tool for selecting a density estimate based on n i.i.d. observations from the true density among a collection of candidate density estimators. General examples are the selection of a model indexing a maximum likelihood estimator, and the selection of a bandwidth indexing a nonparametric (e.g. kernel) density estimator. In this article, we establish a finite sample result for a general class of likelihood-based cross-validation procedures (as indexed by the type of sample splitting used, e.g. V-fold cross-validation). This result implies that the cross-validation selector performs asymptotically as well (w.r.t. to the Kullback-Leibler distance to the true density) as a benchmark model selector which is optimal for each given dataset and depends on the true density. Crucial conditions of our theorem are that the size of the validation sample converges to infinity, which excludes leave-one-out cross-validation, and that the candidate density estimates are bounded away from zero and infinity. We illustrate these asymptotic results and the practical performance of likelihood-based cross-validation for the purpose of bandwidth selection with a simulation study. Moreover, we use likelihood-based cross-validation in the context of regulatory motif detection in DNA sequences.

Suggested Citation

  • van der Laan Mark J. & Dudoit Sandrine & Keles Sunduz, 2004. "Asymptotic Optimality of Likelihood-Based Cross-Validation," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 3(1), pages 1-25, March.
    • Handle: RePEc:bpj:sagmbi:v:3:y:2004:i:1:n:4
    DOI: 10.2202/1544-6115.1036
    as

    Download full text from publisher

    File URL: https://doi.org/10.2202/1544-6115.1036
    Download Restriction: For access to full text, subscription to the journal or payment for the individual article is required.
    File URL: https://libkey.io/10.2202/1544-6115.1036?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stitelman Ori M & van der Laan Mark J., 2010. "Collaborative Targeted Maximum Likelihood for Time to Event Data," The International Journal of Biostatistics, De Gruyter, vol. 6(1), pages 1-46, June.
    2. Zhang, Xibin & King, Maxwell L. & Hyndman, Rob J., 2006. "A Bayesian approach to bandwidth selection for multivariate kernel density estimation," Computational Statistics & Data Analysis, Elsevier, vol. 50(11), pages 3009-3031, July.
    3. Rosenblum Michael & van der Laan Mark J., 2010. "Simple, Efficient Estimators of Treatment Effects in Randomized Trials Using Generalized Linear Models to Leverage Baseline Variables," The International Journal of Biostatistics, De Gruyter, vol. 6(1), pages 1-44, April.
    4. Bruce Desmarais, 2012. "Lessons in disguise: multivariate predictive mistakes in collective choice models," Public Choice, Springer, vol. 151(3), pages 719-737, June.
    5. van der Laan Mark J., 2010. "Targeted Maximum Likelihood Based Causal Inference: Part I," The International Journal of Biostatistics, De Gruyter, vol. 6(2), pages 1-45, February.
    6. Filippone, Maurizio & Sanguinetti, Guido, 2011. "Approximate inference of the bandwidth in multivariate kernel density estimation," Computational Statistics & Data Analysis, Elsevier, vol. 55(12), pages 3104-3122, December.
    7. Susanne M. Schennach, 2013. "Regressions with Berkson errors in covariates - A nonparametric approach," Papers 1308.2836, arXiv.org.
    8. Brooks Jordan & van der Laan Mark J. & Go Alan S., 2012. "Targeted Maximum Likelihood Estimation for Prediction Calibration," The International Journal of Biostatistics, De Gruyter, vol. 8(1), pages 1-35, October.
    9. Arafat Tayeb & Aurélie Labbe & Alexandre Bureau & Chantal Mérette, 2011. "Solving genetic heterogeneity in extended families by identifying sub-types of complex diseases," Computational Statistics, Springer, vol. 26(3), pages 539-560, September.
    10. Roberto Rocci & Stefano Antonio Gattone & Roberto Di Mari, 2018. "A data driven equivariant approach to constrained Gaussian mixture modeling," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 12(2), pages 235-260, June.
    11. Benkeser David & Mertens Andrew & Colford John M. & Hubbard Alan & Arnold Benjamin F. & Stein Aryeh & van der Laan Mark J., 2021. "A machine learning-based approach for estimating and testing associations with multivariate outcomes," The International Journal of Biostatistics, De Gruyter, vol. 17(1), pages 7-21, May.
    12. Tomas Ruzgas & Mantas Lukauskas & Gedmantas Čepkauskas, 2021. "Nonparametric Multivariate Density Estimation: Case Study of Cauchy Mixture Model," Mathematics, MDPI, vol. 9(21), pages 1-22, October.
    13. van der Laan Mark J. & Gruber Susan, 2010. "Collaborative Double Robust Targeted Maximum Likelihood Estimation," The International Journal of Biostatistics, De Gruyter, vol. 6(1), pages 1-71, May.
    14. Shang, Han Lin, 2013. "Bayesian bandwidth estimation for a nonparametric functional regression model with unknown error density," Computational Statistics & Data Analysis, Elsevier, vol. 67(C), pages 185-198.
    15. Qu, Leming & Yin, Wotao, 2012. "Copula density estimation by total variation penalized likelihood with linear equality constraints," Computational Statistics & Data Analysis, Elsevier, vol. 56(2), pages 384-398.
    16. Hu, Yingyao & Schennach, Susanne & Shiu, Ji-Liang, 2022. "Identification of nonparametric monotonic regression models with continuous nonclassical measurement errors," Journal of Econometrics, Elsevier, vol. 226(2), pages 269-294.
    17. Molinaro, Annette M. & Dudoit, Sandrine & van der Laan, M.J.Mark J., 2004. "Tree-based multivariate regression and density estimation with right-censored data," Journal of Multivariate Analysis, Elsevier, vol. 90(1), pages 154-177, July.
    18. Díaz Muñoz Iván & van der Laan Mark J., 2011. "Super Learner Based Conditional Density Estimation with Application to Marginal Structural Models," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-20, October.
    19. Stitelman Ori M & Wester C. William & De Gruttola Victor & van der Laan Mark J., 2011. "Targeted Maximum Likelihood Estimation of Effect Modification Parameters in Survival Analysis," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-34, March.
    20. Labbe Aurelie & Bureau Alexandre & Merette Chantal, 2009. "Integration of Genetic Familial Dependence Structure in Latent Class Models," The International Journal of Biostatistics, De Gruyter, vol. 5(1), pages 1-30, January.
    21. Perrin, G. & Soize, C. & Ouhbi, N., 2018. "Data-driven kernel representations for sampling with an unknown block dependence structure under correlation constraints," Computational Statistics & Data Analysis, Elsevier, vol. 119(C), pages 139-154.
    22. Mahmood Zafar & Khan Salahuddin, 2009. "On the Use of K-Fold Cross-Validation to Choose Cutoff Values and Assess the Performance of Predictive Models in Stepwise Regression," The International Journal of Biostatistics, De Gruyter, vol. 5(1), pages 1-21, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:sagmbi:v:3:y:2004:i:1:n:4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyter.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.