IDEAS home Printed from https://ideas.repec.org/a/bla/istatr/v78y2010i2p240-256.html
   My bibliography  Save this article

On Model Selection Curves

Author

Listed:
  • Samuel Müller
  • Alan H. Welsh

Abstract

Many popular methods of model selection involve minimizing a penalized function of the data (such as the maximized log‐likelihood or the residual sum of squares) over a set of models. The penalty in the criterion function is controlled by a penalty multiplier λ which determines the properties of the procedure. In this paper, we first review model selection criteria of the simple form “Loss + Penalty” and then propose studying such model selection criteria as functions of the penalty multiplier. This approach can be interpreted as exploring the stability of model selection criteria through what we call model selection curves. It leads to new insights into model selection and new proposals on how to select models. We use the bootstrap to enhance the basic model selection curve and develop convenient numerical and graphical summaries of the results. The methodology is illustrated on two data sets and supported by a small simulation. We show that the new methodology can outperform methods such as AIC and BIC which correspond to single points on a model selection curve. Beaucoup de méthodes populaires de sélection de variables impliquent la minimisation d'une fonction pénalisée des données (comme la vraisemblance maximisée ou la somme résiduelle carrés) sur un jeu de modèles. La pénalité dans la fonction de critère est contrôlée par un multiplicateur de pénalitéλ qui détermine les propriétés de la procédure. Nous reconsidérons d'abord des critères de sélection modèles de la forme simple ‘Perte + Pénalité’ et proposons ensuite d'étudier de telles fonctions comme les fonctions du multiplicateur de pénalité. Cette approche peut être interprétée comme l'exploration de la stabilité de fonctions de critère par ce que nous appelons des courbes de choix modèles. Il mène à de nouvelles compréhensions dans le sélection de variables et de nouvelles propositions de la façon d'utiliser ces fonctions de critère pour sélectionner de variables. Nous utilisons le bootstrap pour augmenter des courbes de choix modèle et développent les résumés numériques et graphiques des résultats. La méthodologie est illustrée sur deux jeux de données et soutenue par une petite simulation. Nous montrons que la nouvelle méthodologie peut surpasser des méthodes comme AIC et BIC qui correspond aux points simples sur une courbe de choix modèle.

Suggested Citation

  • Samuel Müller & Alan H. Welsh, 2010. "On Model Selection Curves," International Statistical Review, International Statistical Institute, vol. 78(2), pages 240-256, August.
  • Handle: RePEc:bla:istatr:v:78:y:2010:i:2:p:240-256
    DOI: 10.1111/j.1751-5823.2010.00108.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1751-5823.2010.00108.x
    Download Restriction: no

    File URL: https://libkey.io/10.1111/j.1751-5823.2010.00108.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Yuhong Yang, 2005. "Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation," Biometrika, Biometrika Trust, vol. 92(4), pages 937-950, December.
    2. Claeskens,Gerda & Hjort,Nils Lid, 2008. "Model Selection and Model Averaging," Cambridge Books, Cambridge University Press, number 9780521852258.
    3. Zhao, L. C. & Krishnaiah, P. R. & Bai, Z. D., 1986. "On detection of the number of signals when the noise covariance matrix is arbitrary," Journal of Multivariate Analysis, Elsevier, vol. 20(1), pages 26-49, October.
    4. Wisnowski, James W. & Simpson, James R. & Montgomery, Douglas C. & Runger, George C., 2003. "Resampling methods for variable selection in robust regression," Computational Statistics & Data Analysis, Elsevier, vol. 43(3), pages 341-355, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jiming Jiang & Thuan Nguyen & J. Sunil Rao, 2015. "The E-MS Algorithm: Model Selection With Incomplete Data," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(511), pages 1136-1147, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. HAEDO, Christian & MOUCHART , Michel & ,, 2013. "Specialized agglomerations with areal data: model and detection," LIDAM Discussion Papers CORE 2013060, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    2. Kira Alhorn & Holger Dette & Kirsten Schorning, 2021. "Optimal Designs for Model Averaging in non-nested Models," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 83(2), pages 745-778, August.
    3. Jun Yu & HaiYing Wang, 2022. "Subdata selection algorithm for linear model discrimination," Statistical Papers, Springer, vol. 63(6), pages 1883-1906, December.
    4. Yonekura, Shouto & Beskos, Alexandros & Singh, Sumeetpal S., 2021. "Asymptotic analysis of model selection criteria for general hidden Markov models," Stochastic Processes and their Applications, Elsevier, vol. 132(C), pages 164-191.
    5. DiTraglia, Francis J., 2016. "Using invalid instruments on purpose: Focused moment selection and averaging for GMM," Journal of Econometrics, Elsevier, vol. 195(2), pages 187-208.
    6. Lee, Yoonseok & Phillips, Peter C.B., 2015. "Model selection in the presence of incidental parameters," Journal of Econometrics, Elsevier, vol. 188(2), pages 474-489.
    7. Zhang, Xinyu & Liu, Chu-An, 2023. "Model averaging prediction by K-fold cross-validation," Journal of Econometrics, Elsevier, vol. 235(1), pages 280-301.
    8. Ruoyao Shi, 2021. "An Averaging Estimator for Two Step M Estimation in Semiparametric Models," Working Papers 202105, University of California at Riverside, Department of Economics.
    9. Mathias Drton & Martyn Plummer, 2017. "A Bayesian information criterion for singular models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(2), pages 323-380, March.
    10. Kitagawa, Toru & Muris, Chris, 2016. "Model averaging in semiparametric estimation of treatment effects," Journal of Econometrics, Elsevier, vol. 193(1), pages 271-289.
    11. Jeffrey S. Racine & Qi Li & Dalei Yu & Li Zheng, 2023. "Optimal Model Averaging of Mixed-Data Kernel-Weighted Spline Regressions," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 41(4), pages 1251-1261, October.
    12. Chirwa, Themba G. & Odhiambo, Nicholas M., 2016. "What Drives Long-Run Economic Growth? Empirical Evidence from South Africa," Economia Internazionale / International Economics, Camera di Commercio Industria Artigianato Agricoltura di Genova, vol. 69(4), pages 429-456.
    13. Philippe Goulet Coulombe & Maxime Leroux & Dalibor Stevanovic & Stéphane Surprenant, 2022. "How is machine learning useful for macroeconomic forecasting?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 920-964, August.
    14. Davide Fiaschi & Andrea Mario Lavezzi & Angela Parenti, 2020. "Deep and Proximate Determinants of the World Income Distribution," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 66(3), pages 677-710, September.
    15. Fabio Canova & Christian Matthes, 2021. "Dealing with misspecification in structural macroeconometric models," Quantitative Economics, Econometric Society, vol. 12(2), pages 313-350, May.
    16. Yu, Jun & Meng, Xiran & Wang, Yaping, 2023. "Optimal designs for semi-parametric dose-response models under random contamination," Computational Statistics & Data Analysis, Elsevier, vol. 178(C).
    17. Worthington, Andrew C. & Zelenyuk, Valentin, 2018. "Data envelopment analysis, truncated regression and double-bootstrap for panel data with application to Chinese bankingAuthor-Name: Du, Kai," European Journal of Operational Research, Elsevier, vol. 265(2), pages 748-764.
    18. Zhongqi Liang & Qihua Wang & Yuting Wei, 2022. "Robust model selection with covariables missing at random," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 74(3), pages 539-557, June.
    19. Bhattacharya, Debopam & Dupas, Pascaline, 2012. "Inferring welfare maximizing treatment assignment under budget constraints," Journal of Econometrics, Elsevier, vol. 167(1), pages 168-196.
    20. Schomaker Michael & Heumann Christian, 2011. "Model Averaging in Factor Analysis: An Analysis of Olympic Decathlon Data," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 7(1), pages 1-15, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:istatr:v:78:y:2010:i:2:p:240-256. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/isiiinl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.