IDEAS home Printed from https://ideas.repec.org/a/spr/alstar/v97y2013i4p349-385.html
   My bibliography  Save this article

Penalized likelihood and Bayesian function selection in regression models

Author

Listed:
  • Fabian Scheipl
  • Thomas Kneib
  • Ludwig Fahrmeir

Abstract

Challenging research in various fields has driven a wide range of methodological advances in variable selection for regression models with high-dimensional predictors. In comparison, selection of nonlinear functions in models with additive predictors has been considered only more recently. Several competing suggestions have been developed at about the same time and often do not refer to each other. This article provides a state-of-the-art review on function selection, focusing on penalized likelihood and Bayesian concepts, relating various approaches to each other in a unified framework. In an empirical comparison, also including boosting, we evaluate several methods through applications to simulated and real data, thereby providing some guidance on their performance in practice. Copyright Springer-Verlag Berlin Heidelberg 2013

Suggested Citation

  • Fabian Scheipl & Thomas Kneib & Ludwig Fahrmeir, 2013. "Penalized likelihood and Bayesian function selection in regression models," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 97(4), pages 349-385, October.
    • Handle: RePEc:spr:alstar:v:97:y:2013:i:4:p:349-385
    DOI: 10.1007/s10182-013-0211-3
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s10182-013-0211-3
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s10182-013-0211-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zou, Hui, 2006. "The Adaptive Lasso and Its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1418-1429, December.
    2. Lukas Meier & Sara Van De Geer & Peter Bühlmann, 2008. "The group lasso for logistic regression," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 70(1), pages 53-71, February.
    3. Panagiotelis, Anastasios & Smith, Michael, 2008. "Bayesian identification, selection and estimation of semiparametric functions in high-dimensional additive models," Journal of Econometrics, Elsevier, vol. 143(2), pages 291-316, April.
    4. Thomas Kneib & Torsten Hothorn & Gerhard Tutz, 2009. "Variable Selection and Model Choice in Geoadditive Regression Models," Biometrics, The International Biometric Society, vol. 65(2), pages 626-634, June.
    5. Radchenko, Peter & James, Gareth M., 2010. "Variable Selection Using Adaptive Nonlinear Interaction Structures in High Dimensions," Journal of the American Statistical Association, American Statistical Association, vol. 105(492), pages 1541-1553.
    6. Smith, Michael & Kohn, Robert, 1996. "Nonparametric regression using Bayesian variable selection," Journal of Econometrics, Elsevier, vol. 75(2), pages 317-343, December.
    7. Belitz, Christiane & Lang, Stefan, 2008. "Simultaneous selection of variables and smoothing parameters in structured additive regression models," Computational Statistics & Data Analysis, Elsevier, vol. 53(1), pages 61-81, September.
    8. Pradeep Ravikumar & John Lafferty & Han Liu & Larry Wasserman, 2009. "Sparse additive models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 71(5), pages 1009-1030, November.
    9. Fan J. & Li R., 2001. "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1348-1360, December.
    10. Cottet, Remy & Kohn, Robert J. & Nott, David J., 2008. "Variable Selection and Model Averaging in Semiparametric Overdispersed Generalized Linear Models," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 661-671, June.
    11. Park, Trevor & Casella, George, 2008. "The Bayesian Lasso," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 681-686, June.
    12. Nicholas G. Polson & James G. Scott, 2012. "Local shrinkage rules, Lévy processes and regularized regression," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 74(2), pages 287-311, March.
    13. Gerhard Tutz & Harald Binder, 2006. "Generalized Additive Modeling with Implicit Variable Selection by Likelihood-Based Boosting," Biometrics, The International Biometric Society, vol. 62(4), pages 961-971, December.
    14. Marra, Giampiero & Wood, Simon N., 2011. "Practical variable selection for generalized additive models," Computational Statistics & Data Analysis, Elsevier, vol. 55(7), pages 2372-2387, July.
    15. Zhang, Hao Helen & Cheng, Guang & Liu, Yufeng, 2011. "Linear or Nonlinear? Automatic Structure Discovery for Partially Linear Models," Journal of the American Statistical Association, American Statistical Association, vol. 106(495), pages 1099-1112.
    16. Thomas Kneib & Susanne Konrath & Ludwig Fahrmeir, 2011. "High dimensional structured additive regression models: Bayesian regularization, smoothing and predictive performance," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 60(1), pages 51-70, January.
    17. Avalos, Marta & Grandvalet, Yves & Ambroise, Christophe, 2007. "Parsimonious additive models," Computational Statistics & Data Analysis, Elsevier, vol. 51(6), pages 2851-2870, March.
    18. Fahrmeir, Ludwig & Kneib, Thomas, 2011. "Bayesian Smoothing and Regression for Longitudinal, Spatial and Event History Data," OUP Catalogue, Oxford University Press, number 9780199533022.
    19. Ming Yuan & Yi Lin, 2006. "Model selection and estimation in regression with grouped variables," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 68(1), pages 49-67, February.
    20. Sally Wood & Robert Kohn & Tom Shively & Wenxin Jiang, 2002. "Model selection in spline nonparametric regression," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(1), pages 119-139, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hirofumi Michimae & Takeshi Emura, 2022. "Bayesian ridge estimators based on copula-based joint prior distributions for regression coefficients," Computational Statistics, Springer, vol. 37(5), pages 2741-2769, November.
    2. Ngandu Balekelayi & Solomon Tesfamariam, 2020. "Geoadditive Quantile Regression Model for Sewer Pipes Deterioration Using Boosting Optimization Algorithm," Sustainability, MDPI, vol. 12(20), pages 1-24, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Diego Vidaurre & Concha Bielza & Pedro Larrañaga, 2013. "A Survey of L1 Regression," International Statistical Review, International Statistical Institute, vol. 81(3), pages 361-387, December.
    2. McKay Curtis, S. & Banerjee, Sayantan & Ghosal, Subhashis, 2014. "Fast Bayesian model assessment for nonparametric additive regression," Computational Statistics & Data Analysis, Elsevier, vol. 71(C), pages 347-358.
    3. Umberto Amato & Anestis Antoniadis & Italia De Feis, 2016. "Additive model selection," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 25(4), pages 519-564, November.
    4. Du, Pang & Cheng, Guang & Liang, Hua, 2012. "Semiparametric regression models with additive nonparametric components and high dimensional parametric components," Computational Statistics & Data Analysis, Elsevier, vol. 56(6), pages 2006-2017.
    5. Tutz, Gerhard & Pößnecker, Wolfgang & Uhlmann, Lorenz, 2015. "Variable selection in general multinomial logit models," Computational Statistics & Data Analysis, Elsevier, vol. 82(C), pages 207-222.
    6. Marra, Giampiero & Wood, Simon N., 2011. "Practical variable selection for generalized additive models," Computational Statistics & Data Analysis, Elsevier, vol. 55(7), pages 2372-2387, July.
    7. Yoshida, Takuma, 2018. "Semiparametric method for model structure discovery in additive regression models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 124-136.
    8. Young Joo Yoon & Cheolwoo Park & Erik Hofmeister & Sangwook Kang, 2012. "Group variable selection in cardiopulmonary cerebral resuscitation data for veterinary patients," Journal of Applied Statistics, Taylor & Francis Journals, vol. 39(7), pages 1605-1621, January.
    9. Stefan Lang & Nikolaus Umlauf & Peter Wechselberger & Kenneth Harttgen & Thomas Kneib, 2012. "Multilevel structured additive regression," Working Papers 2012-07, Faculty of Economics and Statistics, Universität Innsbruck.
    10. G. Yi & J. Q. Shi & T. Choi, 2011. "Penalized Gaussian Process Regression and Classification for High-Dimensional Nonlinear Data," Biometrics, The International Biometric Society, vol. 67(4), pages 1285-1294, December.
    11. Kaixu Yang & Tapabrata Maiti, 2022. "Ultrahigh‐dimensional generalized additive model: Unified theory and methods," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 49(3), pages 917-942, September.
    12. Alhamzawi, Rahim, 2016. "Bayesian model selection in ordinal quantile regression," Computational Statistics & Data Analysis, Elsevier, vol. 103(C), pages 68-78.
    13. repec:hum:wpaper:sfb649dp2012-061 is not listed on IDEAS
    14. Honda, Toshio & Härdle, Wolfgang Karl, 2012. "Variable selection in Cox regression models with varying coefficients," SFB 649 Discussion Papers 2012-061, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    15. Bernardi, Mauro & Costola, Michele, 2019. "High-dimensional sparse financial networks through a regularised regression model," SAFE Working Paper Series 244, Leibniz Institute for Financial Research SAFE.
    16. Loann David Denis Desboulets, 2018. "A Review on Variable Selection in Regression Analysis," Econometrics, MDPI, vol. 6(4), pages 1-27, November.
    17. Zanhua Yin, 2020. "Variable selection for sparse logistic regression," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 83(7), pages 821-836, October.
    18. Juan Armando Torres Munguía, 2018. "What is behind homicide gender gaps in Mexico? A spatial semiparametric approach," Ibero America Institute for Econ. Research (IAI) Discussion Papers 236, Ibero-America Institute for Economic Research.
    19. Ricardo P. Masini & Marcelo C. Medeiros & Eduardo F. Mendes, 2023. "Machine learning advances for time series forecasting," Journal of Economic Surveys, Wiley Blackwell, vol. 37(1), pages 76-111, February.
    20. Lichun Wang & Yuan You & Heng Lian, 2015. "Convergence and sparsity of Lasso and group Lasso in high-dimensional generalized linear models," Statistical Papers, Springer, vol. 56(3), pages 819-828, August.
    21. Pei Wang & Shunjie Chen & Sijia Yang, 2022. "Recent Advances on Penalized Regression Models for Biological Data," Mathematics, MDPI, vol. 10(19), pages 1-24, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:alstar:v:97:y:2013:i:4:p:349-385. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.