IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v75y2014icp190-202.html
   My bibliography  Save this article

Choice of generalized linear mixed models using predictive crossvalidation

Author

Listed:
  • Braun, Julia
  • Sabanés Bové, Daniel
  • Held, Leonhard

Abstract

The choice of generalized linear mixed models is difficult, because it involves the selection of both fixed and random effects. Classical criteria like Akaike’s information criterion (AIC) are often not suitable for the latter task, and others which are useful in linear mixed models are difficult to extend to the generalized case, especially for overdispersed data. A predictive leave-one-out crossvalidation approach is suggested that can be applied for choosing both fixed and random effects, even in models with overdispersion, and is based on proper scoring rules. An attractive feature of this approach is the fact that the model has to be fitted just once to the data set, which makes computations fast and convenient. As the calculation of the leave-one-out predictive distribution is not possible analytically, it is shown how an iteratively weighted least squares algorithm combined with some analytic approximations can be used for this task. A simulation study and two applications of the methodology to binary and count data are provided, as well as comparisons with two other methods.

Suggested Citation

  • Braun, Julia & Sabanés Bové, Daniel & Held, Leonhard, 2014. "Choice of generalized linear mixed models using predictive crossvalidation," Computational Statistics & Data Analysis, Elsevier, vol. 75(C), pages 190-202.
    • Handle: RePEc:eee:csdana:v:75:y:2014:i:c:p:190-202
    DOI: 10.1016/j.csda.2014.02.008
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947314000486
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2014.02.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yu, Dalei & Yau, Kelvin K.W., 2012. "Conditional Akaike information criterion for generalized linear mixed models," Computational Statistics & Data Analysis, Elsevier, vol. 56(3), pages 629-644.
    2. M. C. Donohue & R. Overholser & R. Xu & F. Vaida, 2011. "Conditional Akaike information under generalized linear and proportional hazards mixed models," Biometrika, Biometrika Trust, vol. 98(3), pages 685-700.
    3. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    4. Hua Liang & Hulin Wu & Guohua Zou, 2008. "A note on conditional aic for linear mixed-effects models," Biometrika, Biometrika Trust, vol. 95(3), pages 773-778.
    5. R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
    6. Claudia Czado & Tilmann Gneiting & Leonhard Held, 2009. "Predictive Model Assessment for Count Data," Biometrics, The International Biometric Society, vol. 65(4), pages 1254-1261, December.
    7. L. Held & K. Rufibach & F. Balabdaoui, 2010. "A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions," Biometrics, The International Biometric Society, vol. 66(4), pages 1295-1305, December.
    8. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    9. Claeskens,Gerda & Hjort,Nils Lid, 2008. "Model Selection and Model Averaging," Cambridge Books, Cambridge University Press, number 9780521852258.
    10. Sonja Greven & Thomas Kneib, 2010. "On the behaviour of marginal and conditional AIC in linear mixed models," Biometrika, Biometrika Trust, vol. 97(4), pages 773-789.
    11. Bates, Douglas M. & DebRoy, Saikat, 2004. "Linear mixed models and penalized least squares," Journal of Multivariate Analysis, Elsevier, vol. 91(1), pages 1-17, October.
    12. Florin Vaida & Suzette Blanchard, 2005. "Conditional Akaike information for mixed-effects models," Biometrika, Biometrika Trust, vol. 92(2), pages 351-370, June.
    13. Bo Cai & David B. Dunson, 2006. "Bayesian Covariance Selection in Generalized Linear Mixed Models," Biometrics, The International Biometric Society, vol. 62(2), pages 446-457, June.
    14. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Rejoinder on: Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 256-264, August.
    15. Ciprian M. Crainiceanu & David Ruppert, 2004. "Likelihood ratio tests in linear mixed models with one variance component," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 66(1), pages 165-185, February.
    16. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 211-235, August.
    17. Rainer Winkelmann, 2008. "Econometric Analysis of Count Data," Springer Books, Springer, edition 0, number 978-3-540-78389-3, June.
    18. Julia Braun & Leonhard Held & Bruno Ledergerber, 2012. "Predictive Cross-validation for the Choice of Linear Mixed-Effects Models with Application to Data from the Swiss HIV Cohort Study," Biometrics, The International Biometric Society, vol. 68(1), pages 53-61, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wei Wei & Leonhard Held, 2014. "Calibration tests for count data," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 23(4), pages 787-805, December.
    2. Julia Braun & Leonhard Held & Bruno Ledergerber, 2012. "Predictive Cross-validation for the Choice of Linear Mixed-Effects Models with Application to Data from the Swiss HIV Cohort Study," Biometrics, The International Biometric Society, vol. 68(1), pages 53-61, March.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Simona Buscemi & Antonella Plaia, 2020. "Model selection in linear mixed-effect models," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 104(4), pages 529-575, December.
    5. Malte Knuppel & Fabian Kruger & Marc-Oliver Pohle, 2022. "Score-based calibration testing for multivariate forecast distributions," Papers 2211.16362, arXiv.org, revised Dec 2023.
    6. L. Held & K. Rufibach & F. Balabdaoui, 2010. "A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions," Biometrics, The International Biometric Society, vol. 66(4), pages 1295-1305, December.
    7. Yu, Dalei & Zhang, Xinyu & Yau, Kelvin K.W., 2013. "Information based model selection criteria for generalized linear mixed models with unknown variance component parameters," Journal of Multivariate Analysis, Elsevier, vol. 116(C), pages 245-262.
    8. Warne, Anders, 2023. "DSGE model forecasting: rational expectations vs. adaptive learning," Working Paper Series 2768, European Central Bank.
    9. Jenny Brynjarsdottir & Jonathan Hobbs & Amy Braverman & Lukas Mandrake, 2018. "Optimal Estimation Versus MCMC for $$\mathrm{{CO}}_{2}$$ CO 2 Retrievals," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 23(2), pages 297-316, June.
    10. Kawakubo, Yuki & Kubokawa, Tatsuya, 2014. "Modified conditional AIC in linear mixed models," Journal of Multivariate Analysis, Elsevier, vol. 129(C), pages 44-56.
    11. Fabian Krüger & Sebastian Lerch & Thordis Thorarinsdottir & Tilmann Gneiting, 2021. "Predictive Inference Based on Markov Chain Monte Carlo Output," International Statistical Review, International Statistical Institute, vol. 89(2), pages 274-301, August.
    12. Chan, Moon-tong & Yu, Dalei & Yau, Kelvin K.W., 2015. "Multilevel cumulative logistic regression model with random effects: Application to British social attitudes panel survey data," Computational Statistics & Data Analysis, Elsevier, vol. 88(C), pages 173-186.
    13. Grothe, Oliver & Kächele, Fabian & Krüger, Fabian, 2023. "From point forecasts to multivariate probabilistic forecasts: The Schaake shuffle for day-ahead electricity price forecasting," Energy Economics, Elsevier, vol. 120(C).
    14. Wei, Wei & Balabdaoui, Fadoua & Held, Leonhard, 2017. "Calibration tests for multivariate Gaussian forecasts," Journal of Multivariate Analysis, Elsevier, vol. 154(C), pages 216-233.
    15. Thordis L. Thorarinsdottir & Tilmann Gneiting, 2010. "Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(2), pages 371-388, April.
    16. Yu, Dalei & Yau, Kelvin K.W., 2012. "Conditional Akaike information criterion for generalized linear mixed models," Computational Statistics & Data Analysis, Elsevier, vol. 56(3), pages 629-644.
    17. Gensler, André & Sick, Bernhard & Vogt, Stephan, 2018. "A review of uncertainty representations and metaverification of uncertainty assessment techniques for renewable energies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 96(C), pages 352-379.
    18. Overholser, Rosanna & Xu, Ronghui, 2014. "Effective degrees of freedom and its application to conditional AIC for linear mixed-effects models with correlated error structures," Journal of Multivariate Analysis, Elsevier, vol. 132(C), pages 160-170.
    19. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    20. Kruse, René-Marcel & Silbersdorff, Alexander & Säfken, Benjamin, 2022. "Model averaging for linear mixed models via augmented Lagrangian," Computational Statistics & Data Analysis, Elsevier, vol. 167(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:75:y:2014:i:c:p:190-202. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.