IDEAS home Printed from https://ideas.repec.org/a/spr/advdac/v13y2019i4d10.1007_s11634-018-00351-6.html
   My bibliography  Save this article

Exploration of the variability of variable selection based on distances between bootstrap sample results

Author

Listed:
  • Christian Hennig

    (University of Bologna
    University College London)

  • Willi Sauerbrei

    (University of Bologna
    Faculty of Medicine and Medical Center - University of Freiburg)

Abstract

It is well known that variable selection in multiple regression can be unstable and that the model uncertainty can be considerable. The model uncertainty can be quantified and explored by bootstrap resampling, see Sauerbrei et al. (Biom J 57:531–555, 2015). Here approaches are introduced that use the results of bootstrap replications of the variable selection process to obtain more detailed information about the data. Analyses will be based on dissimilarities between the results of the analyses of different bootstrap samples. Dissimilarities are computed between the vector of predictions, and between the sets of selected variables. The dissimilarities are used to map the models by multidimensional scaling, to cluster them, and to construct heatplots. Clusters can point to different interpretations of the data that could arise from different selections of variables supported by different bootstrap samples. A new measure of variable selection instability is also defined. The methodology can be applied to various regression models, estimators, and variable selection methods. It will be illustrated by three real data examples, using linear regression and a Cox proportional hazards model, and model selection by AIC and BIC.

Suggested Citation

  • Christian Hennig & Willi Sauerbrei, 2019. "Exploration of the variability of variable selection based on distances between bootstrap sample results," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 13(4), pages 933-963, December.
    • Handle: RePEc:spr:advdac:v:13:y:2019:i:4:d:10.1007_s11634-018-00351-6
    DOI: 10.1007/s11634-018-00351-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11634-018-00351-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11634-018-00351-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Riani, Marco & Atkinson, Anthony C., 2010. "Robust model selection with flexible trimming," Computational Statistics & Data Analysis, Elsevier, vol. 54(12), pages 3300-3312, December.
    2. Bradley Efron, 2014. "Estimation and Accuracy After Model Selection," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(507), pages 991-1007, September.
    3. Buchholz, Anika & Hollander, Norbert & Sauerbrei, Willi, 2008. "On properties of predictors derived with a two-step bootstrap model averaging approach--A simulation study in the linear regression model," Computational Statistics & Data Analysis, Elsevier, vol. 52(5), pages 2778-2793, January.
    4. Chris T. Volinsky & Adrian E. Raftery, 2000. "Bayesian Information Criterion for Censored Survival Models," Biometrics, The International Biometric Society, vol. 56(1), pages 256-262, March.
    5. de Leeuw, Jan & Mair, Patrick, 2009. "Multidimensional Scaling Using Majorization: SMACOF in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 31(i03).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wang, Lu & Wu, Rui & Ma, WeiChun & Xu, Weiju, 2023. "Examining the volatility of soybean market in the MIDAS framework: The importance of bagging-based weather information," International Review of Financial Analysis, Elsevier, vol. 89(C).
    2. Qin, Yichen & Wang, Linna & Li, Yang & Li, Rong, 2023. "Visualization and assessment of model selection uncertainty," Computational Statistics & Data Analysis, Elsevier, vol. 178(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Malloy, Elizabeth J. & Spiegelman, Donna & Eisen, Ellen A., 2009. "Comparing measures of model selection for penalized splines in Cox models," Computational Statistics & Data Analysis, Elsevier, vol. 53(7), pages 2605-2616, May.
    2. Miao Han & Liuquan Sun & Yutao Liu & Jun Zhu, 2018. "Joint analysis of recurrent event data with additive–multiplicative hazards model for the terminal event time," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 81(5), pages 523-547, July.
    3. Wu Wang & Xuming He & Zhongyi Zhu, 2020. "Statistical inference for multiple change‐point models," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(4), pages 1149-1170, December.
    4. Gonzalo García-Donato & María Eugenia Castellanos & Alicia Quirós, 2021. "Bayesian Variable Selection with Applications in Health Sciences," Mathematics, MDPI, vol. 9(3), pages 1-16, January.
    5. Walesiak Marek & Dudek Andrzej, 2017. "Selecting the Optimal Multidimensional Scaling Procedure for Metric Data With R Environment," Statistics in Transition New Series, Statistics Poland, vol. 18(3), pages 521-540, September.
    6. Lee Kyu Ha & Chakraborty Sounak & Sun Jianguo, 2011. "Bayesian Variable Selection in Semiparametric Proportional Hazards Model for High Dimensional Survival Data," The International Journal of Biostatistics, De Gruyter, vol. 7(1), pages 1-32, April.
    7. John Copas & Shinto Eguchi, 2020. "Strong model dependence in statistical analysis: goodness of fit is not enough for model choice," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 72(2), pages 329-352, April.
    8. Anwar, Sajid & Sun, Sizhong, 2012. "Trade liberalisation, market competition and wage inequality in China's manufacturing sector," Economic Modelling, Elsevier, vol. 29(4), pages 1268-1277.
    9. Dzemyda, Gintautas & Sabaliauskas, Martynas, 2021. "Geometric multidimensional scaling: A new approach for data dimensionality reduction," Applied Mathematics and Computation, Elsevier, vol. 409(C).
    10. Subhadeep & Mukhopadhyay, 2022. "Modelplasticity and Abductive Decision Making," Papers 2203.03040, arXiv.org, revised Mar 2023.
    11. Susan Athey & Julie Tibshirani & Stefan Wager, 2016. "Generalized Random Forests," Papers 1610.01271, arXiv.org, revised Apr 2018.
    12. Min Zhang & Marie Davidian, 2008. "“Smooth” Semiparametric Regression Analysis for Arbitrarily Censored Time-to-Event Data," Biometrics, The International Biometric Society, vol. 64(2), pages 567-576, June.
    13. Farrell, Max H., 2015. "Robust inference on average treatment effects with possibly more covariates than observations," Journal of Econometrics, Elsevier, vol. 189(1), pages 1-23.
    14. repec:jss:jstsof:34:i10 is not listed on IDEAS
    15. Lenard Lieb & Stephan Smeekes, 2017. "Inference for Impulse Responses under Model Uncertainty," Papers 1709.09583, arXiv.org, revised Oct 2019.
    16. Liang, Hua & Zou, Guohua, 2008. "Improved AIC selection strategy for survival analysis," Computational Statistics & Data Analysis, Elsevier, vol. 52(5), pages 2538-2548, January.
    17. Francesca Torti & Aldo Corbellini & Anthony C. Atkinson, 2021. "fsdaSAS: A Package for Robust Regression for Very Large Datasets Including the Batch Forward Search," Stats, MDPI, vol. 4(2), pages 1-21, April.
    18. Long Mark C. & Rooklyn Jordan, 2024. "Regression(s) discontinuity: Using bootstrap aggregation to yield estimates of RD treatment effects," Journal of Causal Inference, De Gruyter, vol. 12(1), pages 1-21, January.
    19. D. J. Eck & R. D. Cook, 2017. "Weighted envelope estimation to handle variability in model selection," Biometrika, Biometrika Trust, vol. 104(3), pages 743-749.
    20. Arlen Dean & Amirhossein Meisami & Henry Lam & Mark P. Van Oyen & Christopher Stromblad & Nick Kastango, 2022. "Quantile regression forests for individualized surgery scheduling," Health Care Management Science, Springer, vol. 25(4), pages 682-709, December.
    21. Chou, Jui-Sheng & Chien, Ya-Ling & Nguyen, Ngoc-Mai & Truong, Dinh-Nhat, 2018. "Pricing policy of floating ticket fare for riding high speed rail based on time-space compression," Transport Policy, Elsevier, vol. 69(C), pages 179-192.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:advdac:v:13:y:2019:i:4:d:10.1007_s11634-018-00351-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.