IDEAS home Printed from https://ideas.repec.org/a/oup/biomet/v105y2018i3p517-527..html
   My bibliography  Save this article

When is the first spurious variable selected by sequential regression procedures?

Author

Listed:
  • Weijie J Su

Abstract

SummaryApplied statisticians use sequential regression procedures to rank explanatory variables and, in settings of low correlations between variables and strong true effect sizes, expect that variables at the top of this ranking are truly relevant to the response. In a regime of certain sparsity levels, however, we show that the lasso, forward stepwise regression, and least angle regression include the first spurious variable unexpectedly early. We derive a sharp prediction of the rank of the first spurious variable for these three procedures, demonstrating that it occurs earlier and earlier as the regression coefficients become denser. This phenomenon persists for statistically independent Gaussian random designs and arbitrarily large true effects. We gain insight by identifying the underlying cause and then introduce a simple visualization tool termed the double-ranking diagram to improve on these methods. We obtain the first result establishing the exact equivalence between the lasso and least angle regression in the early stages of solution paths beyond orthogonal designs. This equivalence implies that many important model selection results concerning the lasso can be carried over to least angle regression.

Suggested Citation

  • Weijie J Su, 2018. "When is the first spurious variable selected by sequential regression procedures?," Biometrika, Biometrika Trust, vol. 105(3), pages 517-527.
  • Handle: RePEc:oup:biomet:v:105:y:2018:i:3:p:517-527.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/biomet/asy032
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Max Grazier G'Sell & Stefan Wager & Alexandra Chouldechova & Robert Tibshirani, 2016. "Sequential selection procedures and false discovery rate control," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(2), pages 423-444, March.
    2. Xiangyu Wang & Chenlei Leng, 2016. "High dimensional ordinary least squares projection for screening variables," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 589-611, June.
    3. Ryan J. Tibshirani & Jonathan Taylor & Richard Lockhart & Robert Tibshirani, 2016. "Exact Post-Selection Inference for Sequential Regression Procedures," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(514), pages 600-620, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Markus Pelger & Jiacheng Zou, 2022. "Inference for Large Panel Data with Many Covariates," Papers 2301.00292, arXiv.org, revised Mar 2023.
    2. Gregory Vaughan & Robert Aseltine & Kun Chen & Jun Yan, 2017. "Stagewise generalized estimating equations with grouped variables," Biometrics, The International Biometric Society, vol. 73(4), pages 1332-1342, December.
    3. Liang, Weijuan & Zhang, Qingzhao & Ma, Shuangge, 2024. "Hierarchical false discovery rate control for high-dimensional survival analysis with interactions," Computational Statistics & Data Analysis, Elsevier, vol. 192(C).
    4. X. Jessie Jeng & Huimin Peng & Wenbin Lu, 2021. "Model Selection With Mixed Variables on the Lasso Path," Sankhya B: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 83(1), pages 170-184, May.
    5. Claude Renaux & Laura Buzdugan & Markus Kalisch & Peter Bühlmann, 2020. "Rejoinder on: Hierarchical inference for genome-wide association studies: a view on methodology with software," Computational Statistics, Springer, vol. 35(1), pages 59-67, March.
    6. Xu Zhao & Zhongxian Zhang & Weihu Cheng & Pengyue Zhang, 2019. "A New Parameter Estimator for the Generalized Pareto Distribution under the Peaks over Threshold Framework," Mathematics, MDPI, vol. 7(5), pages 1-18, May.
    7. Jelle J Goeman & Aldo Solari, 2024. "On selection and conditioning in multiple testing and selective inference," Biometrika, Biometrika Trust, vol. 111(2), pages 393-416.
    8. Jun Lu & Dan Wang & Qinqin Hu, 2022. "Interaction screening via canonical correlation," Computational Statistics, Springer, vol. 37(5), pages 2637-2670, November.
    9. Michael J. Weir & Thomas W. Sproul, 2019. "Identifying Drivers of Genetically Modified Seafood Demand: Evidence from a Choice Experiment," Sustainability, MDPI, vol. 11(14), pages 1-21, July.
    10. Zhao, Bangxin & Liu, Xin & He, Wenqing & Yi, Grace Y., 2021. "Dynamic tilted current correlation for high dimensional variable screening," Journal of Multivariate Analysis, Elsevier, vol. 182(C).
    11. Damian Kozbur, 2020. "Analysis of Testing‐Based Forward Model Selection," Econometrica, Econometric Society, vol. 88(5), pages 2147-2173, September.
    12. Liu, Jingyuan & Sun, Ao & Ke, Yuan, 2024. "A generalized knockoff procedure for FDR control in structural change detection," Journal of Econometrics, Elsevier, vol. 239(2).
    13. Haofeng Wang & Hongxia Jin & Xuejun Jiang & Jingzhi Li, 2022. "Model Selection for High Dimensional Nonparametric Additive Models via Ridge Estimation," Mathematics, MDPI, vol. 10(23), pages 1-22, December.
    14. Gong, Siliang & Zhang, Kai & Liu, Yufeng, 2018. "Efficient test-based variable selection for high-dimensional linear models," Journal of Multivariate Analysis, Elsevier, vol. 166(C), pages 17-31.
    15. Qiu, Debin & Ahn, Jeongyoun, 2020. "Grouped variable screening for ultra-high dimensional data for linear model," Computational Statistics & Data Analysis, Elsevier, vol. 144(C).
    16. The Tien Mai, 2023. "Reliable Genetic Correlation Estimation via Multiple Sample Splitting and Smoothing," Mathematics, MDPI, vol. 11(9), pages 1-13, May.
    17. Rügamer, David & Baumann, Philipp F.M. & Greven, Sonja, 2022. "Selective inference for additive and linear mixed models," Computational Statistics & Data Analysis, Elsevier, vol. 167(C).
    18. Vitor Pessoa Colombo & Jérôme Chenal & Brama Koné & Martí Bosch & Jürg Utzinger, 2022. "Using Open-Access Data to Explore Relations between Urban Landscapes and Diarrhoeal Diseases in Côte d’Ivoire," IJERPH, MDPI, vol. 19(13), pages 1-20, June.
    19. He, Kevin & Kang, Jian & Hong, Hyokyoung G. & Zhu, Ji & Li, Yanming & Lin, Huazhen & Xu, Han & Li, Yi, 2019. "Covariance-insured screening," Computational Statistics & Data Analysis, Elsevier, vol. 132(C), pages 100-114.
    20. Jixiong Wang & Ashish Patel & James M.S. Wason & Paul J. Newcombe, 2022. "Two‐stage penalized regression screening to detect biomarker–treatment interactions in randomized clinical trials," Biometrics, The International Biometric Society, vol. 78(1), pages 141-150, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:biomet:v:105:y:2018:i:3:p:517-527.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/biomet .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.