IDEAS home Printed from https://ideas.repec.org/a/spr/compst/v38y2023i2d10.1007_s00180-022-01249-w.html
   My bibliography  Save this article

A global two-stage algorithm for non-convex penalized high-dimensional linear regression problems

Author

Listed:
  • Peili Li

    (East China Normal University)

  • Min Liu

    (Wuhan University)

  • Zhou Yu

    (East China Normal University)

Abstract

By the asymptotic oracle property, non-convex penalties represented by minimax concave penalty (MCP) and smoothly clipped absolute deviation (SCAD) have attracted much attentions in high-dimensional data analysis, and have been widely used in signal processing, image restoration, matrix estimation, etc. However, in view of their non-convex and non-smooth characteristics, they are computationally challenging. Almost all existing algorithms converge locally, and the proper selection of initial values is crucial. Therefore, in actual operation, they often combine a warm-starting technique to meet the rigid requirement that the initial value must be sufficiently close to the optimal solution of the corresponding problem. In this paper, based on the DC (difference of convex functions) property of MCP and SCAD penalties, we aim to design a global two-stage algorithm for the high-dimensional least squares linear regression problems. A key idea for making the proposed algorithm to be efficient is to use the primal dual active set with continuation (PDASC) method to solve the corresponding sub-problems. Theoretically, we not only prove the global convergence of the proposed algorithm, but also verify that the generated iterative sequence converges to a d-stationary point. In terms of computational performance, the abundant research of simulation and real data show that the algorithm in this paper is superior to the latest SSN method and the classic coordinate descent (CD) algorithm for solving non-convex penalized high-dimensional linear regression problems.

Suggested Citation

  • Peili Li & Min Liu & Zhou Yu, 2023. "A global two-stage algorithm for non-convex penalized high-dimensional linear regression problems," Computational Statistics, Springer, vol. 38(2), pages 871-898, June.
    • Handle: RePEc:spr:compst:v:38:y:2023:i:2:d:10.1007_s00180-022-01249-w
    DOI: 10.1007/s00180-022-01249-w
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00180-022-01249-w
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00180-022-01249-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mazumder, Rahul & Friedman, Jerome H. & Hastie, Trevor, 2011. "SparseNet: Coordinate Descent With Nonconvex Penalties," Journal of the American Statistical Association, American Statistical Association, vol. 106(495), pages 1125-1138.
    2. Fan J. & Li R., 2001. "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1348-1360, December.
    3. Le Thi, H.A. & Pham Dinh, T. & Le, H.M. & Vo, X.T., 2015. "DC approximation approaches for sparse optimization," European Journal of Operational Research, Elsevier, vol. 244(1), pages 26-46.
    4. Jong-Shi Pang & Meisam Razaviyayn & Alberth Alvarado, 2017. "Computing B-Stationary Points of Nonsmooth DC Programs," Mathematics of Operations Research, INFORMS, vol. 42(1), pages 95-118, January.
    5. Shan Luo & Zehua Chen, 2014. "Sequential Lasso Cum EBIC for Feature Selection With Ultra-High Dimensional Feature Space," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(507), pages 1229-1240, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Miju Ahn, 2020. "Consistency bounds and support recovery of d-stationary solutions of sparse sample average approximations," Journal of Global Optimization, Springer, vol. 78(3), pages 397-422, November.
    2. Margherita Giuzio, 2017. "Genetic algorithm versus classical methods in sparse index tracking," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 40(1), pages 243-256, November.
    3. Min Tao & Jiang-Ning Li, 2023. "Error Bound and Isocost Imply Linear Convergence of DCA-Based Algorithms to D-Stationarity," Journal of Optimization Theory and Applications, Springer, vol. 197(1), pages 205-232, April.
    4. Bartosz Uniejewski, 2024. "Regularization for electricity price forecasting," Operations Research and Decisions, Wroclaw University of Science and Technology, Faculty of Management, vol. 34(3), pages 267-286.
    5. Li, Peili & Jiao, Yuling & Lu, Xiliang & Kang, Lican, 2022. "A data-driven line search rule for support recovery in high-dimensional data analysis," Computational Statistics & Data Analysis, Elsevier, vol. 174(C).
    6. Jin, Shaobo & Moustaki, Irini & Yang-Wallentin, Fan, 2018. "Approximated penalized maximum likelihood for exploratory factor analysis: an orthogonal case," LSE Research Online Documents on Economics 88118, London School of Economics and Political Science, LSE Library.
    7. Shaobo Jin & Irini Moustaki & Fan Yang-Wallentin, 2018. "Approximated Penalized Maximum Likelihood for Exploratory Factor Analysis: An Orthogonal Case," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 628-649, September.
    8. Honda, Toshio & 本田, 敏雄 & Lin, Chien-Tong, 2022. "Forward variable selection for ultra-high dimensional quantile regression models," Discussion Papers 2021-02, Graduate School of Economics, Hitotsubashi University.
    9. Anda Tang & Pei Quan & Lingfeng Niu & Yong Shi, 2022. "A Survey for Sparse Regularization Based Compression Methods," Annals of Data Science, Springer, vol. 9(4), pages 695-722, August.
    10. Ben-Ameur, Walid & Neto, José, 2022. "New bounds for subset selection from conic relaxations," European Journal of Operational Research, Elsevier, vol. 298(2), pages 425-438.
    11. Rahul Ghosal & Arnab Maity & Timothy Clark & Stefano B. Longo, 2020. "Variable selection in functional linear concurrent regression," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(3), pages 565-587, June.
    12. Po-Hsien Huang & Hung Chen & Li-Jen Weng, 2017. "A Penalized Likelihood Method for Structural Equation Modeling," Psychometrika, Springer;The Psychometric Society, vol. 82(2), pages 329-354, June.
    13. Hoai An Le Thi & Manh Cuong Nguyen, 2017. "DCA based algorithms for feature selection in multi-class support vector machine," Annals of Operations Research, Springer, vol. 249(1), pages 273-300, February.
    14. Xiang Zhang & Yichao Wu & Lan Wang & Runze Li, 2016. "Variable selection for support vector machines in moderately high dimensions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(1), pages 53-76, January.
    15. Hong, Hyokyoung G. & Zheng, Qi & Li, Yi, 2019. "Forward regression for Cox models with high-dimensional covariates," Journal of Multivariate Analysis, Elsevier, vol. 173(C), pages 268-290.
    16. VÁZQUEZ-ALCOCER, Alan & SCHOEN, Eric D. & GOOS, Peter, 2018. "A mixed integer optimization approach for model selection in screening experiments," Working Papers 2018007, University of Antwerp, Faculty of Business and Economics.
    17. Hu, Jianwei & Chai, Hao, 2013. "Adjusted regularized estimation in the accelerated failure time model with high dimensional covariates," Journal of Multivariate Analysis, Elsevier, vol. 122(C), pages 96-114.
    18. Fan Wu & Wei Bian, 2020. "Accelerated iterative hard thresholding algorithm for $$l_0$$l0 regularized regression problem," Journal of Global Optimization, Springer, vol. 76(4), pages 819-840, April.
    19. Minh Pham & Xiaodong Lin & Andrzej Ruszczyński & Yu Du, 2021. "An outer–inner linearization method for non-convex and nondifferentiable composite regularization problems," Journal of Global Optimization, Springer, vol. 81(1), pages 179-202, September.
    20. Yingying Fan & Jinchi Lv, 2014. "Asymptotic properties for combined L1 and concave regularization," Biometrika, Biometrika Trust, vol. 101(1), pages 57-70.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:compst:v:38:y:2023:i:2:d:10.1007_s00180-022-01249-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.