IDEAS home Printed from https://ideas.repec.org/a/eee/stapro/v180y2022ics016771522100208x.html
   My bibliography  Save this article

Learning sparse deep neural networks with a spike-and-slab prior

Author

Listed:
  • Sun, Yan
  • Song, Qifan
  • Liang, Faming

Abstract

Deep learning has achieved great successes in many machine learning tasks. However, the deep neural networks (DNNs) are often severely over-parameterized, making them computationally expensive, memory intensive, less interpretable and mis-calibrated. We study sparse DNNs under the Bayesian framework: we establish posterior consistency and structure selection consistency for Bayesian DNNs with a spike-and-slab prior, and illustrate their performance using examples on high-dimensional nonlinear variable selection, large network compression and model calibration. Our numerical results indicate that sparsity is essential for improving the prediction accuracy and calibration of the DNN.

Suggested Citation

  • Sun, Yan & Song, Qifan & Liang, Faming, 2022. "Learning sparse deep neural networks with a spike-and-slab prior," Statistics & Probability Letters, Elsevier, vol. 180(C).
    • Handle: RePEc:eee:stapro:v:180:y:2022:i:c:s016771522100208x
    DOI: 10.1016/j.spl.2021.109246
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S016771522100208X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.spl.2021.109246?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Faming Liang & Qifan Song & Kai Yu, 2013. "Bayesian Subset Modeling for High-Dimensional Generalized Linear Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(502), pages 589-606, June.
    2. Faming Liang & Qizhai Li & Lei Zhou, 2018. "Bayesian Neural Networks for Selection of Drug Sensitive Genes," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 955-972, July.
    3. Faming Liang & Jian Zhang, 2008. "Estimating the false discovery rate using the stochastic approximation algorithm," Biometrika, Biometrika Trust, vol. 95(4), pages 961-977.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Faming Liang & Momiao Xiong, 2013. "Bayesian Detection of Causal Rare Variants under Posterior Consistency," PLOS ONE, Public Library of Science, vol. 8(7), pages 1-16, July.
    2. Guangbao Guo & Guoqi Qian & Lu Lin & Wei Shao, 2021. "Parallel inference for big data with the group Bayesian method," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 84(2), pages 225-243, February.
    3. Qifan Song & Guang Cheng, 2020. "Bayesian Fusion Estimation via t Shrinkage," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 82(2), pages 353-385, August.
    4. Zhang, Xin & Zhao, Junlong, 2024. "Group variable selection via group sparse neural network," Computational Statistics & Data Analysis, Elsevier, vol. 192(C).
    5. Shiqiang Jin & Gyuhyeong Goh, 2021. "Bayesian selection of best subsets via hybrid search," Computational Statistics, Springer, vol. 36(3), pages 1991-2007, September.
    6. Wang, Jia & Cai, Xizhen & Li, Runze, 2021. "Variable selection for partially linear models via Bayesian subset modeling with diffusing prior," Journal of Multivariate Analysis, Elsevier, vol. 183(C).
    7. Li, Cheng & Jiang, Wenxin, 2016. "On oracle property and asymptotic validity of Bayesian generalized method of moments," Journal of Multivariate Analysis, Elsevier, vol. 145(C), pages 132-147.
    8. Yi Liu & Veronika Ročková & Yuexi Wang, 2021. "Variable selection with ABC Bayesian forests," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(3), pages 453-481, July.
    9. Shi, Guiling & Lim, Chae Young & Maiti, Tapabrata, 2019. "Bayesian model selection for generalized linear models using non-local priors," Computational Statistics & Data Analysis, Elsevier, vol. 133(C), pages 285-296.
    10. David Rossell & Donatello Telesca, 2017. "Nonlocal Priors for High-Dimensional Estimation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(517), pages 254-265, January.
    11. Xueying Tang & Xiaofan Xu & Malay Ghosh & Prasenjit Ghosh, 2018. "Bayesian Variable Selection and Estimation Based on Global-Local Shrinkage Priors," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 80(2), pages 215-246, August.
    12. Runmin Shi & Faming Liang & Qifan Song & Ye Luo & Malay Ghosh, 2018. "A Blockwise Consistency Method for Parameter Estimation of Complex Models," Sankhya B: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 80(1), pages 179-223, December.
    13. Naveen Naidu Narisetty, 2020. "Discussion," International Statistical Review, International Statistical Institute, vol. 88(2), pages 330-334, August.
    14. Park, Seyoung & Kim, Hyunjin & Lee, Eun Ryung, 2023. "Regional quantile regression for multiple responses," Computational Statistics & Data Analysis, Elsevier, vol. 188(C).
    15. Qifan Song & Faming Liang, 2015. "High-Dimensional Variable Selection With Reciprocal L 1 -Regularization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1607-1620, December.
    16. Pan, Lanfeng & Li, Yehua & He, Kevin & Li, Yanming & Li, Yi, 2020. "Generalized linear mixed models with Gaussian mixture random effects: Inference and application," Journal of Multivariate Analysis, Elsevier, vol. 175(C).
    17. Ho-Hsiang Wu & Marco A. R. Ferreira & Mohamed Elkhouly & Tieming Ji, 2020. "Hyper Nonlocal Priors for Variable Selection in Generalized Linear Models," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 82(1), pages 147-185, February.
    18. Sierra A. Bainter & Thomas G. McCauley & Mahmoud M. Fahmy & Zachary T. Goodman & Lauren B. Kupis & J. Sunil Rao, 2023. "Comparing Bayesian Variable Selection to Lasso Approaches for Applications in Psychology," Psychometrika, Springer;The Psychometric Society, vol. 88(3), pages 1032-1055, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:stapro:v:180:y:2022:i:c:s016771522100208x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/622892/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.