IDEAS home Printed from https://ideas.repec.org/a/bla/istatr/v88y2020i2p302-320.html
   My bibliography  Save this article

Horseshoe Regularisation for Machine Learning in Complex and Deep Models

Author

Listed:
  • Anindya Bhadra
  • Jyotishka Datta
  • Yunfan Li
  • Nicholas Polson

Abstract

Since the advent of the horseshoe priors for regularisation, global–local shrinkage methods have proved to be a fertile ground for the development of Bayesian methodology in machine learning, specifically for high‐dimensional regression and classification problems. They have achieved remarkable success in computation and enjoy strong theoretical support. Most of the existing literature has focused on the linear Gaussian case; for which systematic surveys are available. The purpose of the current article is to demonstrate that the horseshoe regularisation is useful far more broadly, by reviewing both methodological and computational developments in complex models that are more relevant to machine learning applications. Specifically, we focus on methodological challenges in horseshoe regularisation in non‐linear and non‐Gaussian models, multivariate models and deep neural networks. We also outline the recent computational developments in horseshoe shrinkage for complex models along with a list of available software implementations that allows one to venture out beyond the comfort zone of the canonical linear regression problems.

Suggested Citation

  • Anindya Bhadra & Jyotishka Datta & Yunfan Li & Nicholas Polson, 2020. "Horseshoe Regularisation for Machine Learning in Complex and Deep Models," International Statistical Review, International Statistical Institute, vol. 88(2), pages 302-320, August.
  • Handle: RePEc:bla:istatr:v:88:y:2020:i:2:p:302-320
    DOI: 10.1111/insr.12360
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/insr.12360
    Download Restriction: no

    File URL: https://libkey.io/10.1111/insr.12360?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Nicholas G. Polson & James G. Scott, 2016. "Mixtures, envelopes and hierarchical duality," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(4), pages 701-727, September.
    2. Carpenter, Bob & Gelman, Andrew & Hoffman, Matthew D. & Lee, Daniel & Goodrich, Ben & Betancourt, Michael & Brubaker, Marcus & Guo, Jiqiang & Li, Peter & Riddell, Allen, 2017. "Stan: A Probabilistic Programming Language," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 76(i01).
    3. Li, Hanning & Pati, Debdeep, 2017. "Variable selection using shrinkage priors," Computational Statistics & Data Analysis, Elsevier, vol. 107(C), pages 107-119.
    4. Carlos M. Carvalho & Nicholas G. Polson & James G. Scott, 2010. "The horseshoe estimator for sparse signals," Biometrika, Biometrika Trust, vol. 97(2), pages 465-480.
    5. Veronika Ročková & Edward I. George, 2014. "EMVS: The EM Approach to Bayesian Variable Selection," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(506), pages 828-846, June.
    6. Daniel R. Kowal & David S. Matteson & David Ruppert, 2019. "Dynamic shrinkage processes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 81(4), pages 781-804, September.
    7. Anindya Bhadra & Bani K. Mallick, 2013. "Joint High-Dimensional Bayesian Variable and Covariance Selection with an Application to eQTL Analysis," Biometrics, The International Biometric Society, vol. 69(2), pages 447-457, June.
    8. Nicholas G. Polson & James G. Scott & Jesse Windle, 2013. "Bayesian Inference for Logistic Models Using Pólya--Gamma Latent Variables," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(504), pages 1339-1349, December.
    9. Anindya Bhadra & Jyotishka Datta & Nicholas G. Polson & Brandon Willard, 2016. "Default Bayesian analysis with global-local shrinkage priors," Biometrika, Biometrika Trust, vol. 103(4), pages 955-969.
    10. Lam, Clifford & Fan, Jianqing, 2009. "Sparsistency and rates of convergence in large covariance matrix estimation," LSE Research Online Documents on Economics 31540, London School of Economics and Political Science, LSE Library.
    11. P. Richard Hahn & Jingyu He & Hedibert Lopes, 2018. "Bayesian Factor Model Shrinkage for Linear IV Regression With Many Instruments," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(2), pages 278-287, April.
    12. Jacquier, Eric & Johannes, Michael & Polson, Nicholas, 2007. "MCMC maximum likelihood for latent state models," Journal of Econometrics, Elsevier, vol. 137(2), pages 615-640, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matthew F. Dixon & Nicholas G. Polson & Kemen Goicoechea, 2022. "Deep Partial Least Squares for Empirical Asset Pricing," Papers 2206.10014, arXiv.org.
    2. Robert B. Gramacy, 2020. "Discussion," International Statistical Review, International Statistical Institute, vol. 88(2), pages 326-329, August.
    3. Korobilis, Dimitris & Landau, Bettina & Musso, Alberto & Phella, Anthoulla, 2021. "The time-varying evolution of inflation risks," Working Paper Series 2600, European Central Bank.
    4. Dimitris Korobilis & Kenichi Shimizu, 2022. "Bayesian Approaches to Shrinkage and Sparse Estimation," Foundations and Trends(R) in Econometrics, now publishers, vol. 11(4), pages 230-354, June.
    5. Anindya Bhadra, 2022. "Discussion to: Bayesian graphical models for modern biological applications by Y. Ni, V. Baladandayuthapani, M. Vannucci and F.C. Stingo," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 31(2), pages 235-239, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dimitris Korobilis & Kenichi Shimizu, 2022. "Bayesian Approaches to Shrinkage and Sparse Estimation," Foundations and Trends(R) in Econometrics, now publishers, vol. 11(4), pages 230-354, June.
    2. Hu, Guanyu, 2021. "Spatially varying sparsity in dynamic regression models," Econometrics and Statistics, Elsevier, vol. 17(C), pages 23-34.
    3. Anindya Bhadra & Jyotishka Datta & Nicholas G. Polson & Brandon T. Willard, 2020. "Global-Local Mixtures: A Unifying Framework," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 82(2), pages 426-447, August.
    4. Lee, Kyoungjae & Jo, Seongil & Lee, Jaeyong, 2022. "The beta-mixture shrinkage prior for sparse covariances with near-minimax posterior convergence rate," Journal of Multivariate Analysis, Elsevier, vol. 192(C).
    5. Uddin, Md Nazir & Gaskins, Jeremy T., 2023. "Shared Bayesian variable shrinkage in multinomial logistic regression," Computational Statistics & Data Analysis, Elsevier, vol. 177(C).
    6. Paul A. Parker & Scott H. Holan, 2023. "A Bayesian functional data model for surveys collected under informative sampling with application to mortality estimation using NHANES," Biometrics, The International Biometric Society, vol. 79(2), pages 1397-1408, June.
    7. Anindya Bhadra & Jyotishka Datta & Nicholas G. Polson & Brandon T. Willard, 2021. "The Horseshoe-Like Regularization for Feature Subset Selection," Sankhya B: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 83(1), pages 185-214, May.
    8. Cássio Roberto de Andrade Alves & Márcio Laurini, 2023. "Estimating the Capital Asset Pricing Model with Many Instruments: A Bayesian Shrinkage Approach," Mathematics, MDPI, vol. 11(17), pages 1-20, September.
    9. Anindya Bhadra & Arvind Rao & Veerabhadran Baladandayuthapani, 2018. "Inferring network structure in non†normal and mixed discrete†continuous genomic data," Biometrics, The International Biometric Society, vol. 74(1), pages 185-195, March.
    10. Hauzenberger, Niko, 2021. "Flexible Mixture Priors for Large Time-varying Parameter Models," Econometrics and Statistics, Elsevier, vol. 20(C), pages 87-108.
    11. Sifat, Imtiaz & Zarei, Alireza & Hosseini, Seyedmehdi & Bouri, Elie, 2022. "Interbank liquidity risk transmission to large emerging markets in crisis periods," International Review of Financial Analysis, Elsevier, vol. 82(C).
    12. Andreas Kryger Jensen & Claus Thorn Ekstrøm, 2021. "Quantifying the trendiness of trends," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 70(1), pages 98-121, January.
    13. Qi Zhang & Yihui Zhang & Yemao Xia, 2024. "Bayesian Feature Extraction for Two-Part Latent Variable Model with Polytomous Manifestations," Mathematics, MDPI, vol. 12(5), pages 1-23, March.
    14. Dimitris Korobilis, 2020. "Sign restrictions in high-dimensional vector autoregressions," Working Paper series 20-09, Rimini Centre for Economic Analysis.
    15. Peter Knaus & Sylvia Fruhwirth-Schnatter, 2023. "The Dynamic Triple Gamma Prior as a Shrinkage Process Prior for Time-Varying Parameter Models," Papers 2312.10487, arXiv.org.
    16. Li, Yunfan & Datta, Jyotishka & Craig, Bruce A. & Bhadra, Anindya, 2021. "Joint mean–covariance estimation via the horseshoe," Journal of Multivariate Analysis, Elsevier, vol. 183(C).
    17. Daniel Spencer & Rajarshi Guhaniyogi & Raquel Prado, 2020. "Joint Bayesian Estimation of Voxel Activation and Inter-regional Connectivity in fMRI Experiments," Psychometrika, Springer;The Psychometric Society, vol. 85(4), pages 845-869, December.
    18. Hannaford, Naomi E. & Heaps, Sarah E. & Nye, Tom M.W. & Curtis, Thomas P. & Allen, Ben & Golightly, Andrew & Wilkinson, Darren J., 2023. "A sparse Bayesian hierarchical vector autoregressive model for microbial dynamics in a wastewater treatment plant," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).
    19. Adam N. Smith & Jim E. Griffin, 2023. "Shrinkage priors for high-dimensional demand estimation," Quantitative Marketing and Economics (QME), Springer, vol. 21(1), pages 95-146, March.
    20. Posch, Konstantin & Arbeiter, Maximilian & Pilz, Juergen, 2020. "A novel Bayesian approach for variable selection in linear regression models," Computational Statistics & Data Analysis, Elsevier, vol. 144(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:istatr:v:88:y:2020:i:2:p:302-320. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/isiiinl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.