IDEAS home Printed from https://ideas.repec.org/a/bla/istatr/v90y2022i3p563-591.html
   My bibliography  Save this article

Priors in Bayesian Deep Learning: A Review

Author

Listed:
  • Vincent Fortuin

Abstract

While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in this regard.

Suggested Citation

  • Vincent Fortuin, 2022. "Priors in Bayesian Deep Learning: A Review," International Statistical Review, International Statistical Institute, vol. 90(3), pages 563-591, December.
    • Handle: RePEc:bla:istatr:v:90:y:2022:i:3:p:563-591
    DOI: 10.1111/insr.12502
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/insr.12502
    Download Restriction: no
    File URL: https://libkey.io/10.1111/insr.12502?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Simone Cerreia-Vioglio & Lars Peter Hansen & Fabio Maccheroni & Massimo Marinacci, 2020. "Making Decisions under Model Misspecification," Papers 2008.01071, arXiv.org, revised Aug 2022.
    2. Anindya Bhadra & Jyotishka Datta & Nicholas G. Polson & Brandon Willard, 2016. "Default Bayesian analysis with global-local shrinkage priors," Biometrika, Biometrika Trust, vol. 103(4), pages 955-969.
    3. repec:dau:papers:123456789/1908 is not listed on IDEAS
    4. David M. Blei & Alp Kucukelbir & Jon D. McAuliffe, 2017. "Variational Inference: A Review for Statisticians," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(518), pages 859-877, April.
    5. Andreas Kopf & Vincent Fortuin & Vignesh Ram Somnath & Manfred Claassen, 2021. "Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations on single cell data," PLOS Computational Biology, Public Library of Science, vol. 17(6), pages 1-17, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shen Liu & Hongyan Liu, 2021. "Tagging Items Automatically Based on Both Content Information and Browsing Behaviors," INFORMS Journal on Computing, INFORMS, vol. 33(3), pages 882-897, July.
    2. Luo, Nanyu & Ji, Feng & Han, Yuting & He, Jinbo & Zhang, Xiaoya, 2024. "Fitting item response theory models using deep learning computational frameworks," OSF Preprints tjxab, Center for Open Science.
    3. Liu, Jie & Ye, Zifeng & Chen, Kun & Zhang, Panpan, 2024. "Variational Bayesian inference for bipartite mixed-membership stochastic block model with applications to collaborative filtering," Computational Statistics & Data Analysis, Elsevier, vol. 189(C).
    4. Djohan Bonnet & Tifenn Hirtzlin & Atreya Majumdar & Thomas Dalgaty & Eduardo Esmanhotto & Valentina Meli & Niccolo Castellani & Simon Martin & Jean-François Nodin & Guillaume Bourgeois & Jean-Michel P, 2023. "Bringing uncertainty quantification to the extreme-edge with memristor-based Bayesian neural networks," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    5. Iacopini, Matteo & Poon, Aubrey & Rossini, Luca & Zhu, Dan, 2023. "Bayesian mixed-frequency quantile vector autoregression: Eliciting tail risks of monthly US GDP," Journal of Economic Dynamics and Control, Elsevier, vol. 157(C).
    6. Seokhyun Chung & Raed Al Kontar & Zhenke Wu, 2022. "Weakly Supervised Multi-output Regression via Correlated Gaussian Processes," INFORMS Joural on Data Science, INFORMS, vol. 1(2), pages 115-137, October.
    7. Anderson, Robert M. & Duanmu, Haosui & Ghosh, Aniruddha & Khan, M. Ali, 2024. "On existence of Berk-Nash equilibria in misspecified Markov decision processes with infinite spaces," Journal of Economic Theory, Elsevier, vol. 217(C).
    8. Gary Koop & Dimitris Korobilis, 2023. "Bayesian Dynamic Variable Selection In High Dimensions," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 64(3), pages 1047-1074, August.
    9. Ziqi Zhang & Xinye Zhao & Mehak Bindra & Peng Qiu & Xiuwei Zhang, 2024. "scDisInFact: disentangled learning for integration and prediction of multi-batch multi-condition single-cell RNA-sequencing data," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    10. Dimitris Korobilis & Davide Pettenuzzo, 2020. "Machine Learning Econometrics: Bayesian algorithms and methods," Working Papers 2020_09, Business School - Economics, University of Glasgow.
    11. Jan Prüser & Florian Huber, 2024. "Nonlinearities in macroeconomic tail risk through the lens of big data quantile regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(2), pages 269-291, March.
    12. Bansal, Prateek & Krueger, Rico & Graham, Daniel J., 2021. "Fast Bayesian estimation of spatial count data models," Computational Statistics & Data Analysis, Elsevier, vol. 157(C).
    13. Korobilis, Dimitris & Koop, Gary, 2018. "Variational Bayes inference in high-dimensional time-varying parameter models," Essex Finance Centre Working Papers 22665, University of Essex, Essex Business School.
    14. Etienne Côme & Nicolas Jouvin & Pierre Latouche & Charles Bouveyron, 2021. "Hierarchical clustering with discrete latent variable models and the integrated classification likelihood," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 15(4), pages 957-986, December.
    15. Alex Burnap & John R. Hauser & Artem Timoshenko, 2023. "Product Aesthetic Design: A Machine Learning Augmentation," Marketing Science, INFORMS, vol. 42(6), pages 1029-1056, November.
    16. Anindya Bhadra & Jyotishka Datta & Nicholas G. Polson & Brandon T. Willard, 2020. "Global-Local Mixtures: A Unifying Framework," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 82(2), pages 426-447, August.
    17. Yuan Fang & Dimitris Karlis & Sanjeena Subedi, 2022. "Infinite Mixtures of Multivariate Normal-Inverse Gaussian Distributions for Clustering of Skewed Data," Journal of Classification, Springer;The Classification Society, vol. 39(3), pages 510-552, November.
    18. Stéphane Bonhomme, 2021. "Selection on Welfare Gains: Experimental Evidence from Electricity Plan Choice," Working Papers 2021-15, Becker Friedman Institute for Research In Economics.
    19. Junming Yin & Jerry Luo & Susan A. Brown, 2021. "Learning from Crowdsourced Multi-labeling: A Variational Bayesian Approach," Information Systems Research, INFORMS, vol. 32(3), pages 752-773, September.
    20. Jeong, Kuhwan & Chae, Minwoo & Kim, Yongdai, 2023. "Online learning for the Dirichlet process mixture model via weakly conjugate approximation," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:istatr:v:90:y:2022:i:3:p:563-591. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/isiiinl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.