Shrinkage priors for Bayesian penalized regression

In linear regression problems with many predictors, penalized regression techniques are often used to guard against overfitting and to select variables relevant for predicting the outcome. Classical regression techniques find coefficients that minimize a squared residual; penalized regression adds a penalty term to this residual to limit the coefficients’ sizes, thereby preventing over- fitting. Many classical penalization techniques have a Bayesian counterpart, which result in the same solutions when a specific prior distribution is used in combination with posterior mode estimates. Compared to classical penalization techniques, the Bayesian penalization techniques perform similarly or even better, and they offer additional advantages such as readily available uncertainty estimates, automatic estimation of the penalty parameter, and more flexibility in terms of penalties that can be considered. As a result, Bayesian penalization is becoming increasingly popular. The aim of this paper is to provide a comprehensive overview of the literature on Bayesian penalization. We will compare different priors for penalization that have been proposed in the literature in terms of their characteristics, shrinkage behavior, and performance in terms of prediction and variable selection in order to aid researchers to navigate the many prior options.