A Review on Variable Selection in Regression Analysis

In this paper, we investigate on 39 Variable Selection procedures to give an overview of the existing literature for practitioners. "Let the data speak for themselves" has become the motto of many applied researchers since the amount of data has significantly grew. Automatic model selection have been raised by the search for data-driven theories for quite a long time now. However while great extensions have been made on the theoretical side still basic procedures are used in most empirical work, eg. Stepwise Regression. Some reviews are already available in the literature for variable selection, but always focus on a specific topic like linear regression, groups of variables or smoothly varying coefficients. Here we provide a review of main methods and state-of-the art extensions as well as a topology of them over a wide range of model structures (linear, grouped, additive, partially linear and non-parametric). We provide explanations for which methods to use for different model purposes and what are key differences among them. We also review two methods for improving variable selection in the general sense.