OLS: Is That So Useless for Regression with Categorical Data?

Binary/categorical response data abound in many application areas poses a unique problem; OLS-based model may lead to negative estimate for probability of a particular category and does not provide coherent forecast for the response variable. This unique and undesirable property of linear regression with categorical data impedes the use of OLS which otherwise is the simplest and distributionally robust method. The logit or probit kind of solution is heavily distribution dependent or link function dependent. Failure of such distributional assumption of the underlying latent variable model may cost the estimators heavily and may lead to biased and inconsistent estimates, in general. In this paper, we attempt to fix the inherent problem of linear regression by suggesting a simple manipulation which, in turn, leads to consistent estimates of probability of a category, and results in coherent forecasts for the response variable. We show that the proposed solution provides comparable estimates, and sometimes, with respect to some criterion, the proposed method is even slightly better than the logit kind of models. Here, we consider different underlying error distributions and compare the performances of the two models (in terms of their respective residual sum of squares and also in terms of relative entropy) based on simulated data. It is evidenced that the OLS performs better for many distributions, viz., Gamma, Laplace, and Uniform error distributions.