Improving predictive scoring models through model aggregation

Scoring models predict responses to some contact that will be made in the future, helping an organization decide which customers to target. They are usually built from a single “proxy” contact from the past, for which responses have already been observed. This approach is risky because there could be differences between the proxy and future contact, and other exogenous factors could have changed. We propose averaging predictions from multiple scoring models and develop a rationale for this approach by showing under certain assumptions that the expected squared difference between the true responses to the future contact and the predicted values from the averaged model is less than or equal to the expected squared difference from a single previous contact. The improvement of the aggregated model over the single model increases as (1) the variation in effect sizes across contacts increases, (2) the number of averaged contacts increases, and (3) the variance of the effect estimates increases. We incorporate the effects of external factors in our model by weighting the coefficients with a general linear model (GLM). Using data from a retail catalog company and a nonprofit organization, we evaluate our model empirically by testing whether our assumptions hold, examine the extent of variation in slopes and predicted values across models build from various previous contacts, evaluate the amount of improvement over extant models in terms of prediction error and performance as measured by a gains table, and study how improvement depends on the number of averaged contacts. Conservative estimates suggest that our method could increase annual profits for the nonprofit organization by over a half-million dollars and tens of thousands of dollars for the small catalog company.