Stacking with Dynamic Weights on Base Models

Stacking is used to combine models based on different techniques using a second-level model to come up with higher accuracy. The second-level model essentially uses the values predicted by different base-level models as independent variables, while the dependent variable remains the observed one. Though fit of the base-level models differ at various parts of the data, the second-level model uses same set of weights on base-level models on the whole data. We have derived two methods where we replace the second-level model by a linear combination of base model outputs where the weights vary. In our methods, we select a part of the data based on some predefined condition of proximity for classification of a new observation. Then, weights are assigned on different base models considering their accuracy in that part of the data. In one method, all points in the neighbourhood get equal importance, while in the other method, points get importance based on proximity. The algorithms apply same principle on each of the new observations which get their neighbourhoods in different parts of the data; thus, weights vary. The new ensemble methods are tried on different datasets from different fields and found to give better results than conventional stacking.