AutoFusion of feature pruning for decision making in operations research

Recently, the fusion of algorithms in machine learning studies has taken a lot of attention, emphasizing the power of communal decision-making over-relying on a single decision-maker. One of the crucial questions in the aggregation of algorithms is which and how many models should be combined to achieve both the best accuracy and low complexity. It is already known in machine learning that as the complexity of the model increases too much, prediction accuracy decreases. There is a trade-off between these two features. In order to answer such questions, the diversity notion gets involved in overall consensus models. It is also shown that diversity alone does not determine the best ensemble (fusion), so accuracy and diversity together have been taken into account recently in such problems. We took into account those two notions simultaneously so that the number of algorithms and which algorithms should be in the ensemble is answered while solving the feature selection problems. The proposed method in this work is unique in that it includes an optimization model in the pruning phase, which finds the cardinality of the ensemble optimally. Using this optimization model, the size of the ensemble is found directly from the optimization model, instead of considered as a hyper-parameter. Our study shows a significant improvement in accuracy that achieves 0.702 on average among 8 datasets when compared to an unpruned case of 0.625. These results highlight the efficiency of our method both in model accuracy and in obtaining an optimal model complexity. We have validated our algorithm on different domains of data sets which shows better prediction accuracy values than existing ensemble-based feature selection methods.