A simulation of factors affecting machine learning techniques: an examination of partitioning and class proportions

Machine learning techniques, such as neural networks and rule induction, are becoming popular alternatives to traditional statistical techniques for solving classification problems. However, much of the research has been devoted to comparing performances upon sample data sets, with little attention paid to why a technique sometimes outperforms another. This study describes a simulation, which examined the effects of factors with theoretical support for their differential impacts upon three machine learning techniques (a backpropagation neural network and two rule induction techniques: CART and ID3) and discriminant analysis. The results demonstrate significant differences in the techniques' abilities to reduce overfitting, to form diagonal partitions, and to compensate for variations between actual and sample data class proportions. This helps explain why a particular technique may perform well in one context and not in another.