CWBCM method to determine the importance of classification performance evaluation criteria in machine learning: Case studies of COVID-19, Diabetes, and Thyroid Disease

Problems with multiple conflicting criteria are usually modeled by the methods proposed in the field of Multi-Criteria Decision Making (MCDM). In MCDM, one of the most important topics is the weighting of criteria. On the other hand, classification is employed in numerous real-world issues, like disease diagnosis. Diverse algorithms have been developed for this purpose. It is important to evaluate the classification performance in every problem by evaluating algorithms. The evaluation of algorithms includes several conflicting criteria; Therefore, it can be represented as an MCDM problem. We aim to develop a new weighting method that can be used for the classification problem with more than two classes, involve the risk of threatening human life, and consider minor features for different diseases in weighting. At present, none of the existing weighting methods fulfill these requirements. This research presents a new method called “Criteria Weighting Based on Confusion Matrix (CWBCM)” and our innovation is that, for the first time, all these gaps are filled with this method. This method calculates the exact importance of criteria using the confusion matrix in machine learning. The proposed method has been implemented on six different datasets of three diseases: COVID-19, thyroid, and diabetes, and compared with two common methods, Shannon and AHP. Two methods, TOPSIS and EDAS, were also used to rank the classifiers. Finally, the results show that our method is superior to the other two weighting methods in all critical factors and has unique features that other methods do not have.