Active learning strategies for interactive elicitation of assignment examples for threshold-based multiple criteria sorting

We consider an interactive elicitation of holistic preference information for multiple criteria sorting approached with a threshold-based value-driven procedure. We introduce several active learning strategies for selecting, in each stage of interaction, an alternative that the Decision Maker (DM) should assign to its desired class. To identify the best assignment-based question, we evaluate each candidate alternative in terms of either ambiguity in its possible assignments at the current stage of interaction or its potential contribution to reducing uncertainty in the assignments of all alternatives once the question is answered. The performance of the proposed heuristic strategies is experimentally verified in view of computational time as well as the average and maximal numbers of questions that need to be answered by the DM until the classification recommended by all compatible preference models is sufficiently robust. We demonstrate that competitive results can be obtained with the heuristics that select the next question based on the analysis of current classification results as compared to the strategies looking ahead the current stage, which takes significantly more time. We also show how the performance of the questioning strategies is affected when, e.g., considering various problem sizes, imposing different stopping criteria for the preference elicitation, or reducing the flexibility of an assumed preference model by fixing the class thresholds.