Harmonization and Categorization of Metrics and Criteria for Evaluation of Recommender Systems in Healthcare From Dual Perspectives

Researchers' choice of metrics and criteria in evaluating recommender systems depends on what the researcher feels is popular among other researchers, or sometimes based on the objective of the research. There is no harmonized set of criteria and metrics that can be referenced when evaluating recommender systems in healthcare. In this article, a set of metrics and criteria are harmonized and categorized as a guide for evaluating recommender systems. By means of an online survey, the opinions of forty-four experienced researchers and other stakeholders from eight countries and four continents were sought on the relevance of identified metrics and criteria. Analysis of the results show speed and timeliness are at the top. Topping the list of criteria is the provision of information that will guide users to useful decisions. The result is presented from two logical perspectives. Four categories are then identified as a useful guide for evaluating recommender systems.