Towards the identification of the optimal number of relevance categories

In this paper, we are concerned with participants' confidence in their judgments of the relevance of bibliographic records to particular research questions. We describe an empirical investigation of the association between judges' confidence and the number of categories for a relevance rating scale. Participants rated the relevance of bibliographic records, and recorded their confidence in the relevance ratings. We hypothesize that confidence in relevance judgments is a function of the number of relevance categories that are available in the rating scale. We consider scales ranging from 2 to 11 points, and define the optimal scale as the one for which participants express a maximum level of confidence. A pilot study finds no optimal number of points (because confidence continues to improve slightly through the 11‐point scale); nevertheless, the study shows little added benefit associated with scales that have more than six points. On the basis of the findings in that study, we adjusted our experimental procedures and found, in our principal study, that the optimal scale for maximizing confidence in relevance judgments has approximately seven points. We also present exploratory results involving gender effects, and the comparison of scales that have an odd number of points (for which a neutral judgment is possible) with scales that have an even number of points.