Fuzzy versus probabilistic models for user relevance judgments

A crucial aspect of information retrieval is the process of making relevance judgments. Although this highly complex decision making procedure still eludes researchers, it is evident that a number of mental models are involved: models of the information need, retrieval system, database, user's knowledge in the subject area etc. In general, it is accepted that relevance judgments are made by evaluating documents for an overall conceptual match with the information need. In this study, we take the view that, given a document and a query, users first judge the document against the individual concepts in the query and then use some inferencing process to derive from these “atomic” decisions a “compound” judgment for the entire query. In other words we adopt a bottom up approach to this decision making process. In this context we examine strategies that may be used to infer compound relevance judgments made from judgments against smaller units of the information need. The probabilistic and fuzzy models are used as two points of reference against which to analyze the user's decision process in making compound judgments. Each model has a different way of approaching the problem. The objective was to empirically examine the relative effectiveness of the models at predicting compound judgments made by users, from their atomic judgments. The results show that the fuzzy model is sometimes a better predictor than the probabilistic model. The conclusions are interesting when comparing performances across AND and OR queries. No differences were observed between the two models for the OR queries. Both models tend to underestimate relevance for the AND queries. However, the probabilistic model regularly underestimated relevance more than the fuzzy model. The conclusion made is that the user seldom employs the AND operator as rigidly as assumed by these models and perhaps by most IR systems. Also, matching functions that better approximate the user's decision process are required to lead to more effective systems. © 1990 John Wiley & Sons, Inc.