The dilemma of measurement in information retrieval research

The problem of measurement in information retrieval research is traced to its source in the first retrieval tests. The problem is seen as presenting a chronic dilemma for the field. This dilemma has taken three forms as the discipline has evolved: (1) The dilemma of measurement in the archetypal approach: Stated relevance versus user relevance; (2) the dilemma of measurement in the probabilistic approach: Realism versus formalism; and (3) the dilemma of measurement in the Information Retrieval‐Expert System (IR‐ES) approach: Linear measures of relevance versus logarithmic measures of knowledge. It is argued that the dilemma of measurement has remained intractable even given the different assumptions of the different approaches for three connected reasons—the nature of the subject matter of the field; the nature of relevance judgment; and the nature of cognition and knowledge. Finally, it is concluded that the original vision of information retrieval research as a discipline founded on quantification proved restricting for its theoretical and methodological development and that increasing recognition of this is reflected in growing interest in qualitative methods in information retrieval research in relation to the cognitive, behavioral, and affective aspects of the information retrieval interaction. © 1996 John Wiley & Sons, Inc.