A utility theoretic examination of the probability ranking principle in information retrieval

We challenge the probability ranking principle in information retrieval from the perspectives of (1) signal detection‐decision theory and (2) utility theory. If three conditions are not met by an IR system that is producing predictive probabilities of relevance, then inquirers may incur costs that are too great by selecting first those documents that the system predicts have the highest probabilities of relevance. These three conditions are that predictive probabilities are well calibrated (predictively accurate); that they are reported with certainty; and that an inquirer independently assesses the relevance of all documents he or she retrieves. When these conditions are met, signal detection analysis with fixed decision‐theoretic costs shows that the probability ranking principle is advisable. More generally, retrieval in adherence with the probability ranking principle is also advisable even when utility‐theoretic costs (or benefits) that vary with the number of relevant documents retrieved are associated with retrieval. Specifically, we prove that the utility an inquirer receives from the relevant documents he or she retrieves is maximized by selecting those documents with the largest predictive probabilities of relevance. © 1991 John Wiley & Sons, Inc.