The Monte Carlo method and the evaluation of retrieval system performance

The ability to distinguish between acceptable and unacceptable levels of retrieval performance and the ability to distinguish between significant and non‐significant differences between retrieval results are important to traditional information retrieval experiments. The Monte Carlo method is shown to represent an attractive alternative to the hypergeometric model for identifying the levels at which random retrieval performance is exceeded in retrieval test collections and for overcoming some of the limitations of the hypergeometric model. The Monte Carlo method produces low performance thresholds for the individual test collections that are very similar to the thresholds derived by the hypergeometric model, both at the test collection level and at the individual query level. In addition, the Monte Carlo method is much less computer‐intensive than the hypergeometric model, can be used with measures of retrieval effectiveness that take the rank order of the retrieved documents into consideration, can be used to derive the probability of obtained results, and can be used to determine the statistical significance of difference between two or more retrieval results. The ability to use the Monte Carlo method to derive the probability of obtained results and to compare two or more retrieval results makes it possible to determine more accurately how well retrieval systems operate under specific conditions and, in conjunction with the presentation of individual query results, makes it possible to determine whether relationships between query characteristics and retrieval system performance exist. Understanding these relationships should lead to improvements in the effectiveness of retrieval systems.