Effects of granularity of search results on the relevance judgment behavior of engineers: Building systems for retrieval and understanding of context

Granularity is a novel concept for presenting information in search result interfaces of hierarchical query‐driven information retrieval systems in a manner that can support understanding and exploration of the context of the retrieved information (e.g., by highlighting its position in the granular hierarchy and exposing its relationship with relatives in the hierarchy). Little research, however, has been conducted on the effects of granularity of search results on the relevance judgment behavior of engineers. Engineers are highly motivated information users who are particularly interested in understanding the context of the retrieved information. Therefore, it is hypothesized that the design of systems with careful regard for granularity would improve engineers' relevance judgment behavior. To test this hypothesis, a prototype system was developed and evaluated in terms of the time needed for users to find relevant information, the accuracy of their relevance judgment, and their subjective satisfaction. To evaluate the prototype, a user study was conducted where participants were asked to complete tasks, complete a satisfaction questionnaire, and be interviewed. The findings showed that participants performed better and were more satisfied when the prototype system presented only relevant information in context. Although this study presents some novel findings about the effects of granularity and context on user relevance judgment behavior, the results should be interpreted with caution. For example, participants in this research were recruited by convenience and performed a set of simulated tasks as opposed to real ones. However, suggestions for further research are presented.