Understanding performance in information systems: Blending relevance and competence

Information systems have many stakeholder groups with differing views and assessment methods of user performance. For example, users find software unfriendly, and system maintainers find users recalcitrant. Performance of information systems usually measures either the competence of the user as seen from the system's perspective or the relevance of the retrieved materials as seen from the user's perspective. This article presents brief accounts of the user‐based performance measure of relevance and the information system‐based performance measure of competence. Relevance and competence are shown to be complex notions that have not been studied conjointly. The article then reports the results of an experiment that used a generalized geographic information system to illustrate how collecting and analyzing data simultaneously from both a system and user views of performance can suggest improvements for information systems. The user's view of this generalized system was formed by respondents describing a recent geographic information need situation and its resolution. The system view of the user in this generalized geographic information system was described by the accuracy and time on tasks of subjects as they read and answered questions concerning geographic text and map documents. This exploratory research generated two major hypotheses: Relevance varies directly with levels of competence and experience, and relevance varies directly with the difficulty of the task. These relationships are interesting and need further investigation because their underlying cause is not easily determined. The findings also indicate that through a merged, no‐fault model, information science can contribute to constructing a holistic view of system performance by illustrating relationships among factors such as competence and relevance, and by exposing new factors such as expectations. © 1995 John Wiley & Sons, Inc.