Evaluating an information system for policy modeling and uncertainty analysis

Some key issues in evaluating advanced information systems are illustrated by the evaluation of a system, named Demos, for decision modeling and uncertainty analysis. Evaluating an information system is particularly hard when it is intended not just to help users do something faster or more reliably, but also to transform the way they do it. Demos is a nonprocedural modeling system designed to encourage policy analysts to deal more effectively with uncertainty. It is intended to facilitate and encourage more systematic sensitivity analysis, a probabilistic treatment of uncertainty, exploration of alternative model formulations, and clearer communication of model assumptions and implications. A wide variety of approaches to experimentation and evaluation of the use of the Demos system for decision modeling suggest that such an environment can indeed facilitate more effective modeling, at least by expert analysts, but its power also opens up new pitfalls for novices. Implications for further developments of such systems and the need for better tutorial material are explored. © 1986 John Wiley & Sons, Inc.