An experimental investigation into the process of knowledge-based systems development

This paper presents an experimental study into the processes that may contribute to building a better knowledge-based system. A model that defines quality to be composed of two related aspects, internal and external quality, is introduced. To test the model, 24 subjects developed a knowledge-based system for MBA course planning in an experimental setting over a 7 week period. Subjects were factored by development methodology (structured vs rapid prototyping), knowledge representation scheme used (rule-based vs hybrid) and programmer quality (naive vs experienced). The major finding is that an appropriate mix of development methodology, knowledge representation and personnel is necessary. No single development methodology or knowledge representation scheme is best, and a considerable number of interactions was observed in the experiment. Rapid prototyping combined with rule-based representation produced the best external quality in terms of functionality, but when combined with hybrid representation produced the worst. Similarly, rapid prototyping combined with hybrid representation produced the worst usability. Programmer quality had a positive effect on coding productivity, which in turn resulted in an increase in system usability. As with conventional software, increasing programmer quality can be very beneficial to both process and content. The study presented provides some evidence of the anomalies that are generated in the course of system development, and how they relate to internal quality. As might be expected, experienced programmers produced significantly fewer anomalies. A relationship was found between internal quality and usability, but not functionality.