An Experimental Comparison of Different Approaches to Determining Weights in Additive Utility Models

Several studies this past decade have examined differences between holistic and decomposed approaches to determining weights in additive utility models. Some have argued that it matters little which procedure is used, whereas others strongly favored particular methods. In this paper we address this controversy experimentally by comparing five conceptually different approaches in terms of their weights and predictive ability. The five methods are (1) multiple linear and non-linear regression analyses of ten and fifteen holistic assessments, (2) direct decomposed tradeoffs as proposed by Keeney and Raiffa (Keeney, R. L., H. Raiffa. 1977. Decisions with Multiple Objectives. Wiley, New York.), (3) a recent eigen-vector technique of Saaty (Saaty, T. L. 1977. A scaling method for priorities in hierarchical structures. J. Math. Psych. 15 (3) 234--281.) involving redundant pairwise comparisons of attributes, (4) a straightforward allocation of hundred importance points, and (5) unit weighting (i.e., equal weighting after standardizing the attributes). The decision task involved college admissions. Subjects were asked to evaluate hypothetical college applicants on the basis of verbal SAT, quantitative SAT, high-school grade point average, and a measure of extra-curricular activity. Linear as well as nonlinear attribute utility functions were used in constructing the additive models. The nonlinear functions were specified graphically by the subjects through selection from five different shapes (i.e., one per attribute). To test the predictive ability of the various models, each subject made twenty separate pairwise comparisons of alternatives (including direction and strength of preference). The prediction criteria were percentage correct predictions as well as correlations (using these twenty pairs). Seventy subjects were tested, using an (order-controlled) within-subject design, in comparing the different methods of weight determination. Monetary incentives were used to enhance motivation. In terms of findings, the methods generally differed systematically concerning the weights given to the various attributes, as well as the variances of the resulting predictions. On average, however, the methods predicted about equally well, except for unit weighting which was clearly inferior. The findings differ in this regard from the general literature. Furthermore, nonlinear models were found to be inferior to linear ones. Finally, subjects judged the methods to differ significantly in difficulty and trustworthiness, which were found to correlate inversely. The overall results raise various applied and theoretical issues, which are discussed.