Coherence and calibration in expert probability judgement

Many decision-aiding technologies require valid probability judgements to be elicited from domain experts. But how valid are experts' probability judgements? It is of considerable practical importance to identify the conditions which affect the quality of these judgments. Descriptive psychological models permit the identification of situations in which judgement is likely to be poor, and suggest methods by which judgement may be improved. We describe two approaches to the assessment of the quality of probability judgement--calibration and coherence--and review research findings following from these two approaches. This review is carried out within a framework of the psychological processes required to make a probability judgement. Three rival psychological models of probability judgement are located within this framework, and are evaluated in the light of the empirical findings. We conclude that none of these three models is unequivocally supported by the empirical data. We suggest that this may be the case because the models, experimental tasks and measurement techniques are not sophisticated enough. We make some specific proposals for the resolution of these problems.