Estimating multiple rater agreement for a rare diagnosis

This paper addresses the problem of estimating the population coefficient of agreement kappa (?) among a set of raters who independently classify a randomly selected subject into one of two categories. Of the many possible probability models for these classifications, only mixtures of binomial models incorporate random rater effects, although limiting forms of additive and multiplicative (log-linear) models may themselves be represented as mixtures of binomials. Mixture models also motivate a simple new estimator of ? that is appropriate in the important situation where one of the categories is rare. In the case of a rare category, simulations under multiplicative and mixture models demonstrate the substantially smaller mean squared error of compared to its more popular competitor. An example of psychiatric classification illustrates the plausibility of a simple mixture model as well as sizable discrepancies among estimators of ?.