Distribution neglect in performance evaluations

Five empirical studies, including both laboratory experiments and an archival investigation, provide evidence that decision makers often fail to consider variability and skew when making judgments about performance. We term this distribution neglect. Participants’ spontaneous explanations for group differences in elite achievement overwhelmingly invoked mean differences rather than group differences in variability, even when the complete distribution and summary statistics were provided (Study 1). A longitudinal examination indicates that NBA teams overweight average performance and underweight consistency of performance when deciding players’ contracts (Study 2), providing evidence that neglecting variance information leads to suboptimal judgments. In a manufacturing scenario involving monitoring assembly line workers, participants were more accurate at identifying top (high mean) performers than consistent (low variability) performers (Study 3). In a hiring simulation, decision makers were more likely to factor in variance when performance data was presented visually as a histogram (Study 4). Finally, participants’ spontaneous explanations for others’ self-assessments of ability assumed egocentric bias, when a skewed performance distribution was also a plausible contributor (Study 5). Individual differences (need for cognition) and task differences (such as style of information display) were associated with increased distribution-based reasoning in multiple studies, suggesting potential boundary conditions for further investigation. Organizational implications, and additional potential remedies for distribution neglect, are discussed.