Systematic Measurement Error Can be Detected and Quantified by Performing Ordinary Statistical Tests on Item Errors

Self-report measures are ubiquitous in behavioral science and have been a key method used to examine evolutionary hypotheses. Indeed, evidence for the existence and functional boundaries of many psychological adaptations is inferred based on responses on self-report measures. Valid inferences based on self-report data require that questionnaire items measure only the construct they were designed to measure, which in statistical terms means that measurement error is predominately random. In contrast, systematic error indicates that items measure constructs in addition to those the instrument was designed to measure. Substantial systematic error associated with measures used in research risks theoretically consistent interpretations that are, instead, spurious effects of method error. Here, we propose a strategy for estimating the portion of systematic measurement error contained in multi-item self-report measures. This strategy rests on the observation that item-errors can be treated like any other set of scores and therefore have the potential to reveal otherwise unnoticed patterns in the data.