Evaluating methods to prevent and detect inattentive respondents in web surveys

Inattentive respondents pose a substantial threat to data quality in web surveys. To minimize this threat, we evaluate methods for preventing and detecting inattentive responding and investigate its impacts on substantive research. First, we test the effect of asking respondents to commit to providing high-quality responses at the beginning of the survey on various data quality measures. Second, we compare the proportion of flagged respondents for two versions of an attention check item instructing them to select a specific response vs. leaving the item blank. Third, we propose a timestamp-based cluster analysis approach that identifies clusters of respondents who exhibit different speeding behaviors. Lastly, we investigate the impact of inattentive respondents on univariate, regression, and experimental analyses. Our findings show that the commitment pledge had no effect on the data quality measures. Instructing respondents to leave the item blank instead of providing a specific response significantly increased the rate of flagged respondents (by 16.8 percentage points). The timestamp-based clustering approach efficiently identified clusters of likely inattentive respondents and outperformed a related method, while providing additional insights on speeding behavior throughout the questionnaire. Lastly, we show that inattentive respondents can have substantial impacts on substantive analyses.