AI-Powered Fraud and the Erosion of Online Survey Integrity: An Analysis of 31 Fraud Detection Strategies

The proliferation of AI-powered bots and sophisticated fraudsters poses a significant threat to the integrity of scientific studies reliant on online surveys across diverse disciplines, including health, social, environmental and political sciences. We found a substantial decline in usable responses from online surveys from 75% to 10% in recent years due to survey fraud. Monetary incentives attract sophisticated fraudsters capable of mimicking genuine open-ended responses and verifying information submitted months prior, showcasing the advanced capabilities of online survey fraud today. This study evaluates the efficacy of 31 fraud indicators and 6 ensembles using two agriculture surveys in California. To evaluate the performance of each indicator, we use predictive power and recall. Predictive power is a novel variation of precision introduced in this study, and both are simple metrics that allow for non-academic survey practitioners to replicate our methods. The best indicators included a novel email address score, MinFraud Risk Score, consecutive submissions, opting-out of incentives, improbable location, and survey start time. Despite multiple methodological innovations, none of the indicators or ensemble tests proved adequate due to the large proportion of fraudulent responses in original data samples. Findings underscore evolving tactics of fraudsters, demonstrating their increased proficiency in responding to matching, domain knowledge, and open-ended questions. We conclude with recommendations for developing adaptable fraud detection strategies.