Going Beyond Black-Box Models by Leveraging Behavioral Insights: An Intent-Based Recommendation Framework

Modern recommender systems, which rely on large-scale machine learning (ML) models to predict the next item a consumer will engage with, often lack generalizability and understanding of consumer behaviors due to their black-box nature. To address these limitations, we propose an intent-based recommendation framework that leverages behavior insights to improve black-box recommender systems. Our ML framework includes an intent prediction phase that dynamically predicts consumer intents and an intent-based planning phase that integrates these predictions throughout various stages of the recommender system. By moving beyond traditional item-level predictions to a higher-order modeling of intent-driven behaviors, our approach aligns the ML system more closely with the underlying data generation process, thereby improving its performance without the need for extra data. We theoretically prove the optimality of our proposed framework by considering consumers’ scrolling behaviors, establishing a solid mathematical foundation for large-scale industrial applications. Our framework was validated through extensive A/B testing on YouTube, the world’s largest video recommendation platform. By incorporating consumer intents related to novelty and familiarity, we achieved a 0.05% increase in daily active users (DAU), one of the most significant business metric improvements observed in recent YouTube experiments. Our work provides empirical evidence in support of how behavioral insights can be utilized to improve ML systems. Contrary to the common belief that introducing structure to ML systems reduces their flexibility, our findings show that imposing a structure which aligns with the underlying data generation process can, in fact, improve the performance of these systems.