Optimization in Online Content Recommendation Services: Beyond Click-Through-Rates

A new class of online services allows publishers to direct readers from articles they are currently reading to other web-based content they may be interested in. A key feature of such a dynamic recommendation service is that users interact with the provider along their browsing path. While the click-through rate of articles (a myopic performance indicator) is often the key metric accounted for in the recommendation process, we quantify the performance improvement that one may capture by accounting for the potential future path of users. To that end, using a large data set of user path history at major media sites, we develop a representation of content along two key dimensions: clickability, the likelihood to click to an article when it is recommended; and engageability, the likelihood to click from an article when it hosts a recommendation. We introduce a class of path-focused heuristics that leverages engageability values, quantify its performance and then test its impact when integrated into the operating system of a worldwide leading provider of content recommendations. We conduct a live experiment to compare the performance of these heuristics (adjusting for the limitations of real-time information flow) to that of current algorithms used by the service provider. The experiment documents the improvement relative to current practice, which is attributable to accounting for the future path of users through the engageability parameters when optimizing recommendations.