Evaluating human behaviour in response to AI recommendations for judgemental forecasting

Various advanced systems deploy artificial intelligence (AI) and machine learning (ML) to improve demand forecasting. Supply chain planners need to become familiar with these systems and trust them, considering real-world complexities and challenges the systems are exposed to. However, planners have the opportunity to intervene based on their experience or information that the systems may not capture. In this context, we study planners’ adjustments to AI-generated demand forecasts. We collect a large amount of data from a leading AI provider and a large European retailer. Our dataset contains 30 million forecasts at the SKU-store-day level for 2019, plus variables related to products, weather, and holidays. In our two-phase analysis, we aim to understand the adjustments made by planners and the quality of these adjustments. Within each phase, we first identify the drivers of adjustments and their quality using random forest, a well-known ML algorithm. Next, we investigate the collective effects of the different drivers on the occurrence and the quality of the adjustments using a decision tree approach. We find that product characteristics such as price, freshness, and discounts are important factors when making adjustments. Large positive adjustments occur more frequently but are often inaccurate, while large negative adjustments are generally more accurate but fewer in number. Thus, planners do not contribute to accuracy on average. Our findings provide insights for the better use of human knowledge in judgmental forecasting.