Evaluating Human Behaviour in Response to AI Recommendations for Judgemental Forecasting

Various advanced systems deploy artificial intelligence (AI) and machine learning (ML) to improve demand forecasting. Supply chain planners need to become familiar with these systems and trust them, considering real-world complexities and challenges the systems are exposed to. However, planners have the opportunity to intervene based on their experience or information that the systems may not capture. In this context, we study planners’ adjustments to AI-generated demand forecasts. We collect a large amount of data from a leading AI provider and a large European retailer. Our dataset contains 30 million forecasts at the SKU-store-day level for 2019, plus variables related to products, weather, and holidays. In our two-phase analysis, we aim to understand the adjustments made by planners and the quality of these adjustments. Within each phase, we first identify the drivers of adjustments and their quality using random forest, a well-known ML algorithm. Next, we investigate the collective effects of the different drivers on the occurrence and the quality of the adjustments using a decision tree approach. We find that product characteristics such as price, freshness, and discounts are important factors when making adjustments. Large positive adjustments occur more frequently but are often inaccurate, while large negative adjustments are generally more accurate but fewer in number. Thus, planners do not contribute to accuracy on average. Our findings provide insights for the better use of human knowledge in judgemental forecasting.

Naghmeh Khosrowabadi joined Kühne Logistics University as a PhD Candidate in the field of Supply Chain Management under the supervision of Prof. Dr. Kai Hoberg in May 2019. Her PhD research concentrates on supply chain analytics using data science approaches. Prior to the KLU she received her Master's degree in Industrial Engineering at Iran University of Science and Technology in Iran in Jun. 2018. She won the top student awards in Bachelor and entered to Master degree straightly without entrance examination with brilliant talent ranking in 2016. In addition, she also finished her Master's degree with rank 1 among all of her co-faculty students. The candidate is a holder of Awards and Facilities of the National Elite Foundation in Iran during (2017- 2018). Besides her study, she gained some experience as a member of SSID (Scientific Society of Industrial Engineering Department) in IUST for one year. Moreover, she was a research assistant for more than 9 months during her Master's study.