Trust the machine! Enhancing the cooperation of humans and AI
Supply chain planners have the option of changing forecasts made by self-learning systems based on artificial intelligence. Does this add value? And which forecast is more reliable – the software’s or the human’s? In a cutting-edge study, KLU researchers Naghmeh Khosrowabadi, Prof. Kai Hoberg and Prof. Christina Imdahl (Eindhoven University of Technology) get to the bottom of these questions. According to their study, human intervention – on average – does not increase the accuracy of forecasts.
For the study, the team analyzed data from 30 million forecasts from a leading AI provider and a major European grocery retailer.
Planners’ optimism leads to more inaccurate forecasts
The results show that planners – on average – do not contribute to the accuracy of the forecast. “Instead, planners even tend to overcompensate for effects like weather or a discount that have already been considered by the AI system,” says Khosrowabadi. In the study, only 50 percent of human interventions led to improved results.
Moreover, a closer look at the data revealed that roughly five percent of AI-generated forecasts were adjusted by supply chain planners. “We wanted to know why planners decided to adjust the AI-generated forecasts,” Khosrowabadi explains. “Our results show that characteristics of the product like price, freshness or discounts are key drivers for the frequency of planners’ adjustments to AI forecasts.”
If, for example, the AI system has to make a forecast for a very expensive product, planners tend to pay more attention and are more likely to intervene. “In addition, our results show that major increases from the AI forecast – for example when the human forecast for items to be sold on a given day in a specific store is twice as high as the AI forecast – are more frequent but also often inaccurate. Too much optimism on the part of the planners seems to be an issue here,” says Prof. Kai Hoberg. Decreases from the AI forecast, on the other hand, were found to be less common but more accurate.
Improving cooperation between human planners and AI
“Human planners will rightful continue to play an important role in AI-enabled forecast processes”, says Prof. Hoberg, “In certain cases human planners have knowledge that is not accessible to an AI system, e.g. local events or competitor actions, that enables them to increase the odds for a better forecast to more than 70%. That is why we need to enhance the cooperation of planners and AI.” For this, the team recommends more exchanges between retailers and AI providers: the better planners understand how the system makes its forecasts, the easier it is for them to decide when to intervene. “Using the results of our study, companies may save money and time”, promises Khosrowabadi, “The key is to help planners to decide when they need to intervene – and when the system is fine on its own and they may concentrate on other tasks.”
The study was conducted by Naghmeh Khosrowabadi as part of her doctoral thesis, Prof. Kai Hoberg (KLU) and Prof. Christina Imdahl (Eindhoven University of Technology) and incorporated data from 30 million SKU store-day level forecasts from a major European retailer and a leading AI provider. Data on additional variables such as products, weather or holidays was also considered.
Publications: Naghmeh Khosrowabadi, Kai Hoberg, Christina Imdahl, Evaluating Human Behaviour in Response to AI Recommendations for Judgemental Forecasting, European Journal of Operational Research (2022),
doi: https://doi.org/10.1016/j.ejor.2022.03.017
Contact:
Would you like to get in touch with the authors of this study? Please feel free to contact them:
Naghmeh Khosrowabadi: naghmeh.khosrowabadi@klu.org
Prof. Dr. Kai Hoberg: kai.hoberg@klu.org
More Information:
- Download images: KLU, Naghmeh Khosrowabadi, Prof. Dr. Kai Hoberg