Language translation effects in Chatbots: Evidence from a randomized field experiment on a mobile commerce platform

This study investigates the impact of language translation innovations in artificial intelligence (AI) digital assistants. We use data from a mobile com- merce platform application in India that introduced a Hindi and English version of its previously English-only language chatbot. The data, obtained from a randomized field experiment conducted on new users, help determine the impact of introducing language translation in conversational chatbots by quantifying its effect on user metrics such as purchases and uninstalls. In the experiment, the firm only altered the language of interaction, from English- only to Hindi and English. The firm did not make any other changes in the design or purchase flow within the application. We find that language trans- lation innovations significantly increase the number of user sessions and also improve user purchases and engagement. The increase in the engagement did not emerge from an increase in the number of sessions but from an increase in interactions within a session in the bilingual app. We also observe a sharp rise in uninstalls for the population that received the bilingual app. We find that in the bilingual chatbot, uninstalls rise with increased user interactions in a high involvement product category. In sum, the results from the field experiment show that while language translation in artificially intelligent as- sistants leads to greater purchases, it could also lead to increased uninstalls. This result suggests that implementing similar language translation innova- tions in isolation, without any modifications to in-application experience, has the potential to yield negative outcomes for the firm.