IDEAS home Printed from https://ideas.repec.org/a/eee/bushor/v62y2019i6p785-797.html
   My bibliography  Save this article

In bot we trust: A new methodology of chatbot performance measures

Author

Listed:
  • Przegalinska, Aleksandra
  • Ciechanowski, Leon
  • Stroz, Anna
  • Gloor, Peter
  • Mazurek, Grzegorz

Abstract

Chatbots are used frequently in business to facilitate various processes, particularly those related to customer service and personalization. In this article, we propose novel methods of tracking human-chatbot interactions and measuring chatbot performance that take into consideration ethical concerns, particularly trust. Our proposed methodology links neuroscientific methods, text mining, and machine learning. We argue that trust is the focal point of successful human-chatbot interaction and assess how trust as a relevant category is being redefined with the advent of deep learning supported chatbots. We propose a novel method of analyzing the content of messages produced in human-chatbot interactions, using the Condor Tribefinder system we developed for text mining that is based on a machine learning classification engine. Our results will help build better social bots for interaction in business or commercial environments.

Suggested Citation

  • Przegalinska, Aleksandra & Ciechanowski, Leon & Stroz, Anna & Gloor, Peter & Mazurek, Grzegorz, 2019. "In bot we trust: A new methodology of chatbot performance measures," Business Horizons, Elsevier, vol. 62(6), pages 785-797.
    • Handle: RePEc:eee:bushor:v:62:y:2019:i:6:p:785-797
    DOI: 10.1016/j.bushor.2019.08.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S000768131930117X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.bushor.2019.08.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Anna-Maria Seeger & Armin Heinzl, 2018. "Human Versus Machine: Contingency Factors of Anthropomorphism as a Trust-Inducing Design Strategy for Conversational Agents," Lecture Notes in Information Systems and Organization, in: Fred D. Davis & René Riedl & Jan vom Brocke & Pierre-Majorique Léger & Adriane B. Randolph (ed.), Information Systems and Neuroscience, pages 129-139, Springer.
    2. Alun Preece, 2018. "Asking ‘Why’ in AI: Explainability of intelligent systems – perspectives and challenges," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 25(2), pages 63-72, April.
    3. Francesca Grippa & João Leitão & Julia Gluesing & Ken Riopelle & Peter Gloor (ed.), 2018. "Collaborative Innovation Networks," Studies on Entrepreneurship, Structural Change and Industrial Dynamics, Springer, number 978-3-319-74295-3, March.
    4. Bernard Cova & Veronique Cova, 2002. "Tribal marketing: The tribalisation of society and its impact on the conduct of marketing," Post-Print hal-01822665, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yuehuan He & Oleksandr Romanko & Alina Sienkiewicz & Robert Seidman & Roy Kwon, 2021. "Cognitive User Interface for Portfolio Optimization," JRFM, MDPI, vol. 14(4), pages 1-15, April.
    2. Tao Zhang & Chao Feng & Hui Chen & Junjie Xian, 2022. "Calming the customers by AI: Investigating the role of chatbot acting-cute strategies in soothing negative customer emotions," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(4), pages 2277-2292, December.
    3. Gkinko, Lorentsa & Elbanna, Amany, 2023. "Designing trust: The formation of employees’ trust in conversational AI in the digital workplace," Journal of Business Research, Elsevier, vol. 158(C).
    4. Kamoonpuri, Sana Zehra & Sengar, Anita, 2023. "Hi, May AI help you? An analysis of the barriers impeding the implementation and use of artificial intelligence-enabled virtual assistants in retail," Journal of Retailing and Consumer Services, Elsevier, vol. 72(C).
    5. James Lappeman & Siddeeqah Marlie & Tamryn Johnson & Sloane Poggenpoel, 2023. "Trust and digital privacy: willingness to disclose personal information to banking chatbot services," Journal of Financial Services Marketing, Palgrave Macmillan, vol. 28(2), pages 337-357, June.
    6. Tran, Anh D. & Pallant, Jason I. & Johnson, Lester W., 2021. "Exploring the impact of chatbots on consumer sentiment and expectations in retail," Journal of Retailing and Consumer Services, Elsevier, vol. 63(C).
    7. Dung Minh Nguyen & Yen-Ting Helena Chiu & Huy Duc Le, 2021. "Determinants of Continuance Intention towards Banks’ Chatbot Services in Vietnam: A Necessity for Sustainable Development," Sustainability, MDPI, vol. 13(14), pages 1-24, July.
    8. Sewoong Hwang & Jonghyuk Kim, 2021. "Toward a Chatbot for Financial Sustainability," Sustainability, MDPI, vol. 13(6), pages 1-18, March.
    9. Le, Hoang Tran Phuoc Mai & Park, Jungkun & Lee, Sangwoo, 2023. "Emotion and trust in virtual service assistant design for effective service recovery," Journal of Retailing and Consumer Services, Elsevier, vol. 74(C).
    10. Liu, Wenlong & Jiang, Min & Li, Wangjie & Mou, Jian, 2024. "How does the anthropomorphism of AI chatbots facilitate users' reuse intention in online health consultation services? The moderating role of disease severity," Technological Forecasting and Social Change, Elsevier, vol. 203(C).
    11. Massilva Dekkal & Manon Arcand & Sandrine Prom Tep & Lova Rajaobelina & Line Ricard, 2024. "Factors affecting user trust and intention in adopting chatbots: the moderating role of technology anxiety in insurtech," Journal of Financial Services Marketing, Palgrave Macmillan, vol. 29(3), pages 699-728, September.
    12. Massaro, Alessandro & Magaletti, Nicola & Cosoli, Gabriele & Giardinelli, Vito O. M. & Leogrande, Angelo, 2022. "Text Mining Approaches Oriented on Customer Care Efficiency," MPRA Paper 112300, University Library of Munich, Germany.
    13. Na Cai & Shuhong Gao & Jinzhe Yan, 2024. "How the communication style of chatbots influences consumers’ satisfaction, trust, and engagement in the context of service failure," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-11, December.
    14. Huang, Yu-Shan (Sandy) & Dootson, Paula, 2022. "Chatbots and service failure: When does it lead to customer aggression," Journal of Retailing and Consumer Services, Elsevier, vol. 68(C).
    15. Mark Anthony Camilleri & Ciro Troise, 2023. "Live support by chatbots with artificial intelligence: A future research agenda," Service Business, Springer;Pan-Pacific Business Association, vol. 17(1), pages 61-80, March.
    16. Ciechanowski, Leon & Jemielniak, Dariusz & Gloor, Peter A., 2020. "TUTORIAL: AI research without coding: The art of fighting without fighting: Data science for qualitative researchers," Journal of Business Research, Elsevier, vol. 117(C), pages 322-330.
    17. Mahendar Goli & Anoop Kumar Sahu & Surajit Bag & Pavitra Dhamija, 2023. "Users' Acceptance of Artificial Intelligence-Based Chatbots: An Empirical Study," International Journal of Technology and Human Interaction (IJTHI), IGI Global, vol. 19(1), pages 1-18, January.
    18. Hossein Hassani & Xu Huang & Emmanuel Silva & Mansi Ghodsi, 2020. "Deep Learning and Implementations in Banking," Annals of Data Science, Springer, vol. 7(3), pages 433-446, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nicoletta Buratti & Francesco Derchi & Giorgia Profumo, 2015. "The blurred boundary between empowered and working consumers: insights from the winner taco case," MERCATI & COMPETITIVIT?, FrancoAngeli Editore, vol. 2015(4), pages 133-156.
    2. Valentina Daniela N. CONSTANTIN & Roxana-Denisa G. STOENESCU, 2014. "The Impact Of Origin On Creating A Cult Brand: The Case Of Apple," SEA - Practical Application of Science, Romanian Foundation for Business Intelligence, Editorial Department, issue 3, pages 123-134, April.
    3. Boisvert, Jean & Christodoulides, George & Sajid Khan, M., 2023. "Toward a better understanding of key determinants and consequences of masstige consumption," Journal of Business Research, Elsevier, vol. 161(C).
    4. G. Debucquet & P. Guillotreau & G. Lazuech & F. Salladarré & J. Troiville, 2020. "Sense of belonging and commitment to a community-supported fishery. The case of Yeu Island, France," Review of Agricultural, Food and Environmental Studies, Springer, vol. 101(4), pages 439-459, December.
    5. Neil Higgs, 2007. "Measuring and understanding the well-being of South Africans: Everyday quality of life in South Africa," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 81(2), pages 331-356, April.
    6. Sharifah Faridah Syed Alwi & T. C. Melewar & Maria Teresa Cuomo & Manfred Schwaiger, 2020. "Digital Society and Corporate Reputation: Towards the Next Generation of Insights," Corporate Reputation Review, Palgrave Macmillan, vol. 23(3), pages 129-132, August.
    7. Krzysztof Herman & Leon Ciechanowski & Aleksandra Przegalińska, 2021. "Emotional Well-Being in Urban Wilderness: Assessing States of Calmness and Alertness in Informal Green Spaces (IGSs) with Muse—Portable EEG Headband," Sustainability, MDPI, vol. 13(4), pages 1-20, February.
    8. Hemant C. Sashittal & Avan R. Jassawalla, 2020. "The personal influence of Instagram bloggers on consumer–brand interactions: brands as tribal artifacts," Journal of Brand Management, Palgrave Macmillan, vol. 27(6), pages 679-690, November.
    9. Douglas Bryson & Glyn Atwal & Peter Hultén & Klaus Heine, 2021. "Antecedents of Luxury Brand Hate : A Quantitative Study," Post-Print hal-03188197, HAL.
    10. Ranfagni, Silvia & Runfola, Andrea, 2018. "Connecting passion: Distinctive features from emerging entrepreneurial profiles," Journal of Business Research, Elsevier, vol. 92(C), pages 403-411.
    11. Baraldi, Enrico & Gressetvold, Espen & Harrison, Debbie, 2012. "Resource interaction in inter-organizational networks: Foundations, comparison, and a research agenda," Journal of Business Research, Elsevier, vol. 65(2), pages 266-276.
    12. Soniya Billore, 2021. "Cultural Consumption and Citizen Engagement—Strategies for Built Heritage Conservation and Sustainable Development. A Case Study of Indore City, India," Sustainability, MDPI, vol. 13(5), pages 1-15, March.
    13. Luigi Cantone & Pierpaolo Testa & Teresa Marrone, 2022. "Issues in defining and placing consumer brand engagement," Italian Journal of Marketing, Springer, vol. 2022(2), pages 135-172, June.
    14. Anido Freire, N., 2014. "When luxury advertising adds the identitary values of luxury: A semiotic analysis," Journal of Business Research, Elsevier, vol. 67(12), pages 2666-2675.
    15. Angelica-Nicoleta NECULAESEI (ONEA), 2017. "Ethnomarketing And Tribal Marketing – General Issues," CrossCultural Management Journal, Fundația Română pentru Inteligența Afacerii, Editorial Department, issue 2, pages 121-127, December.
    16. Hauke A. Wetzel & Stefan Hattula & Maik Hammerschmidt & Harald J. Heerde, 2018. "Building and leveraging sports brands: evidence from 50 years of German professional soccer," Journal of the Academy of Marketing Science, Springer, vol. 46(4), pages 591-611, July.
    17. Thuy Duong Oesterreich & Eduard Anton & Julian Schuir & Alexander Brehm & Frank Teuteberg, 2023. "How can I help you? Design principles for task-oriented speech dialog systems in customer service," Information Systems and e-Business Management, Springer, vol. 21(1), pages 37-79, March.
    18. Stoeckl, Verena E. & Luedicke, Marius K., 2015. "Doing well while doing good? An integrative review of marketing criticism and response," Journal of Business Research, Elsevier, vol. 68(12), pages 2452-2463.
    19. Contreras, Dante & Elacqua, Gregory & Martinez, Matías & Miranda, Álvaro, 2016. "Bullying, identity and school performance: Evidence from Chile," International Journal of Educational Development, Elsevier, vol. 51(C), pages 147-162.
    20. Rodrigo César Tavares Cavalcanti & André Luiz Maranhão de Souza-Leão & Bruno Melo Moura, 2021. "Fan Affirmation: Alethurgy on an Indie Music Fandom," RAC - Revista de Administração Contemporânea (Journal of Contemporary Administration), ANPAD - Associação Nacional de Pós-Graduação e Pesquisa em Administração, vol. 25(5), pages 190395-1903.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:bushor:v:62:y:2019:i:6:p:785-797. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/bushor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.