IDEAS home Printed from https://ideas.repec.org/a/spr/elmark/v33y2023i1d10.1007_s12525-023-00662-3.html
   My bibliography  Save this article

Enhancing conversational agents for successful operation: A multi-perspective evaluation approach for continuous improvement

Author

Listed:
  • Tom Lewandowski

    (University of Hamburg)

  • Emir Kučević

    (University of Hamburg)

  • Stephan Leible

    (University of Hamburg)

  • Mathis Poser

    (University of Hamburg)

  • Tilo Böhmann

    (University of Hamburg)

Abstract

Contemporary organizations increasingly adopt conversational agents (CAs) as intelligent and natural language-based solutions for providing services and information. CAs offer new forms of personalization, speed, (cost-)effectiveness, and automation. However, despite their hype in research and practice, many organizations still fail to seize CAs’ potential because they lack knowledge of how to evaluate and improve the quality of CAs to sustain them in organizational operations. We aim to fill this knowledge gap by conducting a design science research project in which we aggregate insights from the literature and practice to derive an applicable set of quality criteria for CAs. Our article contributes to CA research and guides practitioners by providing a blueprint to structure the evaluation of CAs and to discover areas for systematic improvement.

Suggested Citation

  • Tom Lewandowski & Emir Kučević & Stephan Leible & Mathis Poser & Tilo Böhmann, 2023. "Enhancing conversational agents for successful operation: A multi-perspective evaluation approach for continuous improvement," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-20, December.
    • Handle: RePEc:spr:elmark:v:33:y:2023:i:1:d:10.1007_s12525-023-00662-3
    DOI: 10.1007/s12525-023-00662-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s12525-023-00662-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s12525-023-00662-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Handan ÇAM, 2017. "3rd International conference on social sciences & education research," Journal of Economic and Social Thought, KSP Journals, vol. 4(3), pages 352-353, Seprember.
    2. Severin Weiler & Christian Matt & Thomas Hess, 2022. "Immunizing with information – Inoculation messages against conversational agents’ response failures," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(1), pages 239-258, March.
    3. Mengjun Li & Ayoung Suh, 2022. "Anthropomorphism in AI-enabled technology: A literature review," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(4), pages 2245-2275, December.
    4. Stephan Diederich & Alfred Benedikt Brendel & Lutz M. Kolbe, 2020. "Designing Anthropomorphic Enterprise Conversational Agents," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 62(3), pages 193-209, June.
    5. John Venable & Jan Pries-Heje & Richard Baskerville, 2016. "FEDS: a Framework for Evaluation in Design Science Research," European Journal of Information Systems, Taylor & Francis Journals, vol. 25(1), pages 77-89, January.
    6. Feng, Shengjia & Buxmann, Peter, 2020. "My Virtual Colleague: A State-of-the-Art Analysis of Conversational Agents for the Workplace," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 118817, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    7. Giulia Pavon & Lars Meyer-Waarden & Thanida Poocharoentou & Piyanut Prayatsup & Maelis Ratinaud & Sara Torne, 2020. "How Service Quality Influences Customer Acceptance and Usage of Chatbots?," Post-Print hal-03145866, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Carolin Stein & Timm Teubner & Stefan Morana, 2024. "Designing a conversational agent for supporting data exploration in citizen science," Electronic Markets, Springer;IIM University of St. Gallen, vol. 34(1), pages 1-18, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Darima Fotheringham & Michael A. Wiles, 2023. "The effect of implementing chatbot customer service on stock returns: an event study analysis," Journal of the Academy of Marketing Science, Springer, vol. 51(4), pages 802-822, July.
    2. Raffaele Fabio Ciriello & Alexandra Cecilie Gjøl Torbensen & Magnus Rotvit Perlt Hansen & Christoph Müller-Bloch, 2023. "Blockchain-based digital rights management systems: Design principles for the music industry," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-21, December.
    3. Myriam Schaschek & Fabian Gwinner & Nicolas Neis & Christoph Tomitza & Christian Zeiß & Axel Winkelmann, 2024. "Managing next generation BP-x initiatives," Information Systems and e-Business Management, Springer, vol. 22(3), pages 457-500, September.
    4. Andreas Engelmann & Gerhard Schwabe, 2024. "Certified data chats for future used car markets," Electronic Markets, Springer;IIM University of St. Gallen, vol. 34(1), pages 1-22, December.
    5. Stefan Morana & Jella Pfeiffer & Marc T. P. Adam, 2020. "User Assistance for Intelligent Systems," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 62(3), pages 189-192, June.
    6. Tao Zhang & Chao Feng & Hui Chen & Junjie Xian, 2022. "Calming the customers by AI: Investigating the role of chatbot acting-cute strategies in soothing negative customer emotions," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(4), pages 2277-2292, December.
    7. Tim Kollmer & Andreas Eckhardt, 2023. "Dark Patterns," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 65(2), pages 201-208, April.
    8. Paola Lara Machado & Montijn Ven & Banu Aysolmaz & Oktay Turetken & Jan Brocke, 2024. "Navigating Business Model Redesign: The Compass Method for Identifying Changes to the Operating Model," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 66(5), pages 607-638, October.
    9. Justina Sidlauskiene & Yannick Joye & Vilte Auruskeviciene, 2023. "AI-based chatbots in conversational commerce and their effects on product and price perceptions," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-21, December.
    10. Hendrik Haße & Hendrik Valk & Frederik Möller & Boris Otto, 2022. "Design Principles for Shared Digital Twins in Distributed Systems," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(6), pages 751-772, December.
    11. Carolin Stein & Timm Teubner & Stefan Morana, 2024. "Designing a conversational agent for supporting data exploration in citizen science," Electronic Markets, Springer;IIM University of St. Gallen, vol. 34(1), pages 1-18, December.
    12. Ulrich Gnewuch & Stefan Morana & Marc T. P. Adam & Alexander Maedche, 2022. "Opposing Effects of Response Time in Human–Chatbot Interaction," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(6), pages 773-791, December.
    13. Joshua Holstein & Max Schemmer & Johannes Jakubik & Michael Vössing & Gerhard Satzger, 2023. "Sanitizing data for analysis: Designing systems for data understanding," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-18, December.
    14. Laleh Rafati & Geert Poels, 2017. "Value-Driven Strategic Sourcing Based on Service-Dominant Logic," Service Science, INFORMS, vol. 9(4), pages 275-287, December.
    15. Thuy Duong Oesterreich & Eduard Anton & Julian Schuir & Alexander Brehm & Frank Teuteberg, 2023. "How can I help you? Design principles for task-oriented speech dialog systems in customer service," Information Systems and e-Business Management, Springer, vol. 21(1), pages 37-79, March.
    16. Rainer Alt, 2022. "Electronic Markets on platform dualities," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(1), pages 1-10, March.
    17. Stefan Stieglitz & Milad Mirbabaie & Nicholas R. J. Möllmann & Jannik Rzyski, 2022. "Collaborating with Virtual Assistants in Organizations: Analyzing Social Loafing Tendencies and Responsibility Attribution," Information Systems Frontiers, Springer, vol. 24(3), pages 745-770, June.
    18. Moutinho, José da Assunção & Fernandes, Gabriela & Rabechini, Roque, 2024. "Evaluation in design science: A framework to support project studies in the context of University Research Centres," Evaluation and Program Planning, Elsevier, vol. 102(C).
    19. Milad Mirbabaie & Stefan Stieglitz & Felix Brünker & Lennart Hofeditz & Björn Ross & Nicholas R. J. Frick, 2021. "Understanding Collaboration with Virtual Assistants – The Role of Social Identity and the Extended Self," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(1), pages 21-37, February.
    20. Brady D. Lund & Ting Wang & Nishith Reddy Mannuru & Bing Nie & Somipam Shimray & Ziang Wang, 2023. "ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(5), pages 570-581, May.

    More about this item

    Keywords

    Artificial intelligence assistants; Conversational agents; Chatbots; Quality criteria set; Design science research (DSR);
    All these keywords.

    JEL classification:

    • M15 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Business Administration - - - IT Management

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:elmark:v:33:y:2023:i:1:d:10.1007_s12525-023-00662-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.