IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v17y2025i2p64-d1583830.html
   My bibliography  Save this article

A Weighted Composite Metric for Evaluating User Experience in Educational Chatbots: Balancing Usability, Engagement, and Effectiveness

Author

Listed:
  • Abeer Alabbas

    (Department of Technical Support, Faculty of Technical College for Girls, Technical and Vocational Training Corporation, Najran 66253, Saudi Arabia)

  • Khalid Alomar

    (Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia)

Abstract

Evaluating user experience (UX) is essential for optimizing educational chatbots to enhance learning outcomes and student productivity. This study introduces a novel weighted composite metric integrating interface usability assessment (via the Chatbot Usability Questionnaire, CUQ), engagement measurements (via the User Engagement Scale—Short Form, UES-SF), and objective performance indicators (through error rates and response times), addressing gaps in existing evaluation methods across interaction modes (text-based, menu-based, and hybrid) and question complexities. A 3 × 3 within-subject experimental design ( n = 30) was conducted, measuring these distinct UX dimensions through standardized instruments and performance metrics, supplemented by qualitative feedback. Principal Component Analysis (PCA) was used to derive weights for the composite UX metric based on empirical patterns in user interactions. Repeated-measures ANOVA revealed that the hybrid interaction mode outperformed the others, achieving significantly higher usability (F(2,58) = 89.32, p < 0.001) and engagement (F(2,58) = 8.67, p < 0.001), with fewer errors and faster response times under complex query conditions. These findings demonstrate the hybrid mode’s adaptability across question complexities. The proposed framework establishes a standardized method for evaluating educational chatbots, providing actionable insights for interface optimization and sustainable learning tools.

Suggested Citation

  • Abeer Alabbas & Khalid Alomar, 2025. "A Weighted Composite Metric for Evaluating User Experience in Educational Chatbots: Balancing Usability, Engagement, and Effectiveness," Future Internet, MDPI, vol. 17(2), pages 1-35, February.
    • Handle: RePEc:gam:jftint:v:17:y:2025:i:2:p:64-:d:1583830
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/17/2/64/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/17/2/64/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Heather L. O'Brien & Elaine G. Toms, 2008. "What is user engagement? A conceptual framework for defining user engagement with technology," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(6), pages 938-955, April.
    2. Humaid Al Naqbi & Zied Bahroun & Vian Ahmed, 2024. "Enhancing Work Productivity through Generative Artificial Intelligence: A Comprehensive Literature Review," Sustainability, MDPI, vol. 16(3), pages 1-37, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mariani, Marcello M. & Hashemi, Novin & Wirtz, Jochen, 2023. "Artificial intelligence empowered conversational agents: A systematic literature review and research agenda," Journal of Business Research, Elsevier, vol. 161(C).
    2. Mariana Cernicova-Buca & Adina Palea, 2021. "An Appraisal of Communication Practices Demonstrated by Romanian District Public Health Authorities at the Outbreak of the COVID-19 Pandemic," Sustainability, MDPI, vol. 13(5), pages 1-19, February.
    3. Afi Fouad El & Ouiddad Smail, 2021. "Consumer engagement in value co-creation within virtual video game communities," Management & Marketing, Sciendo, vol. 16(4), pages 370-386, December.
    4. Luca Longo, 2018. "Experienced mental workload, perception of usability, their interaction and impact on task performance," PLOS ONE, Public Library of Science, vol. 13(8), pages 1-36, August.
    5. Shahbaznezhad, Hamidreza & Dolan, Rebecca & Rashidirad, Mona, 2021. "The Role of Social Media Content Format and Platform in Users' Engagement Behavior," Journal of Interactive Marketing, Elsevier, vol. 53(C), pages 47-65.
    6. Jezdancher Watti & Máté Millner & Kata Siklósi & Hedvig Kiss & Oguz Kelemen & Dávid Pócs, 2022. "Smokers’ Engagement Behavior on Facebook: Verbalizing and Visual Expressing the Smoking Cessation Process," IJERPH, MDPI, vol. 19(16), pages 1-14, August.
    7. Hwang, Angel Hsing-Chi & Oh, Jeeyun, 2020. "Interacting with background music engages E-Customers more: The impact of interactive music on consumer perception and behavioral intention," Journal of Retailing and Consumer Services, Elsevier, vol. 54(C).
    8. Peters, Twan & Işık, Öykü & Tona, Olgerta & Popovič, Aleš, 2016. "How system quality influences mobile BI use: The mediating role of engagement," International Journal of Information Management, Elsevier, vol. 36(5), pages 773-783.
    9. Carolina Herrando & Julio Jiménez-Martínez & María José Martín-De Hoyos, 2017. "Passion at first sight: how to engage users in social commerce contexts," Electronic Commerce Research, Springer, vol. 17(4), pages 701-720, December.
    10. Li, Manning & Mao, Jiye, 2015. "Hedonic or utilitarian? Exploring the impact of communication style alignment on user's perception of virtual health advisory services," International Journal of Information Management, Elsevier, vol. 35(2), pages 229-243.
    11. Nripesh Trivedi & Daniel Adomako Asamoah & Derek Doran, 0. "Keep the conversations going: engagement-based customer segmentation on online social service platforms," Information Systems Frontiers, Springer, vol. 0, pages 1-19.
    12. Ayla Schwarz & Greet Cardon & Sebastien Chastin & Jeroen Stragier & Lieven De Marez & Consortium SmartLife & Ann DeSmet, 2021. "Does Dynamic Tailoring of A Narrative-Driven Exergame Result in Higher User Engagement among Adolescents? Results from A Cluster-Randomized Controlled Trial," IJERPH, MDPI, vol. 18(14), pages 1-19, July.
    13. Park, JungKun & Hong, EunPyo & Ahn, Jiseon & Hyun, Hyowon, 2023. "Role of multidimensional customer brand engagement on customer behavior for online grocery shopping," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    14. Pagani, Margherita & Malacarne, Giovanni, 2017. "Experiential Engagement and Active vs. Passive Behavior in Mobile Location-based Social Networks: The Moderating Role of Privacy," Journal of Interactive Marketing, Elsevier, vol. 37(C), pages 133-148.
    15. Muna Alharbi & Lisa Kuhn & Julia Morphet, 2021. "Nursing students' engagement with social media as an extracurricular activity: An integrative review," Journal of Clinical Nursing, John Wiley & Sons, vol. 30(1-2), pages 44-55, January.
    16. Yezheng Li & Pinyi Yao & Syuhaily Osman & Norzalina Zainudin & Mohamad Fazli Sabri, 2022. "A Thematic Review on Using Food Delivery Services during the Pandemic: Insights for the Post-COVID-19 Era," IJERPH, MDPI, vol. 19(22), pages 1-22, November.
    17. Omar Besbes & Yonatan Gur & Assaf Zeevi, 2016. "Optimization in Online Content Recommendation Services: Beyond Click-Through Rates," Manufacturing & Service Operations Management, INFORMS, vol. 18(1), pages 15-33, February.
    18. Tulia Maria Casvean & Vasile Hodorogea & Ioan Emanuel Casvean, 2021. "The Experience Created by the Social and Cultural Context Constructed by Video Games: A Digitization of Life and Death," Journal for Social Media Inquiry, Editura Lumen, vol. 3(1), pages 19-31, July.
    19. van der Harst, Jesse Pieter & Angelopoulos, Spyros, 2024. "Less is more: Engagement with the content of social media influencers," Journal of Business Research, Elsevier, vol. 181(C).
    20. Högberg, Johan & Shams, Poja & Wästlund, Erik, 2019. "Gamified in-store mobile marketing: The mixed effect of gamified point-of-purchase advertising," Journal of Retailing and Consumer Services, Elsevier, vol. 50(C), pages 298-304.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:17:y:2025:i:2:p:64-:d:1583830. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.