IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v16y2023i1p4-d1306264.html
   My bibliography  Save this article

Development of an Assessment Scale for Measurement of Usability and User Experience Characteristics of Bing Chat Conversational AI

Author

Listed:
  • Goran Bubaš

    (Faculty of Organization and Informatics, University of Zagreb, 42000 Varaždin, Croatia)

  • Antonela Čižmešija

    (Faculty of Organization and Informatics, University of Zagreb, 42000 Varaždin, Croatia)

  • Andreja Kovačić

    (Faculty of Organization and Informatics, University of Zagreb, 42000 Varaždin, Croatia)

Abstract

After the introduction of the ChatGPT conversational artificial intelligence (CAI) tool in November 2022, there has been a rapidly growing interest in the use of such tools in higher education. While the educational uses of some other information technology (IT) tools (including collaboration and communication tools, learning management systems, chatbots, and videoconferencing tools) have been frequently evaluated regarding technology acceptance and usability attributes of those technologies, similar evaluations of CAI tools and services like ChatGPT, Bing Chat, and Bard have only recently started to appear in the scholarly literature. In our study, we present a newly developed set of assessment scales that are related to the usability and user experiences of CAI tools when used by university students, as well as the results of evaluation of these assessment scales specifically regarding the CAI Bing Chat tool (i.e., Microsoft Copilot). The following scales were developed and evaluated using a convenience sample (N = 126) of higher education students: Perceived Usefulness, General Usability, Learnability, System Reliability, Visual Design and Navigation, Information Quality, Information Display, Cognitive Involvement, Design Appeal, Trust, Personification, Risk Perception, and Intention to Use. For most of the aforementioned scales, internal consistency (Cronbach alpha) was in the range from satisfactory to good, which implies their potential usefulness for further studies of related attributes of CAI tools. A stepwise linear regression revealed that the most influential predictors of Intention to Use Bing Chat (or ChatGPT) in the future were the usability variable Perceived Usefulness and two user experience variables—Trust and Design Appeal. Also, our study revealed that students’ perceptions of various specific usability and user experience characteristics of Bing Chat were predominantly positive. The evaluated assessment scales could be beneficial in further research that would include other CAI tools like ChatGPT/GPT-4 and Bard.

Suggested Citation

  • Goran Bubaš & Antonela Čižmešija & Andreja Kovačić, 2023. "Development of an Assessment Scale for Measurement of Usability and User Experience Characteristics of Bing Chat Conversational AI," Future Internet, MDPI, vol. 16(1), pages 1-19, December.
    • Handle: RePEc:gam:jftint:v:16:y:2023:i:1:p:4-:d:1306264
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/16/1/4/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/16/1/4/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Xinjie Deng & Zhonggen Yu, 2023. "A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education," Sustainability, MDPI, vol. 15(4), pages 1-19, February.
    2. Foroughi, Behzad & Nhan, Pham Viet & Iranmanesh, Mohammad & Ghobakhloo, Morteza & Nilashi, Mehrbakhsh & Yadegaridehkordi, Elaheh, 2023. "Determinants of intention to use autonomous vehicles: Findings from PLS-SEM and ANFIS," Journal of Retailing and Consumer Services, Elsevier, vol. 70(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Soren, Anup Anurag & Chakraborty, Shibashish, 2024. "Adoption, satisfaction, trust, and commitment of over-the-top platforms: An integrated approach," Journal of Retailing and Consumer Services, Elsevier, vol. 76(C).
    2. Jibon Naher, 2023. "A Survey to Understand the experience of ChatGPT Usage among Engineering University Students in Bangladesh," Technium, Technium Science, vol. 15(1), pages 1-8.
    3. Pham, Hong Chuong & Duong, Cong Doanh & Nguyen, Giang Khanh Huyen, 2024. "What drives tourists’ continuance intention to use ChatGPT for travel services? A stimulus-organism-response perspective," Journal of Retailing and Consumer Services, Elsevier, vol. 78(C).
    4. Suhail, Faisal & Adel, Mouhand & Al-Emran, Mostafa & AlQudah, Adi Ahmad, 2024. "Are students ready for robots in higher education? Examining the adoption of robots by integrating UTAUT2 and TTF using a hybrid SEM-ANN approach," Technology in Society, Elsevier, vol. 77(C).
    5. Hu, Qian & Pan, Zhao, 2023. "Can AI benefit individual resilience? The mediation roles of AI routinization and infusion," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    6. Maria Karyotaki & Athanasios Drigas & Charalabos Skianis, 2024. "Contributions of the 9-Layered Model of Giftedness to the Development of a Conversational Agent for Healthy Ageing and Sustainable Living," Sustainability, MDPI, vol. 16(7), pages 1-18, March.
    7. Vikas Chauhan & Rohit Joshi & Vipin Choudhary, 2024. "Understanding intention to adopt telematics-based automobile insurance in an emerging economy: a mixed-method approach," Journal of Financial Services Marketing, Palgrave Macmillan, vol. 29(3), pages 1017-1036, September.
    8. Koh, Le Yi & Yuen, Kum Fai, 2023. "Public acceptance of autonomous vehicles: Examining the joint influence of perceived vehicle performance and intelligent in-vehicle interaction quality," Transportation Research Part A: Policy and Practice, Elsevier, vol. 178(C).
    9. Srivastava, Ankur & Gautam, Vikas & Sharma, Vikram, 2023. "Does consideration for future consequences matter in consumer decision to rent electric vehicles?," Energy Policy, Elsevier, vol. 181(C).
    10. Ma, Xiaoyue & Huo, Yudi, 2023. "Are users willing to embrace ChatGPT? Exploring the factors on the acceptance of chatbots from the perspective of AIDUA framework," Technology in Society, Elsevier, vol. 75(C).
    11. Carlos Cacciuttolo & Yaneth Vásquez & Deyvis Cano & Fernando Valenzuela, 2023. "Research Thesis for Undergraduate Engineering Programs in the Digitalization Era: Learning Strategies and Responsible Research Conduct Road to a University Education 4.0 Paradigm," Sustainability, MDPI, vol. 15(14), pages 1-27, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:16:y:2023:i:1:p:4-:d:1306264. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.