IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v12y2022i1p21582440221082130.html
   My bibliography  Save this article

Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study

Author

Listed:
  • Luo Yunjiu
  • Wei Wei
  • Ying Zheng

Abstract

Artificial intelligence (AI) technologies have the potential to reduce the workload for the second language (L2) teachers and test developers. We propose two AI distractor-generating methods for creating Chinese vocabulary items: semantic similarity and visual similarity. Semantic similarity refers to antonyms and synonyms, while visual similarity refers to the phenomenon that two phrases share one or more characters in common. This study explores the construct validity of the two types of selected-response vocabulary tests (AI-generated items and human expert-designed items) and compares their item difficulty and item discrimination. Both quantitative and qualitative data were collected. Seventy-eight students from Beijing Language and Culture University were asked to respond to AI-generated and human expert-designed items respectively. Students’ scores were analyzed using the two-parameter item response theory (2PL-IRT) model. Thirteen students were then invited to report their test taking strategies in the think-aloud section. The findings from the students’ item responses revealed that the human expert-designed items were easier but had more discriminating power than the AI-generated items. The results of think-aloud data indicated that the AI-generated items and expert-designed items might assess different constructs, in which the former elicited test takers’ bottom-up test-taking strategies while the latter seemed more likely to trigger test takers’ rote memorization ability.

Suggested Citation

  • Luo Yunjiu & Wei Wei & Ying Zheng, 2022. "Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study," SAGE Open, , vol. 12(1), pages 21582440221, March.
    • Handle: RePEc:sae:sagope:v:12:y:2022:i:1:p:21582440221082130
    DOI: 10.1177/21582440221082130
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/21582440221082130
    Download Restriction: no
    File URL: https://libkey.io/10.1177/21582440221082130?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Valery Okulich-Kazarin & Artem Artyukhov & Łukasz Skowron & Nadiia Artyukhova & Tomasz Wołowiec, 2024. "Will AI Become a Threat to Higher Education Sustainability? A Study of Students’ Views," Sustainability, MDPI, vol. 16(11), pages 1-19, May.
    2. Valery Okulich-Kazarin & Artem Artyukhov & Łukasz Skowron & Nadiia Artyukhova & Tomasz Wołowiec, 2024. "When Artificial Intelligence Tools Meet “Non-Violent” Learning Environments (SDG 4.3): Crossroads with Smart Education," Sustainability, MDPI, vol. 16(17), pages 1-21, September.
    3. Cheng-Feng Cheng & Chien-Che Huang & Ming-Chang Lin & Ta-Cheng Chen, 2023. "Exploring Effectiveness of Relationship Marketing on Artificial Intelligence Adopting Intention," SAGE Open, , vol. 13(4), pages 21582440231, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:12:y:2022:i:1:p:21582440221082130. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.