IDEAS home Printed from https://ideas.repec.org/a/pal/palcom/v11y2024i1d10.1057_s41599-024-03755-2.html
   My bibliography  Save this article

Evaluating the role of ChatGPT in enhancing EFL writing assessments in classroom settings: A preliminary investigation

Author

Listed:
  • Junfei Li

    (Shanghai Dianji University)

  • Jinyan Huang

    (Jiangsu University
    Jiangsu University)

  • Wenyan Wu

    (Beijing Institute of Graphic Communication)

  • Patrick B. Whipple

    (The Genesee Valley Board of Cooperative Educational Services)

Abstract

Using generalizability (G-) theory and qualitative feedback analysis, this study evaluated the role of ChatGPT in enhancing English-as-a-foreign-language (EFL) writing assessments in classroom settings. The primary objectives were to assess the reliability of the holistic scores assigned to EFL essays by ChatGPT versions 3.5 and 4 compared to college English teachers and to evaluate the relevance of the qualitative feedback provided by these versions of ChatGPT. The study analyzed 30 College English Test Band 4 (CET-4) essays written by non-English majors at a university in Beijing, China. ChatGPT versions 3.5 and 4, along with four college English teachers, served as raters. They scored the essays holistically following the CET-4 scoring rubric and also provided qualitative feedback on the language, content, and organization of these essays. The G-theory analysis revealed that the scoring reliability of ChatGPT3.5 was consistently lower than that of the teacher raters; however, ChatGPT4 demonstrated consistently higher reliability coefficients than the teachers. The qualitative feedback analysis indicated that both ChatGPT3.5 and 4 consistently provided more relevant feedback on the EFL essays than the teacher raters. Furthermore, ChatGPT versions 3.5 and 4 were equally relevant across the language, content, and organization aspects of the essays, whereas the teacher raters generally focused more on language but provided less relevant feedback on content and organization. Consequently, ChatGPT versions 3.5 and 4 could be useful AI tools for enhancing EFL writing assessments in classroom settings. The implications of adopting ChatGPT for classroom writing assessments are discussed.

Suggested Citation

  • Junfei Li & Jinyan Huang & Wenyan Wu & Patrick B. Whipple, 2024. "Evaluating the role of ChatGPT in enhancing EFL writing assessments in classroom settings: A preliminary investigation," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-9, December.
    • Handle: RePEc:pal:palcom:v:11:y:2024:i:1:d:10.1057_s41599-024-03755-2
    DOI: 10.1057/s41599-024-03755-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1057/s41599-024-03755-2
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1057/s41599-024-03755-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhu Lei, 2017. "Salience of Student Written Feedback by Peer-Revision in EFL Writing Class," English Language Teaching, Canadian Center of Science and Education, vol. 10(12), pages 151-151, December.
    2. Jinyan Huang & Patrick B. Whipple, 2023. "Rater variability and reliability of constructed response questions in New York state high-stakes tests of English language arts and mathematics: implications for educational assessment policy," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-10, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.

      More about this item

      Statistics

      Access and download statistics

      Corrections

      All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pal:palcom:v:11:y:2024:i:1:d:10.1057_s41599-024-03755-2. See general information about how to correct material in RePEc.

      If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

      If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

      If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

      For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: https://www.nature.com/ .

      Please note that corrections may take a couple of weeks to filter through the various RePEc services.

      IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.