IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2404.19699.html
   My bibliography  Save this paper

Generative AI Usage and Exam Performance

Author

Listed:
  • Janik Ole Wecks
  • Johannes Voshaar
  • Benedikt Jost Plate
  • Jochen Zimmermann

Abstract

This study evaluates the impact of students' usage of generative artificial intelligence (GenAI) tools such as ChatGPT on their exam performance. We analyse student essays using GenAI detection systems to identify GenAI users among the cohort. Employing multivariate regression analysis, we find that students using GenAI tools score on average 6.71 (out of 100) points lower than non-users. While GenAI may offer benefits for learning and engagement, the way students actually use it correlates with diminished exam outcomes. Exploring the underlying mechanism, additional analyses show that the effect is particularly detrimental to students with high learning potential, suggesting an effect whereby GenAI tool usage hinders learning. Our findings provide important empirical evidence for the ongoing debate on the integration of GenAI in higher education and underscores the necessity for educators, institutions, and policymakers to carefully consider its implications for student performance.

Suggested Citation

  • Janik Ole Wecks & Johannes Voshaar & Benedikt Jost Plate & Jochen Zimmermann, 2024. "Generative AI Usage and Exam Performance," Papers 2404.19699, arXiv.org, revised Nov 2024.
    • Handle: RePEc:arx:papers:2404.19699
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2404.19699
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Brady D. Lund & Ting Wang & Nishith Reddy Mannuru & Bing Nie & Somipam Shimray & Ziang Wang, 2023. "ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(5), pages 570-581, May.
    2. Timm Teubner & Christoph M. Flath & Christof Weinhardt & Wil Aalst & Oliver Hinz, 2023. "Welcome to the Era of ChatGPT et al," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 65(2), pages 95-101, April.
    3. Tarun Jain & Mudit Kapoor, 2015. "The Impact of Study Groups and Roommates on Academic Performance," The Review of Economics and Statistics, MIT Press, vol. 97(1), pages 44-54, March.
    4. Husam Aldamen & Rajab Al-Esmail & Janice Hollindale, 2015. "Does Lecture Capturing Impact Student Performance and Attendance in an Introductory Accounting Course?," Accounting Education, Taylor & Francis Journals, vol. 24(4), pages 291-317, August.
    5. Hainmueller, Jens, 2012. "Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies," Political Analysis, Cambridge University Press, vol. 20(1), pages 25-46, January.
    6. Cecilia Chiu & Robyn King & Corene Crossin, 2023. "Using colour-coded digital annotation for enhanced case-based learning outcomes," Accounting Education, Taylor & Francis Journals, vol. 32(2), pages 201-221, March.
    7. Neil Hartnett & Jennifer Römcke & Christine Yap, 2004. "Student performance in tertiary‐level accounting: an international student focus," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 44(2), pages 163-185, July.
    8. Cheng, Peng & Ding, Rui, 2021. "The effect of online review exercises on student course engagement and learning performance: A case study of an introductory financial accounting course at an international joint venture university," Journal of Accounting Education, Elsevier, vol. 54(C).
    9. Tahani Z. Aldahdouh & Petri Nokelainen & Vesa Korhonen, 2020. "Technology and Social Media Usage in Higher Education: The Influence of Individual Innovativeness," SAGE Open, , vol. 10(1), pages 21582440198, January.
    10. Breusch, T S & Pagan, A R, 1979. "A Simple Test for Heteroscedasticity and Random Coefficient Variation," Econometrica, Econometric Society, vol. 47(5), pages 1287-1294, September.
    11. Wooldridge, Jeffrey M., 1995. "Selection corrections for panel data models under conditional mean independence assumptions," Journal of Econometrics, Elsevier, vol. 68(1), pages 115-132, July.
    12. Johannes Voshaar & Martin Knipp & Thomas Loy & Jochen Zimmermann & Florian Johannsen, 2023. "The impact of using a mobile app on learning success in accounting education," Accounting Education, Taylor & Francis Journals, vol. 32(2), pages 222-247, March.
    13. Yilmaz Guney, 2009. "Exogenous and Endogenous Factors Influencing Students' Performance in Undergraduate Accounting Modules," Accounting Education, Taylor & Francis Journals, vol. 18(1), pages 51-73.
    14. Minhyeok Lee, 2023. "A Mathematical Investigation of Hallucination and Creativity in GPT Models," Mathematics, MDPI, vol. 11(10), pages 1-17, May.
    15. Daryl Pregibon, 1980. "Goodness of Link Tests for Generalized Linear Models," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 29(1), pages 15-24, March.
    16. T. R. Wijesundara & Sun Xixiang, 2018. "Social Networking Sites Acceptance: The Role of Personal Innovativeness in Information Technology," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(8), pages 1-75, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jones, A.M, 2010. "Models For Health Care," Health, Econometrics and Data Group (HEDG) Working Papers 10/01, HEDG, c/o Department of Economics, University of York.
    2. William Greene, 2007. "Discrete Choice Modeling," Working Papers 07-6, New York University, Leonard N. Stern School of Business, Department of Economics.
    3. Céline Azémar & Rodolphe Desbordes, 2009. "Public Governance, Health and Foreign Direct Investment in Sub-Saharan Africa," Journal of African Economies, Centre for the Study of African Economies, vol. 18(4), pages 667-709, August.
    4. Pacula Rosalie Liccardo & Kilmer Beau & Grossman Michael & Chaloupka Frank J, 2010. "Risks and Prices: The Role of User Sanctions in Marijuana Markets," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 10(1), pages 1-38, February.
    5. Adrian Lüthge & Ulrich Pidun & Dodo zu Knyphausen-Aufseß, 2021. "Approximating relatedness from a business model perspective: towards a taxonomic approach," Review of Managerial Science, Springer, vol. 15(3), pages 813-846, April.
    6. Terence C. Cheng & Alfons Palangkaraya & Jongsay Yong, 2014. "Hospital utilization in mixed public--private system: evidence from Australian hospital data," Applied Economics, Taylor & Francis Journals, vol. 46(8), pages 859-870, March.
    7. Churyk, Natalie Tatiana & Eaton, Tim V. & Matuszewski, Linda J., 2024. "Accounting education literature review (2023)," Journal of Accounting Education, Elsevier, vol. 67(C).
    8. Montagud Mascarell, M. Dolores & Gandía Cabedo, Juan L., 2014. "Entorno virtual de aprendizaje y resultados académicos: evidencia empírica para la enseñanza de la Contabilidad de Gestión," Revista de Contabilidad - Spanish Accounting Review, Elsevier, vol. 17(2), pages 108-115.
    9. Elsa Marques & Sian Noble & Ashley W. Blom & William Hollingworth, 2014. "Disclosing Total Waiting Times For Joint Replacement: Evidence From The English Nhs Using Linked Hes Data," Health Economics, John Wiley & Sons, Ltd., vol. 23(7), pages 806-820, July.
    10. Ahmet Onay* & Serdar Benligiray, 2018. "Internal Factors Affecting Student Performance in Accounting Courses at a Vocational School," Sumerianz Journal of Economics and Finance, Sumerianz Publication, vol. 1(3), pages 82-90, 09-2018.
    11. Atkinson, Anthony C. & Riani, Marco & Corbellini, Aldo, 2021. "The box-cox transformation: review and extensions," LSE Research Online Documents on Economics 103537, London School of Economics and Political Science, LSE Library.
    12. Emine Yilmaz Karakoc, 2016. "The Role of Ethical Sensitivity and Self-Esteem on Academic Performance in Accounting Course," Eurasian Journal of Business and Management, Eurasian Publications, vol. 4(2), pages 95-105.
    13. Johannes Buggle & Thierry Mayer & Seyhun Orcan Sakalli & Mathias Thoenig, 2023. "The Refugee’s Dilemma: Evidence from Jewish Migration out of Nazi Germany," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(2), pages 1273-1345.
    14. Anika Reichert & Rowena Jacobs, 2018. "The impact of waiting time on patient outcomes: Evidence from early intervention in psychosis services in England," Health Economics, John Wiley & Sons, Ltd., vol. 27(11), pages 1772-1787, November.
    15. Leye Li & Louise Yi Lu & Dongyue Wang, 2022. "External labour market competitions and stock price crash risk: evidence from exposures to competitor CEOs’ award‐winning events," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 62(S1), pages 1421-1460, April.
    16. Mehzabin Tuli, Farzana & Mitra, Suman & Crews, Mariah B., 2021. "Factors influencing the usage of shared E-scooters in Chicago," Transportation Research Part A: Policy and Practice, Elsevier, vol. 154(C), pages 164-185.
    17. Balima, Hippolyte Weneyam, 2020. "Coups d’état and the cost of debt," Journal of Comparative Economics, Elsevier, vol. 48(3), pages 509-528.
    18. Inmaculada Garc�a-Mainar & V�ctor M. Montuenga-G�mez, 2017. "Subjective educational mismatch and signalling in Spain," Documentos de Trabajo dt2017-03, Facultad de Ciencias Económicas y Empresariales, Universidad de Zaragoza.
    19. Maria Casanova-Rivas, 2008. "Dynamic Complementarities: A Computational and Empirical Analysis of Couples' Retirement Decisions," 2008 Meeting Papers 1073, Society for Economic Dynamics.
    20. Hideki Murakami & Yukari Matsuse & Koji Mukaigawa & Yushi Tsunoda, 2013. "Product lifecycle and choice of transportation modes: Japan' s evidence of import and export," Discussion Papers 2013-28, Kobe University, Graduate School of Business Administration.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2404.19699. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.