IDEAS home Printed from https://ideas.repec.org/a/spr/reaccs/v29y2024i3d10.1007_s11142-024-09833-9.html
   My bibliography  Save this article

Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries

Author

Listed:
  • Marc Eulerich

    (University of Duisburg-Essen)

  • Aida Sanatizadeh

    (Northern Illinois University)

  • Hamid Vakilzadeh

    (University of Wisconsin – Whitewater)

  • David A. Wood

    (Brigham Young University)

Abstract

ChatGPT frequently appears in the media, with many predicting significant disruptions, especially in the fields of accounting and auditing. Yet research has demonstrated relatively poor performance of ChatGPT on student assessment questions. We extend this research to examine whether more recent ChatGPT models and capabilities can pass major accounting certification exams including the Certified Public Accountant (CPA), Certified Management Accountant (CMA), Certified Internal Auditor (CIA), and Enrolled Agent (EA) certification exams. We find that the ChatGPT 3.5 model cannot pass any exam (average score across all assessments of 53.1%). However, with additional enhancements, ChatGPT can pass all sections of each tested exam: moving to the ChatGPT 4 model improved scores by an average of 16.5%, providing 10-shot training improved scores an additional 6.6%, and allowing the model to use reasoning and acting (e.g., allow ChatGPT to use a calculator and other resources) improved scores an additional 8.9%. After all these improvements, ChatGPT passed all exams with an average score of 85.1%. This high performance indicates that ChatGPT has sufficient capabilities to disrupt the accounting and auditing industries, which we discuss in detail. This research provides practical insights for accounting professionals, investors, and stakeholders on how to adapt and mitigate the potential harms of this technology in accounting and auditing firms.

Suggested Citation

  • Marc Eulerich & Aida Sanatizadeh & Hamid Vakilzadeh & David A. Wood, 2024. "Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries," Review of Accounting Studies, Springer, vol. 29(3), pages 2318-2349, September.
    • Handle: RePEc:spr:reaccs:v:29:y:2024:i:3:d:10.1007_s11142-024-09833-9
    DOI: 10.1007/s11142-024-09833-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11142-024-09833-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11142-024-09833-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Anastassia Fedyk & James Hodson & Natalya Khimich & Tatiana Fedyk, 2022. "Is artificial intelligence improving the audit process?," Review of Accounting Studies, Springer, vol. 27(3), pages 938-985, September.
    2. Tyna Eloundou & Sam Manning & Pamela Mishkin & Daniel Rock, 2023. "GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models," Papers 2303.10130, arXiv.org, revised Aug 2023.
    3. Margaret H. Christ & Scott A. Emett & Scott L. Summers & David A. Wood, 2021. "Prepare for takeoff: improving asset measurement and audit quality with drone-enabled inventory audit procedures," Review of Accounting Studies, Springer, vol. 26(4), pages 1323-1343, December.
    4. Kreitmeir, David & Raschky, Paul Anton, 2023. "The Unintended Consequences of Censoring Digital Technology - Evidence from Italy's ChatGPT Ban," SocArXiv v3cgs, Center for Open Science.
    5. Huang, Feiqi & Vasarhelyi, Miklos A., 2019. "Applying robotic process automation (RPA) in auditing: A framework," International Journal of Accounting Information Systems, Elsevier, vol. 35(C).
    6. Shiva Rajgopal, 2021. "Integrating Practice into Accounting Research," Management Science, INFORMS, vol. 67(9), pages 5430-5454, September.
    7. Eddy Cardinaels & Stephan Hollander & Brian J. White, 2019. "Automatic summarization of earnings releases: attributes and effects on investors’ judgments," Review of Accounting Studies, Springer, vol. 24(3), pages 860-890, September.
    8. Geerts, Guido L., 2011. "A design science research methodology and its application to accounting information systems research," International Journal of Accounting Information Systems, Elsevier, vol. 12(2), pages 142-151.
    9. Prithwiraj Choudhury & Evan Starr & Rajshree Agarwal, 2020. "Machine learning and human capital complementarities: Experimental evidence on bias mitigation," Strategic Management Journal, Wiley Blackwell, vol. 41(8), pages 1381-1411, August.
    10. Cardinaels, Eddy & Hollander, Stephan & White, Brian, 2019. "Automatic summarization of earnings releases : Attributes and effects on investors’ judgments," Other publications TiSEM 721f64f4-033e-453b-a3e7-2, Tilburg University, School of Economics and Management.
    11. Siliang Tong & Nan Jia & Xueming Luo & Zheng Fang, 2021. "The Janus face of artificial intelligence feedback: Deployment versus disclosure effects on employee performance," Strategic Management Journal, Wiley Blackwell, vol. 42(9), pages 1600-1631, September.
    12. Wood, David A. & Achhpilia, Muskan P. & Adams, Mollie T. & Aghazadeh, Sanaz & Akinyele, Kazeem & Akpan, Mfon & Allee, Kristian D. & Allen, Abigail M. & Almer, Elizabeth D. & Ames, Daniel & Arity, Vikt, 2023. "The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions?," Other publications TiSEM b4a29b3e-18a1-4259-a70c-d, Tilburg University, School of Economics and Management.
    13. Kokina, Julia & Blanchette, Shay, 2019. "Early evidence of digital labor in accounting: Innovation with Robotic Process Automation," International Journal of Accounting Information Systems, Elsevier, vol. 35(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christoph Riedl & Eric Bogert, 2024. "Effects of AI Feedback on Learning, the Skill Gap, and Intellectual Diversity," Papers 2409.18660, arXiv.org.
    2. Bavaresco, Rodrigo Simon & Nesi, Luan Carlos & Victória Barbosa, Jorge Luis & Antunes, Rodolfo Stoffel & da Rosa Righi, Rodrigo & da Costa, Cristiano André & Vanzin, Mariangela & Dornelles, Daniel & J, 2023. "Machine learning-based automation of accounting services: An exploratory case study," International Journal of Accounting Information Systems, Elsevier, vol. 49(C).
    3. Milan Miric & Nan Jia & Kenneth G. Huang, 2023. "Using supervised machine learning for large‐scale classification in management research: The case for identifying artificial intelligence patents," Strategic Management Journal, Wiley Blackwell, vol. 44(2), pages 491-519, February.
    4. Li, Yunjian & Song, Yixiao & Sun, Yanming & Zeng, Mingzhuo, 2024. "When do employees learn from artificial intelligence? The moderating effects of perceived enjoyment and task-related complexity," Technology in Society, Elsevier, vol. 77(C).
    5. Emilio Abad-Segura & Mariana-Daniela González-Zamar, 2020. "Research Analysis on Emerging Technologies in Corporate Accounting," Mathematics, MDPI, vol. 8(9), pages 1-29, September.
    6. Vitali, Sonia & Giuliani, Marco, 2024. "Emerging digital technologies and auditing firms: Opportunities and challenges," International Journal of Accounting Information Systems, Elsevier, vol. 53(C).
    7. Flavio Calvino & Luca Fontanelli, 2023. "Artificial intelligence, complementary assets and productivity: evidence from French firms," LEM Papers Series 2023/35, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    8. Syaiful Anwar Mohamed & Moamin A. Mahmoud & Mohammed Najah Mahdi & Salama A. Mostafa, 2022. "Improving Efficiency and Effectiveness of Robotic Process Automation in Human Resource Management," Sustainability, MDPI, vol. 14(7), pages 1-18, March.
    9. Jean Robert Kala Kamdjoug & Hyacinthe Djanan Sando & Jules Raymond Kala & Arielle Ornela Ndassi Teutio & Sunil Tiwari & Samuel Fosso Wamba, 2024. "Data analytics-based auditing: a case study of fraud detection in the banking context," Annals of Operations Research, Springer, vol. 340(2), pages 1161-1188, September.
    10. Costa Diogo António da Silva & Mamede Henrique São & Mira da Silva Miguel, 2022. "Robotic Process Automation (RPA) Adoption: A Systematic Literature Review," Engineering Management in Production and Services, Sciendo, vol. 14(2), pages 1-12, June.
    11. Roman Šperka & Michal Halaška, 2023. "The performance assessment framework (PPAFR) for RPA implementation in a loan application process using process mining," Information Systems and e-Business Management, Springer, vol. 21(2), pages 277-321, June.
    12. Senave, Elseline & Jans, Mieke J. & Srivastava, Rajendra P., 2023. "The application of text mining in accounting," International Journal of Accounting Information Systems, Elsevier, vol. 50(C).
    13. Perdana, Arif & Lee, W. Eric & Mui Kim, Chu, 2023. "Prototyping and implementing Robotic Process Automation in accounting firms: Benefits, challenges and opportunities to audit automation," International Journal of Accounting Information Systems, Elsevier, vol. 51(C).
    14. Cassandra Estep & Emily E. Griffith & Nikki L. MacKenzie, 2024. "How do financial executives respond to the use of artificial intelligence in financial reporting and auditing?," Review of Accounting Studies, Springer, vol. 29(3), pages 2798-2831, September.
    15. Goodson, Brian M. & Grenier, Jonathan H. & Maksymov, Eldar, 2023. "When law students think like audit litigation attorneys: Implications for experimental research," Accounting, Organizations and Society, Elsevier, vol. 104(C).
    16. Czarnitzki, Dirk & Fernández, Gastón P. & Rammer, Christian, 2023. "Artificial intelligence and firm-level productivity," Journal of Economic Behavior & Organization, Elsevier, vol. 211(C), pages 188-205.
    17. Siliang Tong & Nan Jia & Xueming Luo & Zheng Fang, 2021. "The Janus face of artificial intelligence feedback: Deployment versus disclosure effects on employee performance," Strategic Management Journal, Wiley Blackwell, vol. 42(9), pages 1600-1631, September.
    18. Uklańska Anna, 2023. "Robotic Process Automation (RPA) – Bibliometric Analysis and Literature Review," Foundations of Management, Sciendo, vol. 15(1), pages 129-140, January.
    19. Carvajal, Daniel & Franco, Catalina & Isaksson, Siri, 2024. "Will Artificial Intelligence Get in the Way of Achieving Gender Equality?," Discussion Paper Series in Economics 3/2024, Norwegian School of Economics, Department of Economics, revised 31 Oct 2024.
    20. Carlos Alberto Peláez & Andrés Solano, 2023. "A Practice for the Design of Interactive Multimedia Experiences Based on Gamification: A Case Study in Elementary Education," Sustainability, MDPI, vol. 15(3), pages 1-26, January.

    More about this item

    Keywords

    Artificial intelligence; ChatGPT; Large language model; GPT 3.5; GTP 4; Accounting certification;
    All these keywords.

    JEL classification:

    • M4 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Accounting
    • M40 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Accounting - - - General
    • M42 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Accounting - - - Auditing
    • M49 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Accounting - - - Other

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:reaccs:v:29:y:2024:i:3:d:10.1007_s11142-024-09833-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.