IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2403.01770.html
   My bibliography  Save this paper

Experimenting with Generative AI: Does ChatGPT Really Increase Everyone's Productivity?

Author

Listed:
  • Voraprapa Nakavachara
  • Tanapong Potipiti
  • Thanee Chaiwat

Abstract

Generative AI technologies such as ChatGPT, Gemini, and MidJourney have made remarkable progress in recent years. Recent literature has documented ChatGPT's positive impact on productivity in areas where it has strong expertise, attributable to extensive training datasets, such as the English language and Python/SQL programming. However, there is still limited literature regarding ChatGPT's performance in areas where its capabilities could still be further enhanced. This paper aims to fill this gap. We conducted an experiment in which economics students were asked to perform writing analysis tasks in a non-English language (specifically, Thai) and math & data analysis tasks using a less frequently used programming package (specifically, Stata). The findings suggest that, on average, participants performed better using ChatGPT in terms of scores and time taken to complete the tasks. However, a detailed examination reveals that 34% of participants saw no improvement in writing analysis tasks, and 42% did not improve in math & data analysis tasks when employing ChatGPT. Further investigation indicated that higher-ability students, as proxied by their econometrics grades, were the ones who performed worse in writing analysis tasks when using ChatGPT. We also found evidence that students with better digital skills performed better with ChatGPT. This research provides insights on the impact of generative AI. Thus, stakeholders can make informed decisions to implement appropriate policy frameworks or redesign educational systems. It also highlights the critical role of human skills in addressing and complementing the limitations of technology.

Suggested Citation

  • Voraprapa Nakavachara & Tanapong Potipiti & Thanee Chaiwat, 2024. "Experimenting with Generative AI: Does ChatGPT Really Increase Everyone's Productivity?," Papers 2403.01770, arXiv.org.
    • Handle: RePEc:arx:papers:2403.01770
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2403.01770
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Lekfuangfu, Warn N. & Nakavachara, Voraprapa, 2021. "Reshaping Thailand's labor market: The intertwined forces of technology advancements and shifting supply chains," Economic Modelling, Elsevier, vol. 102(C).
    2. Alex Kim & Maximilian Muhn & Valeri Nikolaev, 2023. "From Transcripts to Insights: Uncovering Corporate Risks Using Generative AI," Papers 2310.17721, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alguacil, Maite & Lo Turco, Alessia & Martínez-Zarzoso, Inmaculada, 2022. "Robot adoption and export performance: Firm-level evidence from Spain," Economic Modelling, Elsevier, vol. 114(C).
    2. Yikai Zhao & Jun Nagayasu & Xinyi Geng, 2024. "Measuring Climate Policy Uncertainty with LLMs: New Insights into Corporate Bond Credit Spreads," DSSR Discussion Papers 143, Graduate School of Economics and Management, Tohoku University.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2403.01770. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.