IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_11002.html
   My bibliography  Save this paper

From Helping Hand to Stumbling Block: The ChatGPT Paradox in Competency Experiment

Author

Listed:
  • Duk Gyoo Kim
  • Ahram Moon

Abstract

We ran a controlled laboratory experiment to examine whether ChatGPT’s aid can increase the participants’ performance in three different—reading and writing, mathematical problem-solving, and computational thinking—tasks. We find that the math score significantly decreases with ChatGPT’s assistance. This result is mainly because the low-ability subjects couldn’t discern the hallucinated answers with the correct ones, and it contests the general idea that ChatGPT can complement the workers with less expertise.

Suggested Citation

  • Duk Gyoo Kim & Ahram Moon, 2024. "From Helping Hand to Stumbling Block: The ChatGPT Paradox in Competency Experiment," CESifo Working Paper Series 11002, CESifo.
    • Handle: RePEc:ces:ceswps:_11002
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp11002.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Maarten Goos & Alan Manning & Anna Salomons, 2014. "Explaining Job Polarization: Routine-Biased Technological Change and Offshoring," American Economic Review, American Economic Association, vol. 104(8), pages 2509-2526, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tommaso AGASISTI & Geraint JOHNES & Marco PACCAGNELLA, 2021. "Tasks, occupations and wages in OECD countries," International Labour Review, International Labour Organization, vol. 160(1), pages 85-112, March.
    2. Baird, Matthew D. & Engberg, John & Gutierrez, Italo A., 2022. "RCT evidence on differential impact of US job training programmes by pre-training employment status," Labour Economics, Elsevier, vol. 75(C).
    3. Werner Eichhorst & Ulf Rinne, 2017. "Digital Challenges for the Welfare State," CESifo Forum, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 18(04), pages 03-08, December.
    4. Keller, Wolfgang & Utar, Hale, 2023. "International trade and job polarization: Evidence at the worker level," Journal of International Economics, Elsevier, vol. 145(C).
    5. Rabensteiner, Thomas & Guschanski, Alexander, 2022. "Autonomy and wage divergence: evidence from European survey data," Greenwich Papers in Political Economy 37925, University of Greenwich, Greenwich Political Economy Research Centre.
    6. Jacob Rubæk Holm & Christian Richter Østergaard, 2018. "The high importance of de-industrialization and job polarization for regional diversification," Papers in Evolutionary Economic Geography (PEEG) 1821, Utrecht University, Department of Human Geography and Spatial Planning, Group Economic Geography, revised May 2018.
    7. Michele Cantarella & Chiara Strozzi, 2021. "Workers in the crowd: the labor market impact of the online platform economy [An evaluation of instrumental variable strategies for estimating the effects of catholic schooling]," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 30(6), pages 1429-1458.
    8. repec:gdk:wpaper:51 is not listed on IDEAS
    9. Battisti, Michele & Gatto, Massimo Del & Parmeter, Christopher F., 2022. "Skill-biased technical change and labor market inefficiency," Journal of Economic Dynamics and Control, Elsevier, vol. 139(C).
    10. Antje Mertens & Laura Romeu-Gordo, 2023. "Retirement in Western Germany – How Workplace Tasks Influence Its Timing," Work, Employment & Society, British Sociological Association, vol. 37(2), pages 467-485, April.
    11. de la Vega, Pablo & Porto, Natalia & Cerimelo, Manuela, 2024. "Going green: estimating the potential of green jobs in Argentina," Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 58, pages 1-1.
    12. Lütkenhorst, Wilfried, 2018. "Creating wealth without labour? Emerging contours of a new techno-economic landscape," IDOS Discussion Papers 11/2018, German Institute of Development and Sustainability (IDOS).
    13. Parolin, Zachary & Schmitt, Rafael Pintro & Esping-Andersen, Gøsta & Fallesen, Peter, 2023. "The Intergenerational Persistence of Poverty in High-Income Countries," IZA Discussion Papers 16194, Institute of Labor Economics (IZA).
    14. Furtado, Delia & Kong, Haiyang, 2021. "How Do Low-Skilled Immigrants Adjust to Chinese Import Shocks? Evidence Using English Language Proficiency," IZA Discussion Papers 14152, Institute of Labor Economics (IZA).
    15. Caitlin Allen Whitehead & Haroon Bhorat & Robert Hill & Tim Köhler & François Steenkamp, 2021. "The Potential Employment Implications of the Fourth Industrial Revolution Technologies: The Case of the Manufacturing, Engineering and Related Services Sector," Working Papers 202106, University of Cape Town, Development Policy Research Unit.
    16. Zhou, Yixiao & Tyers, Rod, 2019. "Automation and inequality in China," China Economic Review, Elsevier, vol. 58(C).
    17. Stephan Brunow & Stefanie Lösch & Ostap Okhrin, 2022. "Labor market tightness and individual wage growth: evidence from Germany," Journal for Labour Market Research, Springer;Institute for Employment Research/ Institut für Arbeitsmarkt- und Berufsforschung (IAB), vol. 56(1), pages 1-21, December.
    18. David Deming & Lisa B. Kahn, 2018. "Skill Requirements across Firms and Labor Markets: Evidence from Job Postings for Professionals," Journal of Labor Economics, University of Chicago Press, vol. 36(S1), pages 337-369.
    19. Thiemo Fetzer & Stephan Kyburz, 2018. "Cohesive Institutions and Political Violence," HiCN Working Papers 271, Households in Conflict Network.
    20. T. Gries & R. Grundmann & I. Palnau & M. Redlin, 2017. "Innovations, growth and participation in advanced economies - a review of major concepts and findings," International Economics and Economic Policy, Springer, vol. 14(2), pages 293-351, April.
    21. Vahagn Jerbashian, 2019. "Automation and Job Polarization: On the Decline of Middling Occupations in Europe," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 81(5), pages 1095-1116, October.

    More about this item

    Keywords

    laboratory experiment; ChatGPT; labor productivity;
    All these keywords.

    JEL classification:

    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_11002. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.