IDEAS home Printed from https://ideas.repec.org/a/eee/finlet/v58y2023ipas1544612323007055.html
   My bibliography  Save this article

GPT has become financially literate: Insights from financial literacy tests of GPT and a preliminary test of how people use it as a source of advice

Author

Listed:
  • Niszczota, Paweł
  • Abbas, Sami

Abstract

We assess the ability of GPT–a large language model–to serve as a financial robo-advisor for the masses, by using a financial literacy test. Davinci and ChatGPT based on GPT-3.5 score 66% and 65% on the financial literacy test, respectively, compared to a baseline of 33%. However, ChatGPT based on GPT-4 achieves a near-perfect 99% score, pointing to financial literacy becoming an emergent ability of state-of-the-art models. We use the Judge-Advisor System and a savings dilemma to illustrate how researchers might assess advice-utilization from large language models. We also present a number of directions for future research.

Suggested Citation

  • Niszczota, Paweł & Abbas, Sami, 2023. "GPT has become financially literate: Insights from financial literacy tests of GPT and a preliminary test of how people use it as a source of advice," Finance Research Letters, Elsevier, vol. 58(PA).
    • Handle: RePEc:eee:finlet:v:58:y:2023:i:pa:s1544612323007055
    DOI: 10.1016/j.frl.2023.104333
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1544612323007055
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.frl.2023.104333?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Aileen Heinberg & Angela Hung & Arie Kapteyn & Annamaria Lusardi & Anya Savikhin Samek & Joanne Yoong, 2014. "Five steps to planning success: experimental evidence from US households," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 30(4), pages 697-724.
    2. Disney, Richard & Gathergood, John & Weber, Jörg, 2015. "Credit counseling: a substitute for consumer financial literacy?," Journal of Pension Economics and Finance, Cambridge University Press, vol. 14(4), pages 466-491, October.
    3. van Rooij, Maarten & Lusardi, Annamaria & Alessie, Rob, 2011. "Financial literacy and stock market participation," Journal of Financial Economics, Elsevier, vol. 101(2), pages 449-472, August.
    4. Sniezek, Janet A. & Buckley, Timothy, 1995. "Cueing and Cognitive Conflict in Judge-Advisor Decision Making," Organizational Behavior and Human Decision Processes, Elsevier, vol. 62(2), pages 159-174, May.
    5. Francesco D’Acunto & Nagpurnanand Prabhala & Alberto G Rossi, 2019. "The Promises and Pitfalls of Robo-Advising," The Review of Financial Studies, Society for Financial Studies, vol. 32(5), pages 1983-2020.
    6. Ida Chak & Karen Croxson & Francesco D’Acunto & Jonathan Reuter & Alberto G. Rossi & Jonathan M. Shaw, 2022. "Improving Household Debt Management with Robo-Advice," NBER Working Papers 30616, National Bureau of Economic Research, Inc.
    7. Dowling, Michael & Lucey, Brian, 2023. "ChatGPT for (Finance) research: The Bananarama Conjecture," Finance Research Letters, Elsevier, vol. 53(C).
    8. Isaia, Eleonora & Oggero, Noemi, 2022. "The potential use of robo-advisors among the young generation: Evidence from Italy," Finance Research Letters, Elsevier, vol. 48(C).
    9. Anton Korinek, 2023. "Language Models and Cognitive Automation for Economic Research," NBER Working Papers 30957, National Bureau of Economic Research, Inc.
    10. Harvey, Nigel & Fischer, Ilan, 1997. "Taking Advice: Accepting Help, Improving Judgment, and Sharing Responsibility," Organizational Behavior and Human Decision Processes, Elsevier, vol. 70(2), pages 117-133, May.
    11. David R. Lewis, 2018. "The perils of overconfidence: Why many consumers fail to seek advice when they really should," Journal of Financial Services Marketing, Palgrave Macmillan, vol. 23(2), pages 104-111, June.
    12. Paweł Niszczota & Dániel Kaszás, 2020. "Robo-investment aversion," PLOS ONE, Public Library of Science, vol. 15(9), pages 1-19, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dong, Mengming Michael & Stratopoulos, Theophanis C. & Wang, Victor Xiaoqi, 2024. "A scoping review of ChatGPT research in accounting and finance," International Journal of Accounting Information Systems, Elsevier, vol. 55(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pawe{l} Niszczota & Sami Abbas, 2023. "GPT has become financially literate: Insights from financial literacy tests of GPT and a preliminary test of how people use it as a source of advice," Papers 2309.00649, arXiv.org, revised Sep 2024.
    2. Luis Oberrauch & Tim Kaiser, 2024. "Financial Education or Incentivizing Learning-by-Doing? Evidence from an RCT with Undergraduate Students," CESifo Working Paper Series 11187, CESifo.
    3. Alexia GAUDEUL & Caterina GIANNETTI, 2023. "Trade-offs in the design of financial algorithms," Discussion Papers 2023/288, Dipartimento di Economia e Management (DEM), University of Pisa, Pisa, Italy.
    4. Oehler, Andreas & Horn, Matthias, 2024. "Does ChatGPT provide better advice than robo-advisors?," Finance Research Letters, Elsevier, vol. 60(C).
    5. Sandro Ambuehl & B. Douglas Bernheim & Annamaria Lusardi, 2022. "Evaluating Deliberative Competence: A Simple Method with an Application to Financial Choice," American Economic Review, American Economic Association, vol. 112(11), pages 3584-3626, November.
    6. Niszczota, Paweł & Białek, Michał, 2021. "Women oppose sin stocks more than men do," Finance Research Letters, Elsevier, vol. 41(C).
    7. Patt, Anthony G. & Bowles, Hannah Riley & Cash, David W., 2006. "Mechanisms for Enhancing the Credibility of an Adviser: Prepayment and Aligned Incentives," Working Paper Series rwp06-010, Harvard University, John F. Kennedy School of Government.
    8. Gino, Francesca, 2008. "Do we listen to advice just because we paid for it? The impact of advice cost on its use," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 234-245, November.
    9. Logg, Jennifer M. & Minson, Julia A. & Moore, Don A., 2019. "Algorithm appreciation: People prefer algorithmic to human judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 151(C), pages 90-103.
    10. Alison Wood Brooks & Francesca Gino & Maurice E. Schweitzer, 2015. "Smart People Ask for (My) Advice: Seeking Advice Boosts Perceptions of Competence," Management Science, INFORMS, vol. 61(6), pages 1421-1435, June.
    11. Lusardi, Annamaria & Samek, Anya & Kapteyn, Arie & Glinert, Lewis & Hung, Angela & Heinberg, Aileen, 2017. "Visual tools and narratives: new ways to improve financial literacy," Journal of Pension Economics and Finance, Cambridge University Press, vol. 16(3), pages 297-323, July.
    12. Bonaccio, Silvia & Dalal, Reeshad S., 2006. "Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences," Organizational Behavior and Human Decision Processes, Elsevier, vol. 101(2), pages 127-151, November.
    13. Felix Holzmeister & Martin Holmén & Michael Kirchler & Matthias Stefan & Erik Wengström, 2023. "Delegation Decisions in Finance," Management Science, INFORMS, vol. 69(8), pages 4828-4844, August.
    14. Palmeira, Mauricio, 2020. "Advice in the presence of external cues: The impact of conflicting judgments on perceptions of expertise," Organizational Behavior and Human Decision Processes, Elsevier, vol. 156(C), pages 82-96.
    15. Jodlbauer, Barbara & Jonas, Eva, 2011. "Forecasting clients' reactions: How does the perception of strategic behavior influence the acceptance of advice?," International Journal of Forecasting, Elsevier, vol. 27(1), pages 121-133, January.
    16. Tim Kaiser & Lukas Menkhoff, 2017. "Does Financial Education Impact Financial Literacy and Financial Behavior, and If So, When?," The World Bank Economic Review, World Bank, vol. 31(3), pages 611-630.
    17. Brunen, Ann-Christine & Laubach, Oliver, 2022. "Do sustainable consumers prefer socially responsible investments? A study among the users of robo advisors," Journal of Banking & Finance, Elsevier, vol. 136(C).
    18. Oscar A. Stolper & Andreas Walter, 2017. "Financial literacy, financial advice, and financial behavior," Journal of Business Economics, Springer, vol. 87(5), pages 581-643, July.
    19. Mahapatra, Krushna & Nair, Gireesh & Gustavsson, Leif, 2011. "Swedish energy advisers' perceptions regarding and suggestions for fulfilling homeowner expectations," Energy Policy, Elsevier, vol. 39(7), pages 4264-4273, July.
    20. Alonso-Robisco, Andres & Carbó, José Manuel, 2023. "Analysis of CBDC narrative by central banks using large language models," Finance Research Letters, Elsevier, vol. 58(PC).

    More about this item

    Keywords

    Financial literacy; Robo-advice; Financial advice; Advice-utilization; Large language model; ChatGPT;
    All these keywords.

    JEL classification:

    • D14 - Microeconomics - - Household Behavior - - - Household Saving; Personal Finance
    • G11 - Financial Economics - - General Financial Markets - - - Portfolio Choice; Investment Decisions
    • G53 - Financial Economics - - Household Finance - - - Financial Literacy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:finlet:v:58:y:2023:i:pa:s1544612323007055. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/frl .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.