IDEAS home Printed from https://ideas.repec.org/p/fip/fedkrw/97255.html
   My bibliography  Save this paper

Evaluating Local Language Models: An Application to Bank Earnings Calls

Author

Abstract

This study evaluates the performance of local large language models (LLMs) in interpreting financial texts, compared with closed-source, cloud-based models. We first introduce new benchmarking tasks for assessing LLM performance in analyzing financial and economic texts and explore the refinements needed to improve its performance. Our benchmarking results suggest local LLMs are a viable tool for general natural language processing analysis of these texts. We then leverage local LLMs to analyze the tone and substance of bank earnings calls in the post-pandemic era, including calls conducted during the banking stress of early 2023. We analyze remarks in bank earnings calls in terms of topics discussed, overall sentiment, temporal orientation, and vagueness. We find that after the banking stress in early 2023, banks tended to converge to a similar set of topics for discussion and to espouse a distinctly less positive sentiment.

Suggested Citation

  • Thomas R. Cook & Sophia Kazinnik & Anne Lundgaard Hansen & Peter McAdam, 2023. "Evaluating Local Language Models: An Application to Bank Earnings Calls," Research Working Paper RWP 23-12, Federal Reserve Bank of Kansas City.
    • Handle: RePEc:fip:fedkrw:97255
    as

    Download full text from publisher

    File URL: https://www.kansascityfed.org/Research%20Working%20Papers/documents/9862/rwp23-12cookkazinnikhansenmcadam.pdf
    File Function: Full Text
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. In-Koo Cho & David M. Kreps, 1987. "Signaling Games and Stable Equilibria," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 102(2), pages 179-221.
    2. De Amicis, Chiara & Falconieri, Sonia & Tastan, Mesut, 2021. "Sentiment analysis and gender differences in earnings conference calls," Journal of Corporate Finance, Elsevier, vol. 71(C).
    3. Alejandro Lopez-Lira & Yuehua Tang, 2023. "Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models," Papers 2304.07619, arXiv.org, revised Sep 2024.
    4. Chris Chatfield, 1995. "Model Uncertainty, Data Mining and Statistical Inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(3), pages 419-444, May.
    5. Pekka Malo & Ankur Sinha & Pekka Korhonen & Jyrki Wallenius & Pyry Takala, 2014. "Good debt or bad debt: Detecting semantic orientations in economic texts," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 782-796, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yuqi Nie & Yaxuan Kong & Xiaowen Dong & John M. Mulvey & H. Vincent Poor & Qingsong Wen & Stefan Zohren, 2024. "A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges," Papers 2406.11903, arXiv.org.
    2. Baptiste Lefort & Eric Benhamou & Jean-Jacques Ohana & David Saltiel & Beatrice Guez, 2024. "Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning," Papers 2409.11408, arXiv.org.
    3. Anders Gustafsson, 2019. "Busy doing nothing: why politicians implement inefficient policies," Constitutional Political Economy, Springer, vol. 30(3), pages 282-299, September.
    4. Cao, Melanie & Shi, Shouyong, 2006. "Signaling in the Internet craze of initial public offerings," Journal of Corporate Finance, Elsevier, vol. 12(4), pages 818-833, September.
    5. Thomas de Haan & Theo Offerman & Randolph Sloof, 2015. "Money Talks? An Experimental Investigation Of Cheap Talk And Burned Money," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 56(4), pages 1385-1426, November.
    6. Eduardo Perez & Delphine Prady, 2012. "Complicating to Persuade?," Working Papers hal-03583827, HAL.
    7. Chou, Ping & Chuang, Howard Hao-Chun & Chou, Yen-Chun & Liang, Ting-Peng, 2022. "Predictive analytics for customer repurchase: Interdisciplinary integration of buy till you die modeling and machine learning," European Journal of Operational Research, Elsevier, vol. 296(2), pages 635-651.
    8. Eduardo Perez-Richet, 2014. "Interim Bayesian Persuasion: First Steps," American Economic Review, American Economic Association, vol. 104(5), pages 469-474, May.
    9. V. Bhaskar & Caroline Thomas, 2019. "The Culture of Overconfidence," American Economic Review: Insights, American Economic Association, vol. 1(1), pages 95-110, June.
    10. Vaccari, Federico, 2023. "Competition in costly talk," Journal of Economic Theory, Elsevier, vol. 213(C).
    11. Riccardo (Jack) Lucchetti & Luca Pedini, 2020. "ParMA: Parallelised Bayesian Model Averaging for Generalised Linear Models," Working Papers 2020:28, Department of Economics, University of Venice "Ca' Foscari".
    12. , & ,, 2013. "Selection-free predictions in global games with endogenous information and multiple equilibria," Theoretical Economics, Econometric Society, vol. 8(3), September.
    13. Vladimirov, Vladimir, 2015. "Financing bidders in takeover contests," Journal of Financial Economics, Elsevier, vol. 117(3), pages 534-557.
    14. repec:hum:wpaper:sfb649dp2014-041 is not listed on IDEAS
    15. Miguel Ángel Ropero, 2021. "Entry deterrence when the potential entrant is your competitor in a different market," Southern Economic Journal, John Wiley & Sons, vol. 87(3), pages 1010-1030, January.
    16. Anton Bondarev & Beat Hintermann & Frank C. Krysiak & Ralph Winkler, 2017. "The Intricacy of Adapting to Climate Change: Flood Protection as a Local Public Goods Game," CESifo Working Paper Series 6382, CESifo.
    17. Kirtac, Kemal & Germano, Guido, 2024. "Sentiment trading with large language models," Finance Research Letters, Elsevier, vol. 62(PB).
    18. Dilmé, Francesc, 2019. "Dynamic quality signaling with hidden actions," Games and Economic Behavior, Elsevier, vol. 113(C), pages 116-136.
    19. Robert Lehmann & Antje Weyh, 2016. "Forecasting Employment in Europe: Are Survey Results Helpful?," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 12(1), pages 81-117, September.
    20. Bagwell, Kyle & Wolinsky, Asher, 2002. "Game theory and industrial organization," Handbook of Game Theory with Economic Applications, in: R.J. Aumann & S. Hart (ed.), Handbook of Game Theory with Economic Applications, edition 1, volume 3, chapter 49, pages 1851-1895, Elsevier.
    21. Baliga, Sandeep & Corchon, Luis C. & Sjostrom, Tomas, 1997. "The Theory of Implementation When the Planner Is a Player," Journal of Economic Theory, Elsevier, vol. 77(1), pages 15-33, November.

    More about this item

    Keywords

    data; large language models; quantitative methods; banking and finance;
    All these keywords.

    JEL classification:

    • C45 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Neural Networks and Related Topics
    • G21 - Financial Economics - - Financial Institutions and Services - - - Banks; Other Depository Institutions; Micro Finance Institutions; Mortgages

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedkrw:97255. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Zach Kastens (email available below). General contact details of provider: https://edirc.repec.org/data/frbkcus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.