IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2407.17624.html
   My bibliography  Save this paper

Traditional Methods Outperform Generative LLMs at Forecasting Credit Ratings

Author

Listed:
  • Felix Drinkall
  • Janet B. Pierrehumbert
  • Stefan Zohren

Abstract

Large Language Models (LLMs) have been shown to perform well for many downstream tasks. Transfer learning can enable LLMs to acquire skills that were not targeted during pre-training. In financial contexts, LLMs can sometimes beat well-established benchmarks. This paper investigates how well LLMs perform in the task of forecasting corporate credit ratings. We show that while LLMs are very good at encoding textual information, traditional methods are still very competitive when it comes to encoding numeric and multimodal data. For our task, current LLMs perform worse than a more traditional XGBoost architecture that combines fundamental and macroeconomic data with high-density text-based embedding features.

Suggested Citation

  • Felix Drinkall & Janet B. Pierrehumbert & Stefan Zohren, 2024. "Traditional Methods Outperform Generative LLMs at Forecasting Credit Ratings," Papers 2407.17624, arXiv.org.
    • Handle: RePEc:arx:papers:2407.17624
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2407.17624
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Taylor, Mark P. & Wang, Zigan & Xu, Qi, 2021. "The real effects of exchange rate risk on corporate investment: International evidence," Journal of International Money and Finance, Elsevier, vol. 117(C).
    2. Carling, Kenneth & Jacobson, Tor & Linde, Jesper & Roszbach, Kasper, 2007. "Corporate credit risk modeling and the macroeconomy," Journal of Banking & Finance, Elsevier, vol. 31(3), pages 845-868, March.
    3. Eleni Kalamara & Arthur Turrell & Chris Redl & George Kapetanios & Sujit Kapadia, 2022. "Making text count: Economic forecasting using newspaper text," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 896-919, August.
    4. Luca Barbaglia & Sergio Consoli & Sebastiano Manzan, 2023. "Forecasting with Economic News," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 41(3), pages 708-719, July.
    5. Wuyue An & Lin Wang & Yu‐Rong Zeng, 2023. "Text‐based soybean futures price forecasting: A two‐stage deep learning approach," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 312-330, March.
    6. Mai, Feng & Tian, Shaonan & Lee, Chihoon & Ma, Ling, 2019. "Deep learning models for bankruptcy prediction using textual disclosures," European Journal of Operational Research, Elsevier, vol. 274(2), pages 743-758.
    7. Romain Boulland & François Degeorge & Edith Ginglinger, 2017. "News Dissemination and Investor Attention," Review of Finance, European Finance Association, vol. 21(2), pages 761-791.
    8. Xinli Yu & Zheng Chen & Yuan Ling & Shujing Dong & Zongyi Liu & Yanbin Lu, 2023. "Temporal Data Meets LLM -- Explainable Financial Time Series Forecasting," Papers 2306.11025, arXiv.org.
    9. Mahsa Tavakoli & Rohitash Chandra & Fengrui Tian & Cristi'an Bravo, 2023. "Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams," Papers 2304.10740, arXiv.org, revised Nov 2024.
    10. Hyungjin Cho & Seung-Youb Han & Seungbin Oh & Hee-Yeon Sunwoo, 2020. "Optimistic credit rating and its influence on corporate decisions: evidence from Korea," Asia-Pacific Journal of Accounting & Economics, Taylor & Francis Journals, vol. 27(5), pages 612-629, September.
    11. Stevenson, Matthew & Mues, Christophe & Bravo, Cristián, 2021. "The value of text for small business default prediction: A Deep Learning approach," European Journal of Operational Research, Elsevier, vol. 295(2), pages 758-771.
    12. Baresa, Suzana & Bogdan, Sinisa & Ivanovic, Sasa, 2012. "Role, Interests And Critics Of Credit Rating Agencies," UTMS Journal of Economics, University of Tourism and Management, Skopje, Macedonia, vol. 3(1), pages 71-82.
    13. Yuqi Nie & Yaxuan Kong & Xiaowen Dong & John M. Mulvey & H. Vincent Poor & Qingsong Wen & Stefan Zohren, 2024. "A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges," Papers 2406.11903, arXiv.org.
    14. Xi Zhang & Yunjia Zhang & Senzhang Wang & Yuntao Yao & Binxing Fang & Philip S. Yu, 2018. "Improving Stock Market Prediction via Heterogeneous Information Fusion," Papers 1801.00588, arXiv.org.
    15. Tim Loughran & Bill Mcdonald, 2011. "When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10‐Ks," Journal of Finance, American Finance Association, vol. 66(1), pages 35-65, February.
    16. Yupeng Cao & Zhi Chen & Qingyun Pei & Fabrizio Dimino & Lorenzo Ausiello & Prashant Kumar & K. P. Subbalakshmi & Papa Momar Ndiaye, 2024. "RiskLabs: Predicting Financial Risk Using Large Language Model Based on Multi-Sources Data," Papers 2404.07452, arXiv.org.
    17. Minchae Song & Kyung‐shik Shin, 2019. "Forecasting economic indicators using a consumer sentiment index: Survey‐based versus text‐based data," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 38(6), pages 504-518, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Luca Barbaglia & Sergio Consoli & Sebastiano Manzan, 2024. "Forecasting GDP in Europe with textual data," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(2), pages 338-355, March.
    2. Christopher Gerling & Stefan Lessmann, 2023. "Multimodal Document Analytics for Banking Process Automation," Papers 2307.11845, arXiv.org, revised Nov 2023.
    3. Kriebel, Johannes & Stitz, Lennart, 2022. "Credit default prediction from user-generated text in peer-to-peer lending using deep learning," European Journal of Operational Research, Elsevier, vol. 302(1), pages 309-323.
    4. Katsafados, Apostolos G. & Leledakis, George N. & Pyrgiotakis, Emmanouil G. & Androutsopoulos, Ion & Fergadiotis, Manos, 2024. "Machine learning in bank merger prediction: A text-based approach," European Journal of Operational Research, Elsevier, vol. 312(2), pages 783-797.
    5. Julian Ashwin & Eleni Kalamara & Lorena Saiz, 2024. "Nowcasting Euro area GDP with news sentiment: A tale of two crises," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(5), pages 887-905, August.
    6. Zhao, Shuping & Xu, Kai & Wang, Zhao & Liang, Changyong & Lu, Wenxing & Chen, Bo, 2022. "Financial distress prediction by combining sentiment tone features," Economic Modelling, Elsevier, vol. 106(C).
    7. Martin Baumgaertner & Johannes Zahner, 2021. "Whatever it takes to understand a central banker - Embedding their words using neural networks," MAGKS Papers on Economics 202130, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    8. Mahsa Tavakoli & Rohitash Chandra & Fengrui Tian & Cristi'an Bravo, 2023. "Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams," Papers 2304.10740, arXiv.org, revised Nov 2024.
    9. Mario Sanz-Guerrero & Javier Arroyo, 2024. "Credit Risk Meets Large Language Models: Building a Risk Indicator from Loan Descriptions in P2P Lending," Papers 2401.16458, arXiv.org, revised Aug 2024.
    10. Shuangyan Li & Guangrui Wang & Yongli Luo, 2022. "Tone of language, financial disclosure, and earnings management: a textual analysis of form 20-F," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-24, December.
    11. Stolbov, Mikhail & Shchepeleva, Maria & Karminsky, Alexander, 2022. "When central bank research meets Google search: A sentiment index of global financial stress," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 81(C).
    12. Aprigliano, Valentina & Emiliozzi, Simone & Guaitoli, Gabriele & Luciani, Andrea & Marcucci, Juri & Monteforte, Libero, 2023. "The power of text-based indicators in forecasting Italian economic activity," International Journal of Forecasting, Elsevier, vol. 39(2), pages 791-808.
    13. Jiang, Cuiqing & Lyu, Ximei & Yuan, Yufei & Wang, Zhao & Ding, Yong, 2022. "Mining semantic features in current reports for financial distress prediction: Empirical evidence from unlisted public firms in China," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1086-1099.
    14. Shi, Yong & Qu, Yi & Chen, Zhensong & Mi, Yunlong & Wang, Yunong, 2024. "Improved credit risk prediction based on an integrated graph representation learning approach with graph transformation," European Journal of Operational Research, Elsevier, vol. 315(2), pages 786-801.
    15. Saiz, Lorena & Ashwin, Julian & Kalamara, Eleni, 2021. "Nowcasting euro area GDP with news sentiment: a tale of two crises," Working Paper Series 2616, European Central Bank.
    16. Kamesh Korangi & Christophe Mues & Cristi'an Bravo, 2021. "A transformer-based model for default prediction in mid-cap corporate markets," Papers 2111.09902, arXiv.org, revised Apr 2023.
    17. Nyman, Rickard & Kapadia, Sujit & Tuckett, David, 2021. "News and narratives in financial systems: Exploiting big data for systemic risk assessment," Journal of Economic Dynamics and Control, Elsevier, vol. 127(C).
    18. Zhang, Xiaotao & Li, Guoran & Li, Yishuo & Zou, Gaofeng & Wu, Ji George, 2023. "Which is more important in stock market forecasting: Attention or sentiment?," International Review of Financial Analysis, Elsevier, vol. 89(C).
    19. Hanjo Odendaal, 2021. "A machine learning approach to domain specific dictionary generation. An economic time series framework," Working Papers 06/2021, Stellenbosch University, Department of Economics.
    20. Korangi, Kamesh & Mues, Christophe & Bravo, Cristián, 2023. "A transformer-based model for default prediction in mid-cap corporate markets," European Journal of Operational Research, Elsevier, vol. 308(1), pages 306-320.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2407.17624. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.