IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2412.01069.html
   My bibliography  Save this paper

The Promise and Peril of Generative AI: Evidence from GPT-4 as Sell-Side Analysts

Author

Listed:
  • Edward Li
  • Zhiyuan Tu
  • Dexin Zhou

Abstract

We investigate how advanced large language models (LLMs), specifically GPT-4, process corporate disclosures to forecast earnings. Using earnings press releases issued around GPT-4's knowledge cutoff date, we address two questions: (1) Do GPT-generated earnings forecasts outperform analysts in accuracy? (2) How is GPT's performance related to its processing of textual and quantitative information? Our findings suggest that GPT forecasts are significantly less accurate than those of analysts. This underperformance can be traced to GPT's distinct textual and quantitative approaches: its textual processing follows a consistent, generalized pattern across firms, highlighting its strengths in language tasks. In contrast, its quantitative processing capabilities vary significantly across firms, revealing limitations tied to the uneven availability of domain-specific training data. Additionally, there is some evidence that GPT's forecast accuracy diminishes beyond its knowledge cutoff, underscoring the need to evaluate LLMs under hindsight-free conditions. Overall, this study provides a novel exploration of the "black box" of GPT-4's information processing, offering insights into LLMs' potential and challenges in financial applications.

Suggested Citation

  • Edward Li & Zhiyuan Tu & Dexin Zhou, 2024. "The Promise and Peril of Generative AI: Evidence from GPT-4 as Sell-Side Analysts," Papers 2412.01069, arXiv.org.
    • Handle: RePEc:arx:papers:2412.01069
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2412.01069
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Paul C. Tetlock & Maytal Saar‐Tsechansky & Sofus Macskassy, 2008. "More Than Words: Quantifying Language to Measure Firms' Fundamentals," Journal of Finance, American Finance Association, vol. 63(3), pages 1437-1467, June.
    2. S.P. Kothari & Eric So & Rodrigo Verdi, 2016. "Analysts’ Forecasts and Asset Pricing: A Survey," Annual Review of Financial Economics, Annual Reviews, vol. 8(1), pages 197-219, October.
    3. Alejandro Lopez-Lira & Yuehua Tang, 2023. "Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models," Papers 2304.07619, arXiv.org, revised Sep 2024.
    4. Tim Loughran & Bill Mcdonald, 2014. "Measuring Readability in Financial Disclosures," Journal of Finance, American Finance Association, vol. 69(4), pages 1643-1671, August.
    5. Xi Chen & Yang Ha (Tony) Cho & Yiwei Dou & Baruch Lev, 2022. "Predicting Future Earnings Changes Using Machine Learning and Detailed Financial Data," Journal of Accounting Research, Wiley Blackwell, vol. 60(2), pages 467-515, May.
    6. De Bondt, Werner F M & Thaler, Richard H, 1990. "Do Security Analysts Overreact?," American Economic Review, American Economic Association, vol. 80(2), pages 52-57, May.
    7. Julie Cotter & Irem Tuna & Peter D. Wysocki, 2006. "Expectations Management and Beatable Targets: How Do Analysts React to Explicit Earnings Guidance?," Contemporary Accounting Research, John Wiley & Sons, vol. 23(3), pages 593-624, September.
    8. Shuaiyu Chen & T. Clifton Green & Huseyin Gulen & Dexin Zhou, 2024. "What Does ChatGPT Make of Historical Stock Returns? Extrapolation and Miscalibration in LLM Stock Return Forecasts," Papers 2409.11540, arXiv.org.
    9. Tim Loughran & Bill Mcdonald, 2011. "When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10‐Ks," Journal of Finance, American Finance Association, vol. 66(1), pages 35-65, February.
    10. Li, Feng, 2008. "Annual report readability, current earnings, and earnings persistence," Journal of Accounting and Economics, Elsevier, vol. 45(2-3), pages 221-247, August.
    11. Bradshaw, Mark T. & Christensen, Theodore E. & Gee, Kurt H. & Whipple, Benjamin C., 2018. "Analysts’ GAAP earnings forecasts and their implications for accounting research," Journal of Accounting and Economics, Elsevier, vol. 66(1), pages 46-66.
    12. Harrison Hong & Jeffrey D. Kubik, 2003. "Analyzing the Analysts: Career Concerns and Biased Earnings Forecasts," Journal of Finance, American Finance Association, vol. 58(1), pages 313-351, February.
    13. Alex Kim & Maximilian Muhn & Valeri Nikolaev, 2023. "From Transcripts to Insights: Uncovering Corporate Risks Using Generative AI," Papers 2310.17721, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kumar, Rahul & Deb, Soumya Guha & Mukherjee, Shubhadeep, 2020. "Do words reveal the latent truth? Identifying communication patterns of corporate losers," Journal of Behavioral and Experimental Finance, Elsevier, vol. 26(C).
    2. Yicheng Wang & Brian Wright, 2024. "Tax-strategy-related words, firm’s ability, and tax avoidance," International Journal of Disclosure and Governance, Palgrave Macmillan, vol. 21(4), pages 685-713, December.
    3. Alex Kim & Maximilian Muhn & Valeri Nikolaev, 2024. "Financial Statement Analysis with Large Language Models," Papers 2407.17866, arXiv.org, revised Nov 2024.
    4. Ingrid E. Fisher & Margaret R. Garnsey & Mark E. Hughes, 2016. "Natural Language Processing in Accounting, Auditing and Finance: A Synthesis of the Literature with a Roadmap for Future Research," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 23(3), pages 157-214, July.
    5. Blankespoor, Elizabeth & deHaan, Ed & Marinovic, Iván, 2020. "Disclosure processing costs, investors’ information choice, and equity market outcomes: A review," Journal of Accounting and Economics, Elsevier, vol. 70(2).
    6. Cookson, J. Anthony & Moon, S. Katie & Noh, Joonki, 2020. "Imprecise and Informative: Lessons from Market Reactions to Imprecise Disclosure," SocArXiv akt2c, Center for Open Science.
    7. Michał Dzieliński & Alexander F. Wagner & Richard J. Zeckhauser, 2017. "Straight Talkers and Vague Talkers: The Effects of Managerial Style in Earnings Conference Calls," NBER Working Papers 23425, National Bureau of Economic Research, Inc.
    8. Zhao, Chen & Li, Yubin & Govindaraj, Suresh & Zhong, Zhaodong (Ken), 2022. "CDS trading and analyst optimism," The British Accounting Review, Elsevier, vol. 54(4).
    9. Nadine Gatzert & Dinah Heidinger, 2020. "An Empirical Analysis of Market Reactions to the First Solvency and Financial Condition Reports in the European Insurance Industry," Journal of Risk & Insurance, The American Risk and Insurance Association, vol. 87(2), pages 407-436, June.
    10. Abukari, Kobana & Dutta, Shantanu & Li, Chen & Tang, Songlian & Zhu, Pengcheng, 2024. "Corporate communication and likelihood of data breaches," International Review of Economics & Finance, Elsevier, vol. 94(C).
    11. Ramiro Losada, 2022. "La información pública periódica de los fondos de inversión: como influyen en las decisiones de los inversores," CNMV Documentos de Trabajo CNMV Documentos de Trabaj, CNMV- Comisión Nacional del Mercado de Valores - Departamento de Estudios y Estadísticas.
    12. Frankel, Richard & Jennings, Jared & Lee, Joshua, 2016. "Using unstructured and qualitative disclosures to explain accruals," Journal of Accounting and Economics, Elsevier, vol. 62(2), pages 209-227.
    13. Shuili Du & Kun Yu, 2021. "Do Corporate Social Responsibility Reports Convey Value Relevant Information? Evidence from Report Readability and Tone," Journal of Business Ethics, Springer, vol. 172(2), pages 253-274, August.
    14. Chau, Michael & Lin, Chih-Yung & Lin, Tse-Chun, 2020. "Wisdom of crowds before the 2007–2009 global financial crisis," Journal of Financial Stability, Elsevier, vol. 48(C).
    15. Benson, David F. & Brau, James C. & Cicon, James & Ferris, Stephen P., 2015. "Strategically camouflaged corporate governance in IPOs: Entrepreneurial masking and impression management," Journal of Business Venturing, Elsevier, vol. 30(6), pages 839-864.
    16. Rizzo, Emanuele, 2018. "Essays on corporate governance and the impact of regulation on financial markets," Other publications TiSEM b5158260-ea13-4763-b992-6, Tilburg University, School of Economics and Management.
    17. Tim Loughran & Bill Mcdonald, 2016. "Textual Analysis in Accounting and Finance: A Survey," Journal of Accounting Research, Wiley Blackwell, vol. 54(4), pages 1187-1230, September.
    18. Richard Frankel & Jared Jennings & Joshua Lee, 2022. "Disclosure Sentiment: Machine Learning vs. Dictionary Methods," Management Science, INFORMS, vol. 68(7), pages 5514-5532, July.
    19. Kalelkar, Rachana & Xu, Hongkang & Nguyen, Duong & Chen, Zheng, 2024. "Generalist CEOs and the readability of the 10-K report," Advances in accounting, Elsevier, vol. 65(C).
    20. Hitoshi Iwasaki & Ying Chen & Jun Tu, 2023. "Topic tones of analyst reports and stock returns: A deep learning approach," International Review of Finance, International Review of Finance Ltd., vol. 23(4), pages 831-858, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2412.01069. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.