IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2301.03136.html
   My bibliography  Save this paper

Removing Non-Stationary Knowledge From Pre-Trained Language Models for Entity-Level Sentiment Classification in Finance

Author

Listed:
  • Guijin Son
  • Hanwool Lee
  • Nahyeon Kang
  • Moonjeong Hahm

Abstract

Extraction of sentiment signals from news text, stock message boards, and business reports, for stock movement prediction, has been a rising field of interest in finance. Building upon past literature, the most recent works attempt to better capture sentiment from sentences with complex syntactic structures by introducing aspect-level sentiment classification (ASC). Despite the growing interest, however, fine-grained sentiment analysis has not been fully explored in non-English literature due to the shortage of annotated finance-specific data. Accordingly, it is necessary for non-English languages to leverage datasets and pre-trained language models (PLM) of different domains, languages, and tasks to best their performance. To facilitate finance-specific ASC research in the Korean language, we build KorFinASC, a Korean aspect-level sentiment classification dataset for finance consisting of 12,613 human-annotated samples, and explore methods of intermediate transfer learning. Our experiments indicate that past research has been ignorant towards the potentially wrong knowledge of financial entities encoded during the training phase, which has overestimated the predictive power of PLMs. In our work, we use the term "non-stationary knowledge'' to refer to information that was previously correct but is likely to change, and present "TGT-Masking'', a novel masking pattern to restrict PLMs from speculating knowledge of the kind. Finally, through a series of transfer learning with TGT-Masking applied we improve 22.63% of classification accuracy compared to standalone models on KorFinASC.

Suggested Citation

  • Guijin Son & Hanwool Lee & Nahyeon Kang & Moonjeong Hahm, 2023. "Removing Non-Stationary Knowledge From Pre-Trained Language Models for Entity-Level Sentiment Classification in Finance," Papers 2301.03136, arXiv.org, revised Jan 2023.
  • Handle: RePEc:arx:papers:2301.03136
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2301.03136
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Sanjiv R. Das & Mike Y. Chen, 2007. "Yahoo! for Amazon: Sentiment Extraction from Small Talk on the Web," Management Science, INFORMS, vol. 53(9), pages 1375-1388, September.
    2. Huang, Dashan & Li, Jiangyuan & Wang, Liyao & Zhou, Guofu, 2020. "Time series momentum: Is it there?," Journal of Financial Economics, Elsevier, vol. 135(3), pages 774-794.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pan, Zhiyuan & Zhong, Hao & Wang, Yudong & Huang, Juan, 2024. "Forecasting oil futures returns with news," Energy Economics, Elsevier, vol. 134(C).
    2. Papapostolou, Nikos C. & Pouliasis, Panos K. & Nomikos, Nikos K. & Kyriakou, Ioannis, 2016. "Shipping investor sentiment and international stock return predictability," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 96(C), pages 81-94.
    3. Tsukioka, Yasutomo & Yanagi, Junya & Takada, Teruko, 2018. "Investor sentiment extracted from internet stock message boards and IPO puzzles," International Review of Economics & Finance, Elsevier, vol. 56(C), pages 205-217.
    4. Onishchenko, Olena & Zhao, Jing & Kongahawatte, Sampath & Kuruppuarachchi, Duminda, 2024. "Investor heterogeneity and anchoring-induced momentum," Journal of Behavioral and Experimental Finance, Elsevier, vol. 42(C).
    5. Jiao Ji & Oleksandr Talavera & Shuxing Yin, 2018. "The Hidden Information Content: Evidence from the Tone of Independent Director Reports," Working Papers 2018-28, Swansea University, School of Management.
    6. Christopher N. Avery & Judith A. Chevalier & Richard J. Zeckhauser, 2016. "The "CAPS" Prediction System and Stock Market Returns," Review of Finance, European Finance Association, vol. 20(4), pages 1363-1381.
    7. Chen, Cathy Yi-Hsuan & Fengler, Matthias R. & Härdle, Wolfgang Karl & Liu, Yanchu, 2022. "Media-expressed tone, option characteristics, and stock return predictability," Journal of Economic Dynamics and Control, Elsevier, vol. 134(C).
    8. Mamdouh Medhat & Maik Schmeling, 2022. "Short-term Momentum," The Review of Financial Studies, Society for Financial Studies, vol. 35(3), pages 1480-1526.
    9. Yang-Cheng Lu & Yu-Chen Wei, 2013. "The Chinese News Sentiment around Earnings Announcements," Journal for Economic Forecasting, Institute for Economic Forecasting, vol. 0(3), pages 44-58, October.
    10. Khim-Yong Goh & Cheng-Suang Heng & Zhijie Lin, 2013. "Social Media Brand Community and Consumer Behavior: Quantifying the Relative Impact of User- and Marketer-Generated Content," Information Systems Research, INFORMS, vol. 24(1), pages 88-107, March.
    11. Ying Zhang & Peggy Swanson, 2010. "Are day traders bias free?—evidence from internet stock message boards," Journal of Economics and Finance, Springer;Academy of Economics and Finance, vol. 34(1), pages 96-112, January.
    12. Thomas Renault, 2020. "Sentiment analysis and machine learning in finance: a comparison of methods and models on one million messages," Digital Finance, Springer, vol. 2(1), pages 1-13, September.
    13. Zheng Yan & Wenqian Robertson & Yaosheng Lou & Tom W. Robertson & Sung Yong Park, 2021. "Finding leading scholars in mobile phone behavior: a mixed-method analysis of an emerging interdisciplinary field," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9499-9517, December.
    14. Sudeep Bhatia & Lukasz Walasek & Paul Slovic & Howard Kunreuther, 2021. "The More Who Die, the Less We Care: Evidence from Natural Language Analysis of Online News Articles and Social Media Posts," Risk Analysis, John Wiley & Sons, vol. 41(1), pages 179-203, January.
    15. Sihvonen, Markus, 2021. "Yield curve momentum," Research Discussion Papers 15/2021, Bank of Finland.
    16. Domonkos F. Vamossy, 2020. "Investor Emotions and Earnings Announcements," Papers 2006.13934, arXiv.org, revised Jun 2020.
    17. Tobias Wiest, 2023. "Momentum: what do we know 30 years after Jegadeesh and Titman’s seminal paper?," Financial Markets and Portfolio Management, Springer;Swiss Society for Financial Market Research, vol. 37(1), pages 95-114, March.
    18. Ahmad, Khurshid & Han, JingGuang & Hutson, Elaine & Kearney, Colm & Liu, Sha, 2016. "Media-expressed negative tone and firm-level stock returns," Journal of Corporate Finance, Elsevier, vol. 37(C), pages 152-172.
    19. Paul Brockman & Jim Cicon, 2013. "The Information Content Of Management Earnings Forecasts: An Analysis Of Hard Versus Soft Information," Journal of Financial Research, Southern Finance Association;Southwestern Finance Association, vol. 36(2), pages 147-174, June.
    20. Geng, Yuedan & Ye, Qiang & Jin, Yu & Shi, Wen, 2022. "Crowd wisdom and internet searches: What happens when investors search for stocks?," International Review of Financial Analysis, Elsevier, vol. 82(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2301.03136. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.