IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v37y2021i1p186-204.html
   My bibliography  Save this article

Ranking professional forecasters by the predictive power of their narratives

Author

Listed:
  • Rybinski, Krzysztof

Abstract

This article presents the first ever ranking of professional forecasters based on the predictive power of the narrative of their regular research reports. The ranking is generated by applying the fully automated four-step procedure – called NLP-ForRank – developed in this article. The four steps are data scraping from the internet; data preparation; application of the natural language processing (NLP) models; and evaluation of the predictive power of the NLP indexes with linear regression, Granger causality, vector autoregression (VAR), and random forest forecasting models. Applying this procedure to five large Polish banks and to many time series shows that including the constructed NLP indexes in the forecasting models lowers the forecast errors, and that the optimal model almost always contains the NLP index. The financial news agencies could consider publishing this type of ranking on a regular basis as it would foster accountability, transparency, and a more competitive environment in the professional forecasting industry.

Suggested Citation

  • Rybinski, Krzysztof, 2021. "Ranking professional forecasters by the predictive power of their narratives," International Journal of Forecasting, Elsevier, vol. 37(1), pages 186-204.
    • Handle: RePEc:eee:intfor:v:37:y:2021:i:1:p:186-204
    DOI: 10.1016/j.ijforecast.2020.04.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207020300601
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2020.04.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Grimmer, Justin & Stewart, Brandon M., 2013. "Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts," Political Analysis, Cambridge University Press, vol. 21(3), pages 267-297, July.
    2. Joshua Zoen Git Hiew & Xin Huang & Hao Mou & Duan Li & Qi Wu & Yabo Xu, 2019. "BERT-based Financial Sentiment Index and LSTM-based Stock Return Predictability," Papers 1906.09024, arXiv.org, revised Jul 2022.
    3. Karol Szafranek & Aleksandra Hałka, 2019. "Determinants of Low Inflation in an Emerging, Small Open Economy through the Lens of Aggregated and Disaggregated Approach," Emerging Markets Finance and Trade, Taylor & Francis Journals, vol. 55(13), pages 3094-3111, October.
    4. David-Jan Jansen & Jakob de Haan, 2010. "An Assessment of the Consistency of ECB Communication using Wordscores," DNB Working Papers 259, Netherlands Central Bank, Research Department.
    5. Frantisek Brazdik & Michal Franta, 2017. "A BVAR Model for Forecasting of Czech Inflation," Working Papers 2017/7, Czech National Bank.
    6. Sims, Christopher A, 1980. "Macroeconomics and Reality," Econometrica, Econometric Society, vol. 48(1), pages 1-48, January.
    7. Adam Elbourne & Jakob de Haan, 2009. "Modeling Monetary Policy Transmission in Acceding Countries: Vector Autoregression Versus Structural Vector Autoregression," Emerging Markets Finance and Trade, Taylor & Francis Journals, vol. 45(2), pages 4-20, March.
    8. Ingrid E. Fisher & Margaret R. Garnsey & Mark E. Hughes, 2016. "Natural Language Processing in Accounting, Auditing and Finance: A Synthesis of the Literature with a Roadmap for Future Research," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 23(3), pages 157-214, July.
    9. Mirjana Pejić Bach & Živko Krstić & Sanja Seljan & Lejla Turulja, 2019. "Text Mining for Big Data Analysis in Financial Sector: A Literature Review," Sustainability, MDPI, vol. 11(5), pages 1-27, February.
    10. Tim Loughran & Bill Mcdonald, 2016. "Textual Analysis in Accounting and Finance: A Survey," Journal of Accounting Research, Wiley Blackwell, vol. 54(4), pages 1187-1230, September.
    11. Spyros Makridakis & Evangelos Spiliotis & Vassilios Assimakopoulos, 2018. "Statistical and Machine Learning forecasting methods: Concerns and ways forward," PLOS ONE, Public Library of Science, vol. 13(3), pages 1-26, March.
    12. Yuan Luo & William K. Thompson & Timothy M. Herr & Zexian Zeng & Mark A. Berendsen & Siddhartha R. Jonnalagadda & Matthew B. Carson & Justin Starren, 2017. "Natural Language Processing for EHR-Based Pharmacovigilance: A Structured Review," Drug Safety, Springer, vol. 40(11), pages 1075-1089, November.
    13. Rybinski, Krzysztof, 2020. "The forecasting power of the multi-language narrative of sell-side research: A machine learning evaluation," Finance Research Letters, Elsevier, vol. 34(C).
    14. Granger, C W J, 1969. "Investigating Causal Relations by Econometric Models and Cross-Spectral Methods," Econometrica, Econometric Society, vol. 37(3), pages 424-438, July.
    15. Ulrichs Magdalena, 2018. "Identification of Financial and Macroeconomic Shocks in a Var Model of the Polish Economy. A Stability Analysis," Economics and Business Review, Sciendo, vol. 4(1), pages 29-43, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Foltas, Alexander, 2023. "Quantifying priorities in business cycle reports: Analysis of recurring textual patterns around peaks and troughs," Working Papers 44, German Research Foundation's Priority Programme 1859 "Experience and Expectation. Historical Foundations of Economic Behaviour", Humboldt University Berlin.
    2. Tim Köhler & Jörg Döpke, 2023. "Will the last be the first? Ranking German macroeconomic forecasters based on different criteria," Empirical Economics, Springer, vol. 64(2), pages 797-832, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rybinski, Krzysztof, 2020. "The forecasting power of the multi-language narrative of sell-side research: A machine learning evaluation," Finance Research Letters, Elsevier, vol. 34(C).
    2. Berkin, Anil & Aerts, Walter & Van Caneghem, Tom, 2023. "Feasibility analysis of machine learning for performance-related attributional statements," International Journal of Accounting Information Systems, Elsevier, vol. 48(C).
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Senave, Elseline & Jans, Mieke J. & Srivastava, Rajendra P., 2023. "The application of text mining in accounting," International Journal of Accounting Information Systems, Elsevier, vol. 50(C).
    5. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    6. Bierens, H.J. & Broersma, L., 1991. "The relation between unemployment and interest rate : some international evidence," Serie Research Memoranda 0112, VU University Amsterdam, Faculty of Economics, Business Administration and Econometrics.
    7. Aaryan Gupta & Vinya Dengre & Hamza Abubakar Kheruwala & Manan Shah, 2020. "Comprehensive review of text-mining applications in finance," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 6(1), pages 1-25, December.
    8. Gossé, Jean-Baptiste & Guillaumin, Cyriac, 2013. "L’apport de la représentation VAR de Christopher A. Sims à la science économique," L'Actualité Economique, Société Canadienne de Science Economique, vol. 89(4), pages 309-319, Décembre.
    9. Agiakloglou, Christos & Gkouvakis, Michail, 2015. "Causal interrelations among market fundamentals: Evidence from the European Telecommunications sector," The Quarterly Review of Economics and Finance, Elsevier, vol. 55(C), pages 150-159.
    10. Bentour, El Mostafa, 2013. "Oil Prices, Drought Periods and Growth Forecasts in Morocco," MPRA Paper 52892, University Library of Munich, Germany.
    11. Marin, Dalia, 1992. "Is the Export-Led.Growth Hypothesis Valid for Industrialized Countries?," The Review of Economics and Statistics, MIT Press, vol. 74(4), pages 678-688, November.
    12. Carl Deschamps & Jan Mattijs, 2015. "Anatomy of a performance management system: the elusive path from targets to productivity," Working Papers CEB 15-037, ULB -- Universite Libre de Bruxelles.
    13. M. T. Alguacil & V. Orts, 2003. "Inward Foreign Direct Investment and Imports in Spain," International Economic Journal, Taylor & Francis Journals, vol. 17(3), pages 19-38.
    14. Ramona Dumitriu & Razvan Stefanescu, 2015. "The Relationship Between Romanian Exports And Economic Growth After The Adhesion To European Union," Risk in Contemporary Economy, "Dunarea de Jos" University of Galati, Faculty of Economics and Business Administration, pages 17-26.
    15. Dufour, Jean-Marie & Pelletier, Denis & Renault, Eric, 2006. "Short run and long run causality in time series: inference," Journal of Econometrics, Elsevier, vol. 132(2), pages 337-362, June.
    16. Gediminas Adomavicius & Jesse Bockstedt & Alok Gupta, 2012. "Modeling Supply-Side Dynamics of IT Components, Products, and Infrastructure: An Empirical Analysis Using Vector Autoregression," Information Systems Research, INFORMS, vol. 23(2), pages 397-417, June.
    17. David Bholat & Stephen Hans & Pedro Santos & Cheryl Schonhardt-Bailey, 2015. "Text mining for central banks," Handbooks, Centre for Central Banking Studies, Bank of England, number 33, April.
    18. Ericsson, Neil R & Hendry, David F & Mizon, Grayham E, 1998. "Exogeneity, Cointegration, and Economic Policy Analysis," Journal of Business & Economic Statistics, American Statistical Association, vol. 16(4), pages 370-387, October.
    19. Andre van Stel & Lendert Baljeu & Roy Thurik & Ingrid Verheul, 2006. "The Contribution of Business Ownership in Bringing Down Unemployment in Japan," Papers on Entrepreneurship, Growth and Public Policy 2006-05, Max Planck Institute of Economics, Entrepreneurship, Growth and Public Policy Group.
    20. Marek Rusnak & Tomas Havranek & Roman Horvath, 2013. "How to Solve the Price Puzzle? A Meta-Analysis," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 45(1), pages 37-70, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:37:y:2021:i:1:p:186-204. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.