IDEAS home Printed from https://ideas.repec.org/a/spr/fininn/v9y2023i1d10.1186_s40854-023-00497-z.html
   My bibliography  Save this article

Robust monitoring machine: a machine learning solution for out-of-sample R $$^2$$ 2 -hacking in return predictability monitoring

Author

Listed:
  • James Yae

    (University of Houston)

  • Yang Luo

    (University of Houston)

Abstract

The out-of-sample $$R^2$$ R 2 is designed to measure forecasting performance without look-ahead bias. However, researchers can hack this performance metric even without multiple tests by constructing a prediction model using the intuition derived from empirical properties that appear only in the test sample. Using ensemble machine learning techniques, we create a virtual environment that prevents researchers from peeking into the intuition in advance when performing out-of-sample prediction simulations. We apply this approach to robust monitoring, exploiting a dynamic shrinkage effect by switching between a proposed forecast and a benchmark. Considering stock return forecasting as an example, we show that the resulting robust monitoring forecast improves the average performance of the proposed forecast by 15% (in terms of mean-squared-error) and reduces the variance of its relative performance by 46% while avoiding the out-of-sample $$R^2$$ R 2 -hacking problem. Our approach, as a final touch, can further enhance the performance and stability of forecasts from any models and methods.

Suggested Citation

  • James Yae & Yang Luo, 2023. "Robust monitoring machine: a machine learning solution for out-of-sample R $$^2$$ 2 -hacking in return predictability monitoring," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 9(1), pages 1-28, December.
  • Handle: RePEc:spr:fininn:v:9:y:2023:i:1:d:10.1186_s40854-023-00497-z
    DOI: 10.1186/s40854-023-00497-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1186/s40854-023-00497-z
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1186/s40854-023-00497-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    2. Abel, Andrew B, 1990. "Asset Prices under Habit Formation and Catching Up with the Joneses," American Economic Review, American Economic Association, vol. 80(2), pages 38-42, May.
    3. Guanhao Feng & Stefano Giglio & Dacheng Xiu, 2020. "Taming the Factor Zoo: A Test of New Factors," Journal of Finance, American Finance Association, vol. 75(3), pages 1327-1370, June.
    4. Granziera, Eleonora & Sekhposyan, Tatevik, 2019. "Predicting relative forecasting performance: An empirical investigation," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1636-1657.
    5. Martin, Ian W.R. & Nagel, Stefan, 2022. "Market efficiency in the age of big data," Journal of Financial Economics, Elsevier, vol. 145(1), pages 154-177.
    6. Ferreira, Miguel A. & Santa-Clara, Pedro, 2011. "Forecasting stock market returns: The sum of the parts is more than the whole," Journal of Financial Economics, Elsevier, vol. 100(3), pages 514-537, June.
    7. Kou, Gang & Yüksel, Serhat & Dinçer, Hasan, 2022. "Inventive problem-solving map of innovative carbon emission strategies for solar energy-based transportation investment projects," Applied Energy, Elsevier, vol. 311(C).
    8. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
    9. Julien Cujean & Michael Hasler, 2017. "Why Does Return Predictability Concentrate in Bad Times?," Journal of Finance, American Finance Association, vol. 72(6), pages 2717-2758, December.
    10. Inoue, Atsushi & Kilian, Lutz, 2006. "On the selection of forecasting models," Journal of Econometrics, Elsevier, vol. 130(2), pages 273-306, February.
    11. R. David Mclean & Jeffrey Pontiff, 2016. "Does Academic Research Destroy Stock Return Predictability?," Journal of Finance, American Finance Association, vol. 71(1), pages 5-32, February.
    12. Emilio Abad-Segura & Mariana-Daniela González-Zamar, 2020. "Global Research Trends in Financial Transactions," Mathematics, MDPI, vol. 8(4), pages 1-32, April.
    13. Kyle Jurado & Sydney C. Ludvigson & Serena Ng, 2015. "Measuring Uncertainty," American Economic Review, American Economic Association, vol. 105(3), pages 1177-1216, March.
    14. Peter Klibanoff & Massimo Marinacci & Sujoy Mukerji, 2005. "A Smooth Model of Decision Making under Ambiguity," Econometrica, Econometric Society, vol. 73(6), pages 1849-1892, November.
    15. David E. Rapach & Jack K. Strauss & Guofu Zhou, 2013. "International Stock Return Predictability: What Is the Role of the United States?," Journal of Finance, American Finance Association, vol. 68(4), pages 1633-1662, August.
    16. John Y. Campbell & Samuel B. Thompson, 2008. "Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average?," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1509-1531, July.
    17. Henkel, Sam James & Martin, J. Spencer & Nardari, Federico, 2011. "Time-varying short-horizon predictability," Journal of Financial Economics, Elsevier, vol. 99(3), pages 560-580, March.
    18. James B. Heaton & Nicholas Polson & Jan H. Witte, 2017. "Rejoinder to ‘Deep learning for finance: deep portfolios’," Applied Stochastic Models in Business and Industry, John Wiley & Sons, vol. 33(1), pages 19-21, January.
    19. David E. Rapach & Jack K. Strauss & Guofu Zhou, 2010. "Out-of-Sample Equity Premium Prediction: Combination Forecasts and Links to the Real Economy," The Review of Financial Studies, Society for Financial Studies, vol. 23(2), pages 821-862, February.
    20. Zhu, Xiaoneng & Zhu, Jie, 2013. "Predicting stock returns: A regime-switching combination approach and economic links," Journal of Banking & Finance, Elsevier, vol. 37(11), pages 4120-4133.
    21. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    22. Zhi Da & Umit G. Gurun & Mitch Warachka, 2014. "Frog in the Pan: Continuous Information and Momentum," The Review of Financial Studies, Society for Financial Studies, vol. 27(7), pages 2171-2218.
    23. Dangl, Thomas & Halling, Michael, 2012. "Predictive regressions with time-varying coefficients," Journal of Financial Economics, Elsevier, vol. 106(1), pages 157-181.
    24. Novy-Marx, Robert & Velikov, Mihail, 2022. "Betting against betting against beta," Journal of Financial Economics, Elsevier, vol. 143(1), pages 80-106.
    25. Pesaran, M Hashem & Timmermann, Allan, 1995. "Predictability of Stock Returns: Robustness and Economic Significance," Journal of Finance, American Finance Association, vol. 50(4), pages 1201-1228, September.
    26. David F. Hendry & Michael P. Clements, 2004. "Pooling of forecasts," Econometrics Journal, Royal Economic Society, vol. 7(1), pages 1-31, June.
    27. Mark W. Watson & James H. Stock, 2004. "Combination forecasts of output growth in a seven-country data set," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(6), pages 405-430.
    28. J. B. Heaton & N. G. Polson & J. H. Witte, 2017. "Deep learning for finance: deep portfolios," Applied Stochastic Models in Business and Industry, John Wiley & Sons, vol. 33(1), pages 3-12, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dichtl, Hubert & Drobetz, Wolfgang & Neuhierl, Andreas & Wendt, Viktoria-Sophie, 2021. "Data snooping in equity premium prediction," International Journal of Forecasting, Elsevier, vol. 37(1), pages 72-94.
    2. Daniel Borup & Jonas N. Eriksen & Mads M. Kjær & Martin Thyrsgaard, 2024. "Predicting Bond Return Predictability," Management Science, INFORMS, vol. 70(2), pages 931-951, February.
    3. Wang, Yudong & Pan, Zhiyuan & Liu, Li & Wu, Chongfeng, 2019. "Oil price increases and the predictability of equity premium," Journal of Banking & Finance, Elsevier, vol. 102(C), pages 43-58.
    4. Rapach, David & Zhou, Guofu, 2013. "Forecasting Stock Returns," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 328-383, Elsevier.
    5. Yi, Yongsheng & Ma, Feng & Zhang, Yaojie & Huang, Dengshi, 2019. "Forecasting stock returns with cycle-decomposed predictors," International Review of Financial Analysis, Elsevier, vol. 64(C), pages 250-261.
    6. Stein, Tobias, 2024. "Forecasting the equity premium with frequency-decomposed technical indicators," International Journal of Forecasting, Elsevier, vol. 40(1), pages 6-28.
    7. Gupta, Rangan & Hammoudeh, Shawkat & Modise, Mampho P. & Nguyen, Duc Khuong, 2014. "Can economic uncertainty, financial stress and consumer sentiments predict U.S. equity premium?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 33(C), pages 367-378.
    8. Xi Dong & Yan Li & David E. Rapach & Guofu Zhou, 2022. "Anomalies and the Expected Market Return," Journal of Finance, American Finance Association, vol. 77(1), pages 639-681, February.
    9. Davide Pettenuzzo & Francesco Ravazzolo, 2016. "Optimal Portfolio Choice Under Decision‐Based Model Combinations," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 31(7), pages 1312-1332, November.
    10. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    11. Timmermann, Allan, 2018. "Forecasting Methods in Finance," CEPR Discussion Papers 12692, C.E.P.R. Discussion Papers.
    12. Lawrenz, Jochen & Zorn, Josef, 2017. "Predicting international stock returns with conditional price-to-fundamental ratios," Journal of Empirical Finance, Elsevier, vol. 43(C), pages 159-184.
    13. Allan Timmermann, 2018. "Forecasting Methods in Finance," Annual Review of Financial Economics, Annual Reviews, vol. 10(1), pages 449-479, November.
    14. Nonejad, Nima, 2022. "Predicting equity premium out-of-sample by conditioning on newspaper-based uncertainty measures: A comparative study," International Review of Financial Analysis, Elsevier, vol. 83(C).
    15. Yu, Deshui & Huang, Difang, 2023. "Cross-sectional uncertainty and expected stock returns," Journal of Empirical Finance, Elsevier, vol. 72(C), pages 321-340.
    16. Dai, Zhifeng & Chang, Xiaoming, 2021. "Forecasting stock market volatility: Can the risk aversion measure exert an important role?," The North American Journal of Economics and Finance, Elsevier, vol. 58(C).
    17. Mykola Babiak & Jozef Barunik, 2020. "Deep Learning, Predictability, and Optimal Portfolio Returns," CERGE-EI Working Papers wp677, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    18. Cotter, John & Eyiah-Donkor, Emmanuel & Potì, Valerio, 2023. "Commodity futures return predictability and intertemporal asset pricing," Journal of Commodity Markets, Elsevier, vol. 31(C).
    19. Thomadakis, Apostolos, 2016. "Do Combination Forecasts Outperform the Historical Average? Economic and Statistical Evidence," MPRA Paper 71589, University Library of Munich, Germany.
    20. He, Zhongzhi (Lawrence) & Zhu, Jie & Zhu, Xiaoneng, 2015. "Multi-factor volatility and stock returns," Journal of Banking & Finance, Elsevier, vol. 61(S2), pages 132-149.

    More about this item

    Keywords

    Machine learning; Out-of-sample R $$^2$$ 2 -hacking; Return predictability; Monitoring;
    All these keywords.

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis
    • C58 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Financial Econometrics
    • G17 - Financial Economics - - General Financial Markets - - - Financial Forecasting and Simulation

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:fininn:v:9:y:2023:i:1:d:10.1186_s40854-023-00497-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.