IDEAS home Printed from https://ideas.repec.org/a/eee/empfin/v73y2023icp251-271.html
   My bibliography  Save this article

Forecasting realized volatility with machine learning: Panel data perspective

Author

Listed:
  • Zhu, Haibin
  • Bai, Lu
  • He, Lidan
  • Liu, Zhi

Abstract

Machine learning approaches have become very popular in many fields in this big data age. This paper considers the problem of forecasting realized volatility with machine learning using high-frequency data. Instead of treating the realized volatility as a univariate time series studied by many existing works in literature, we employ panel data analysis to improve forecasting accuracy in the short term. We use six effective machine-learning methods for the realized volatility panel data. We compare our results with the traditional linear-type models under the same panel data framework and with the single time series forecasting via the same machine learning methods. The results show that the panel-data-based machine learning method (PDML) outperforms the other methods.

Suggested Citation

  • Zhu, Haibin & Bai, Lu & He, Lidan & Liu, Zhi, 2023. "Forecasting realized volatility with machine learning: Panel data perspective," Journal of Empirical Finance, Elsevier, vol. 73(C), pages 251-271.
    • Handle: RePEc:eee:empfin:v:73:y:2023:i:c:p:251-271
    DOI: 10.1016/j.jempfin.2023.07.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0927539823000683
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jempfin.2023.07.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    2. Inoue, Atsushi & Jin, Lu & Rossi, Barbara, 2017. "Rolling window selection for out-of-sample forecasting with time-varying parameters," Journal of Econometrics, Elsevier, vol. 196(1), pages 55-67.
    3. James Ming Chen, 2021. "An Introduction to Machine Learning for Panel Data," International Advances in Economic Research, Springer;International Atlantic Economic Society, vol. 27(1), pages 1-16, February.
    4. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," Review of Finance, European Finance Association, vol. 33(5), pages 2223-2273.
    5. Donaldson, R. Glen & Kamstra, Mark, 1997. "An artificial neural network-GARCH model for international stock return volatility," Journal of Empirical Finance, Elsevier, vol. 4(1), pages 17-46, January.
    6. Andrea Bucci, 2020. "Realized Volatility Forecasting with Neural Networks," Journal of Financial Econometrics, Oxford University Press, vol. 18(3), pages 502-531.
    7. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    8. Andrew J. Patton & Kevin Sheppard, 2015. "Good Volatility, Bad Volatility: Signed Jumps and The Persistence of Volatility," The Review of Economics and Statistics, MIT Press, vol. 97(3), pages 683-697, July.
    9. Ole E. Barndorff-Nielsen & Neil Shephard, 2004. "Econometric Analysis of Realized Covariation: High Frequency Based Covariance, Regression, and Correlation in Financial Economics," Econometrica, Econometric Society, vol. 72(3), pages 885-925, May.
    10. Pettenuzzo, Davide & Timmermann, Allan & Valkanov, Rossen, 2014. "Forecasting stock returns under economic constraints," Journal of Financial Economics, Elsevier, vol. 114(3), pages 517-553.
    11. Chang, Chia-Lin & McAleer, Michael & Tansuchat, Roengchai, 2013. "Conditional correlations and volatility spillovers between crude oil and stock index returns," The North American Journal of Economics and Finance, Elsevier, vol. 25(C), pages 116-138.
    12. Audrino, Francesco & Sigrist, Fabio & Ballinari, Daniele, 2020. "The impact of sentiment and attention measures on stock market volatility," International Journal of Forecasting, Elsevier, vol. 36(2), pages 334-357.
    13. John H. Cochrane, 2011. "Presidential Address: Discount Rates," Journal of Finance, American Finance Association, vol. 66(4), pages 1047-1108, August.
    14. Francesco Audrino & Simon D. Knaus, 2016. "Lassoing the HAR Model: A Model Selection Perspective on Realized Volatility Dynamics," Econometric Reviews, Taylor & Francis Journals, vol. 35(8-10), pages 1485-1521, December.
    15. Charlotte Christiansen & Maik Schmeling & Andreas Schrimpf, 2012. "A comprehensive look at financial volatility prediction by economic variables," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(6), pages 956-977, September.
    16. Badi H. Baltagi, 2008. "Forecasting with panel data," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 27(2), pages 153-173.
    17. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    18. Michael P. Clements & Hans-Martin Krolzig, 1998. "A comparison of the forecast performance of Markov-switching and threshold autoregressive models of US GNP," Econometrics Journal, Royal Economic Society, vol. 1(Conferenc), pages 47-75.
    19. Campbell, John Y. & Yogo, Motohiro, 2006. "Efficient tests of stock return predictability," Journal of Financial Economics, Elsevier, vol. 81(1), pages 27-60, July.
    20. Fulvio Corsi, 2009. "A Simple Approximate Long-Memory Model of Realized Volatility," Journal of Financial Econometrics, Oxford University Press, vol. 7(2), pages 174-196, Spring.
    21. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2223-2273.
    22. Eugene F Fama & Kenneth R French, 2020. "Comparing Cross-Section and Time-Series Factor Models," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 1891-1926.
    23. Eugene F Fama & Kenneth R French & Andrew KarolyiEditor, 2020. "Comparing Cross-Section and Time-Series Factor Models," Review of Finance, European Finance Association, vol. 33(5), pages 1891-1926.
    24. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    25. Neil Shephard & Silja Kinnebrock & Ole E. Barndorff-Neilsen, 2008. "Measuring downside risk - realised semivariance," Economics Series Working Papers 382, University of Oxford, Department of Economics.
    26. Jeremiah Green & John R. M. Hand & X. Frank Zhang, 2017. "The Characteristics that Provide Independent Information about Average U.S. Monthly Stock Returns," The Review of Financial Studies, Society for Financial Studies, vol. 30(12), pages 4389-4436.
    27. Kolokolov, Aleksey & Livieri, Giulia & Pirino, Davide, 2020. "Statistical inferences for price staleness," Journal of Econometrics, Elsevier, vol. 218(1), pages 32-81.
    28. Lesmond, David A & Ogden, Joseph P & Trzcinka, Charles A, 1999. "A New Estimate of Transaction Costs," The Review of Financial Studies, Society for Financial Studies, vol. 12(5), pages 1113-1141.
    29. Kim Christensen & Mathias Siggaard & Bezirgen Veliyev, 2023. "A Machine Learning Approach to Volatility Forecasting," Journal of Financial Econometrics, Oxford University Press, vol. 21(5), pages 1680-1727.
    30. McAleer, Michael & Medeiros, Marcelo C., 2008. "A multiple regime smooth transition Heterogeneous Autoregressive model for long memory and asymmetries," Journal of Econometrics, Elsevier, vol. 147(1), pages 104-119, November.
    31. Bu, Ruijun & Hizmeri, Rodrigo & Izzeldin, Marwan & Murphy, Anthony & Tsionas, Mike, 2023. "The contribution of jump signs and activity to forecasting stock price volatility," Journal of Empirical Finance, Elsevier, vol. 70(C), pages 144-164.
    32. Wang, Hui, 2019. "VIX and volatility forecasting: A new insight," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 533(C).
    33. Augustine Addo & Fidelis Sunzuoye, 2013. "The Impact of Treasury Bill Rate and Interest Rate On The Stock Market Returns: Case Of Ghana Stock Exchange," European Journal of Business and Economics, Central Bohemia University, vol. 8(2), pages 3781:8-3781, July.
    34. Torben G. Andersen & Tim Bollerslev & Francis X. Diebold & Paul Labys, 2003. "Modeling and Forecasting Realized Volatility," Econometrica, Econometric Society, vol. 71(2), pages 579-625, March.
    35. John Y. Campbell & Samuel B. Thompson, 2008. "Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average?," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1509-1531, July.
    36. Michiel de Pooter & Martin Martens & Dick van Dijk, 2008. "Predicting the Daily Covariance Matrix for S&P 100 Stocks Using Intraday Data—But Which Frequency to Use?," Econometric Reviews, Taylor & Francis Journals, vol. 27(1-3), pages 199-229.
    37. Yudong Wang & Zhiyuan Pan & Chongfeng Wu, 2017. "Time‐Varying Parameter Realized Volatility Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 36(5), pages 566-580, August.
    38. Chuong Luong & Nikolai Dokuchaev, 2018. "Forecasting of Realised Volatility with the Random Forests Algorithm," JRFM, MDPI, vol. 11(4), pages 1-15, October.
    39. Mittnik, Stefan & Robinzonov, Nikolay & Spindler, Martin, 2015. "Stock market volatility: Identifying major drivers and the nature of their impact," Journal of Banking & Finance, Elsevier, vol. 58(C), pages 1-14.
    40. Amihud, Yakov, 2002. "Illiquidity and stock returns: cross-section and time-series effects," Journal of Financial Markets, Elsevier, vol. 5(1), pages 31-56, January.
    41. Hu, Michael Y. & Tsoukalas, Christos, 1999. "Combining conditional volatility forecasts using neural networks: an application to the EMS exchange rates," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 9(4), pages 407-422, November.
    42. Mauro Bernardi & Leopoldo Catania, 2018. "The model confidence set package for R," International Journal of Computational Economics and Econometrics, Inderscience Enterprises Ltd, vol. 8(2), pages 144-158.
    43. Cederburg, Scott & O’Doherty, Michael S. & Wang, Feifei & Yan, Xuemin (Sterling), 2020. "On the performance of volatility-managed portfolios," Journal of Financial Economics, Elsevier, vol. 138(1), pages 95-117.
    44. Michael W. Brandt & Alon Brav & John R. Graham & Alok Kumar, 2010. "The Idiosyncratic Volatility Puzzle: Time Trend or Speculative Episodes?," The Review of Financial Studies, Society for Financial Studies, vol. 23(2), pages 863-899, February.
    45. Friedman, Jerome H., 2002. "Stochastic gradient boosting," Computational Statistics & Data Analysis, Elsevier, vol. 38(4), pages 367-378, February.
    46. Patton, Andrew J., 2011. "Volatility forecast comparison using imperfect volatility proxies," Journal of Econometrics, Elsevier, vol. 160(1), pages 246-256, January.
    47. Ole E. Barndorff-Nielsen & Silja Kinnebrock & Neil Shephard, 2008. "Measuring downside risk-realised semivariance," Economics Papers 2008-W02, Economics Group, Nuffield College, University of Oxford.
    48. Hardouvelis, Gikas A., 1987. "Macroeconomic information and stock prices," Journal of Economics and Business, Elsevier, vol. 39(2), pages 131-140, May.
    49. Wang, Yudong & Wei, Yu & Wu, Chongfeng & Yin, Libo, 2018. "Oil and the short-term predictability of stock return volatility," Journal of Empirical Finance, Elsevier, vol. 47(C), pages 90-104.
    50. Fama, Eugene F & MacBeth, James D, 1973. "Risk, Return, and Equilibrium: Empirical Tests," Journal of Political Economy, University of Chicago Press, vol. 81(3), pages 607-636, May-June.
    51. Hui Zou & Trevor Hastie, 2005. "Addendum: Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(5), pages 768-768, November.
    52. Wenbo Wu & Jiaqi Chen & Zhibin (Ben) Yang & Michael L. Tindall, 2021. "A Cross-Sectional Machine Learning Approach for Hedge Fund Return Prediction and Selection," Management Science, INFORMS, vol. 67(7), pages 4577-4601, July.
    53. Yufeng Han & David Lesmond, 2011. "Liquidity Biases and the Pricing of Cross-sectional Idiosyncratic Volatility," The Review of Financial Studies, Society for Financial Studies, vol. 24(5), pages 1590-1629.
    54. Hui Zou & Trevor Hastie, 2005. "Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(2), pages 301-320, April.
    55. Farshid Abdi & Angelo Ranaldo, 2017. "A Simple Estimation of Bid-Ask Spreads from Daily Close, High, and Low Prices," The Review of Financial Studies, Society for Financial Studies, vol. 30(12), pages 4437-4480.
    56. Barroso, Pedro & Boons, Martijn & Karehnke, Paul, 2021. "Time-varying state variable risk premia in the ICAPM," Journal of Financial Economics, Elsevier, vol. 139(2), pages 428-451.
    57. Lyócsa, Štefan & Molnár, Peter & Výrost, Tomáš, 2021. "Stock market volatility forecasting: Do we need high-frequency data?," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1092-1110.
    58. Giuseppe Buccheri & Davide Pirino & Luca Trapin, 2021. "Managing liquidity with portfolio staleness," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 44(1), pages 215-239, June.
    59. Donaldson, R Glen & Kamstra, Mark, 1996. "A New Dividend Forecasting Procedure That Rejects Bubbles in Asset Prices: The Case of 1929's Stock Crash," The Review of Financial Studies, Society for Financial Studies, vol. 9(2), pages 333-383.
    60. Engle, Robert F, 1982. "Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation," Econometrica, Econometric Society, vol. 50(4), pages 987-1007, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alessio Brini & Giacomo Toscano, 2024. "SpotV2Net: Multivariate Intraday Spot Volatility Forecasting via Vol-of-Vol-Informed Graph Attention Networks," Papers 2401.06249, arXiv.org, revised Jan 2025.
    2. Francesco Audrino & Jonathan Chassot, 2024. "HARd to Beat: The Overlooked Impact of Rolling Windows in the Era of Machine Learning," Papers 2406.08041, arXiv.org.
    3. Díaz, Juan D. & Hansen, Erwin & Cabrera, Gabriel, 2024. "Machine-learning stock market volatility: Predictability, drivers, and economic value," International Review of Financial Analysis, Elsevier, vol. 94(C).
    4. Niu, Zibo & Demirer, Riza & Suleman, Muhammad Tahir & Zhang, Hongwei & Zhu, Xuehong, 2024. "Do industries predict stock market volatility? Evidence from machine learning models," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 90(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Díaz, Juan D. & Hansen, Erwin & Cabrera, Gabriel, 2024. "Machine-learning stock market volatility: Predictability, drivers, and economic value," International Review of Financial Analysis, Elsevier, vol. 94(C).
    2. Kim Christensen & Mathias Siggaard & Bezirgen Veliyev, 2023. "A Machine Learning Approach to Volatility Forecasting," Journal of Financial Econometrics, Oxford University Press, vol. 21(5), pages 1680-1727.
    3. Niu, Zibo & Demirer, Riza & Suleman, Muhammad Tahir & Zhang, Hongwei & Zhu, Xuehong, 2024. "Do industries predict stock market volatility? Evidence from machine learning models," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 90(C).
    4. Zhang, Hongwei & Zhao, Xinyi & Gao, Wang & Niu, Zibo, 2023. "The role of higher moments in predicting China's oil futures volatility: Evidence from machine learning models," Journal of Commodity Markets, Elsevier, vol. 32(C).
    5. Andrea Bucci, 2020. "Realized Volatility Forecasting with Neural Networks," Journal of Financial Econometrics, Oxford University Press, vol. 18(3), pages 502-531.
    6. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    7. Zhang, Yaojie & Wei, Yu & Zhang, Yi & Jin, Daxiang, 2019. "Forecasting oil price volatility: Forecast combination versus shrinkage method," Energy Economics, Elsevier, vol. 80(C), pages 423-433.
    8. Chao Zhang & Yihuang Zhang & Mihai Cucuringu & Zhongmin Qian, 2022. "Volatility forecasting with machine learning and intraday commonality," Papers 2202.08962, arXiv.org, revised Feb 2023.
    9. Liu, Guangqiang & Guo, Xiaozhu, 2022. "Forecasting stock market volatility using commodity futures volatility information," Resources Policy, Elsevier, vol. 75(C).
    10. Zhang, Yaojie & Lei, Likun & Wei, Yu, 2020. "Forecasting the Chinese stock market volatility with international market volatilities: The role of regime switching," The North American Journal of Economics and Finance, Elsevier, vol. 52(C).
    11. Chun, Dohyun & Cho, Hoon & Ryu, Doojin, 2023. "Discovering the drivers of stock market volatility in a data-rich world," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 82(C).
    12. Li, Zhao-Chen & Xie, Chi & Wang, Gang-Jin & Zhu, You & Zeng, Zhi-Jian & Gong, Jue, 2024. "Forecasting global stock market volatilities: A shrinkage heterogeneous autoregressive (HAR) model with a large cross-market predictor set," International Review of Economics & Finance, Elsevier, vol. 93(PB), pages 673-711.
    13. Francesco Audrino & Jonathan Chassot, 2024. "HARd to Beat: The Overlooked Impact of Rolling Windows in the Era of Machine Learning," Papers 2406.08041, arXiv.org.
    14. Xue Gong & Weiguo Zhang & Yuan Zhao & Xin Ye, 2023. "Forecasting stock volatility with a large set of predictors: A new forecast combination method," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1622-1647, November.
    15. Ricardo P. Masini & Marcelo C. Medeiros & Eduardo F. Mendes, 2023. "Machine learning advances for time series forecasting," Journal of Economic Surveys, Wiley Blackwell, vol. 37(1), pages 76-111, February.
    16. Niu, Zibo & Wang, Chenlu & Zhang, Hongwei, 2023. "Forecasting stock market volatility with various geopolitical risks categories: New evidence from machine learning models," International Review of Financial Analysis, Elsevier, vol. 89(C).
    17. Gong, Xue & Ye, Xin & Zhang, Weiguo & Zhang, Yue, 2023. "Predicting energy futures high-frequency volatility using technical indicators: The role of interaction," Energy Economics, Elsevier, vol. 119(C).
    18. Huei-Wen Teng & Yu-Hsien Li, 2023. "Can deep neural networks outperform Fama-MacBeth regression and other supervised learning approaches in stock returns prediction with asset-pricing factors?," Digital Finance, Springer, vol. 5(1), pages 149-182, March.
    19. He, Mengxi & Wang, Yudong & Zeng, Qing & Zhang, Yaojie, 2023. "Forecasting aggregate stock market volatility with industry volatilities: The role of spillover index," Research in International Business and Finance, Elsevier, vol. 65(C).
    20. Lyócsa, Štefan & Todorova, Neda, 2024. "Forecasting of clean energy market volatility: The role of oil and the technology sector," Energy Economics, Elsevier, vol. 132(C).

    More about this item

    Keywords

    Realized volatility; Panel data analysis; Machine learning; Forecasting;
    All these keywords.

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • G10 - Financial Economics - - General Financial Markets - - - General (includes Measurement and Data)
    • G12 - Financial Economics - - General Financial Markets - - - Asset Pricing; Trading Volume; Bond Interest Rates

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:empfin:v:73:y:2023:i:c:p:251-271. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jempfin .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.