IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v33y2017i1p48-60.html
   My bibliography  Save this article

Model Confidence Sets and forecast combination

Author

Listed:
  • Samuels, Jon D.
  • Sekkel, Rodrigo M.

Abstract

A longstanding finding in the forecasting literature is that averaging the forecasts from a range of models often improves upon forecasts based on a single model, with equal weight averaging working particularly well. This paper analyzes the effects of trimming the set of models prior to averaging. We compare different trimming schemes and propose a new approach based on Model Confidence Sets that takes into account the statistical significance of the out-of-sample forecasting performance. In an empirical application to the forecasting of U.S. macroeconomic indicators, we find significant gains in out-of-sample forecast accuracy from using the proposed trimming method.

Suggested Citation

  • Samuels, Jon D. & Sekkel, Rodrigo M., 2017. "Model Confidence Sets and forecast combination," International Journal of Forecasting, Elsevier, vol. 33(1), pages 48-60.
  • Handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:48-60
    DOI: 10.1016/j.ijforecast.2016.07.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207016300747
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ijforecast.2016.07.004?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marco Aiolfi & Carlo Ambrogio Favero, "undated". "Model Uncertainty, Thick Modelling and the predictability of Stock Returns," Working Papers 221, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    2. Inoue, Atsushi & Kilian, Lutz, 2008. "How Useful Is Bagging in Forecasting Economic Time Series? A Case Study of U.S. Consumer Price Inflation," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 511-522, June.
    3. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    4. Bernanke, Ben S. & Boivin, Jean, 2003. "Monetary policy in a data-rich environment," Journal of Monetary Economics, Elsevier, vol. 50(3), pages 525-546, April.
    5. Spyros Makridakis & Robert L. Winkler, 1983. "Averages of Forecasts: Some Empirical Results," Management Science, INFORMS, vol. 29(9), pages 987-996, September.
    6. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    7. Wright, Jonathan H., 2008. "Bayesian Model Averaging and exchange rate forecasts," Journal of Econometrics, Elsevier, vol. 146(2), pages 329-341, October.
    8. Capistrán, Carlos & Timmermann, Allan, 2009. "Forecast Combination With Entry and Exit of Experts," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 428-440.
    9. Stock, James H & Watson, Mark W, 2002. "Macroeconomic Forecasting Using Diffusion Indexes," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(2), pages 147-162, April.
    10. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    11. David F. Hendry & Michael P. Clements, 2004. "Pooling of forecasts," Econometrics Journal, Royal Economic Society, vol. 7(1), pages 1-31, June.
    12. Jonathan H. Wright, 2009. "Forecasting US inflation by Bayesian model averaging," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(2), pages 131-144.
    13. Jon Faust & Simon Gilchrist & Jonathan H. Wright & Egon Zakrajšsek, 2013. "Credit Spreads as Predictors of Real-Time Economic Activity: A Bayesian Model-Averaging Approach," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1501-1519, December.
    14. Faust, Jon & Wright, Jonathan H., 2009. "Comparing Greenbook and Reduced Form Forecasts Using a Large Realtime Dataset," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 468-479.
    15. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 4, pages 135-196, Elsevier.
    16. Peter Tulip, 2009. "Has the Economy Become More Predictable? Changes in Greenbook Forecast Accuracy," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 41(6), pages 1217-1231, September.
    17. Aiolfi, Marco & Timmermann, Allan, 2006. "Persistence in forecasting performance and conditional combination strategies," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 31-53.
    18. Hansen, Bruce E., 2008. "Least-squares forecast averaging," Journal of Econometrics, Elsevier, vol. 146(2), pages 342-350, October.
    19. Pérez-Quirós, Gabriel & Camacho, Máximo & Alvarez, Rocio, 2012. "Finite sample performance of small versus large scale dynamic factor models," CEPR Discussion Papers 8867, C.E.P.R. Discussion Papers.
    20. G. Elliott & C. Granger & A. Timmermann (ed.), 2006. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 1, number 1.
    21. Mark W. Watson & James H. Stock, 2004. "Combination forecasts of output growth in a seven-country data set," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(6), pages 405-430.
    22. Bruce E. Hansen, 2007. "Least Squares Model Averaging," Econometrica, Econometric Society, vol. 75(4), pages 1175-1189, July.
    23. Clemon, Robert T & Winkler, Robert L, 1986. "Combining Economic Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 39-46, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Luca De Mori & Pietro Millossovich & Rui Zhu & Steven Haberman, 2024. "Two-Population Mortality Forecasting: An Approach Based on Model Averaging," Risks, MDPI, vol. 12(4), pages 1-17, March.
    2. Bravo, Jorge M. & Ayuso, Mercedes & Holzmann, Robert & Palmer, Edward, 2021. "Addressing the life expectancy gap in pension policy," Insurance: Mathematics and Economics, Elsevier, vol. 99(C), pages 200-221.
    3. Roccazzella, Francesco & Gambetti, Paolo & Vrins, Frédéric, 2022. "Optimal and robust combination of forecasts via constrained optimization and shrinkage," International Journal of Forecasting, Elsevier, vol. 38(1), pages 97-116.
    4. Afanasyev, Dmitriy O. & Fedorova, Elena A., 2019. "On the impact of outlier filtering on the electricity price forecasting accuracy," Applied Energy, Elsevier, vol. 236(C), pages 196-210.
    5. WenJie Wang & Qi Xu & Dandan Fan, 2018. "Stein-Rule Combination Forecasting on RFID Based Supply Chain," Asia-Pacific Journal of Operational Research (APJOR), World Scientific Publishing Co. Pte. Ltd., vol. 35(02), pages 1-13, April.
    6. Amendola, Alessandra & Braione, Manuela & Candila, Vincenzo & Storti, Giuseppe, 2020. "A Model Confidence Set approach to the combination of multivariate volatility forecasts," International Journal of Forecasting, Elsevier, vol. 36(3), pages 873-891.
    7. James Younker, 2022. "Calculating Effective Degrees of Freedom for Forecast Combinations and Ensemble Models," Discussion Papers 2022-19, Bank of Canada.
    8. Bravo, Jorge M. & Ayuso, Mercedes & Holzmann, Robert & Palmer, Edward, 2023. "Intergenerational actuarial fairness when longevity increases: Amending the retirement age," Insurance: Mathematics and Economics, Elsevier, vol. 113(C), pages 161-184.
    9. Chiang, I-Hsuan Ethan & Liao, Yin & Zhou, Qing, 2021. "Modeling the cross-section of stock returns using sensible models in a model pool," Journal of Empirical Finance, Elsevier, vol. 60(C), pages 56-73.
    10. Wei, Yu & Liang, Chao & Li, Yan & Zhang, Xunhui & Wei, Guiwu, 2020. "Can CBOE gold and silver implied volatility help to forecast gold futures volatility in China? Evidence based on HAR and Ridge regression models," Finance Research Letters, Elsevier, vol. 35(C).
    11. Chao Liang & Yongan Xu & Zhonglu Chen & Xiafei Li, 2023. "Forecasting China's stock market volatility with shrinkage method: Can Adaptive Lasso select stronger predictors from numerous predictors?," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 28(4), pages 3689-3699, October.
    12. Wenting Zhao & Juanjuan Zhao & Xilong Yao & Zhixin Jin & Pan Wang, 2019. "A Novel Adaptive Intelligent Ensemble Model for Forecasting Primary Energy Demand," Energies, MDPI, vol. 12(7), pages 1-28, April.
    13. Daniel Borup & Jonas N. Eriksen & Mads M. Kjær & Martin Thyrsgaard, 2024. "Predicting Bond Return Predictability," Management Science, INFORMS, vol. 70(2), pages 931-951, February.
    14. Carlos Henrique Dias Cordeiro de Castro & Fernando Antonio Lucena Aiube, 2023. "Forecasting inflation time series using score‐driven dynamic models and combination methods: The case of Brazil," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 369-401, March.
    15. Mercedes Ayuso & Jorge M. Bravo & Robert Holzmann & Edward Palmer, 2021. "Automatic Indexation of the Pension Age to Life Expectancy: When Policy Design Matters," Risks, MDPI, vol. 9(5), pages 1-28, May.
    16. Jun Hao & Xiaolei Sun & Qianqian Feng, 2020. "A Novel Ensemble Approach for the Forecasting of Energy Demand Based on the Artificial Bee Colony Algorithm," Energies, MDPI, vol. 13(3), pages 1-25, January.
    17. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    18. Garcia, Márcio G.P. & Medeiros, Marcelo C. & Vasconcelos, Gabriel F.R., 2017. "Real-time inflation forecasting with high-dimensional models: The case of Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 679-693.
    19. Hengzhen Lu & Qiujin Gao & Ling Xiao & Gurjeet Dhesi, 2024. "Forecasting EUA futures volatility with geopolitical risk: evidence from GARCH-MIDAS models," Review of Managerial Science, Springer, vol. 18(7), pages 1917-1943, July.
    20. Ding, Lili & Zhao, Zhongchao & Wang, Lei, 2022. "Probability density forecasts for natural gas demand in China: Do mixed-frequency dynamic factors matter?," Applied Energy, Elsevier, vol. 312(C).
    21. Ioannis D. Vrontos & John Galakis & Ekaterini Panopoulou & Spyridon D. Vrontos, 2024. "Forecasting GDP growth: The economic impact of COVID‐19 pandemic," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(4), pages 1042-1086, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jon D. Samuels & Rodrigo Sekkel, 2013. "Forecasting with Many Models: Model Confidence Sets and Forecast Combination," Staff Working Papers 13-11, Bank of Canada.
    2. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 4, pages 135-196, Elsevier.
    3. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    4. Cheng, Xu & Hansen, Bruce E., 2015. "Forecasting with factor-augmented regression: A frequentist model averaging approach," Journal of Econometrics, Elsevier, vol. 186(2), pages 280-293.
    5. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    6. Kim, Hyun Hak & Swanson, Norman R., 2018. "Mining big data using parsimonious factor, machine learning, variable selection and shrinkage methods," International Journal of Forecasting, Elsevier, vol. 34(2), pages 339-354.
    7. Knut Are Aastveit & Karsten R. Gerdrup & Anne Sofie Jore & Leif Anders Thorsrud, 2014. "Nowcasting GDP in Real Time: A Density Combination Approach," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 32(1), pages 48-68, January.
    8. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    9. Xiaojie Xu, 2020. "Corn Cash Price Forecasting," American Journal of Agricultural Economics, John Wiley & Sons, vol. 102(4), pages 1297-1320, August.
    10. Laura Carabotta & Peter Claeys, 2024. "Combine to compete: Improving fiscal forecast accuracy over time," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(4), pages 948-982, July.
    11. Genre, Véronique & Kenny, Geoff & Meyler, Aidan & Timmermann, Allan, 2013. "Combining expert forecasts: Can anything beat the simple average?," International Journal of Forecasting, Elsevier, vol. 29(1), pages 108-121.
    12. Soybilgen, Barış & Yazgan, Ege, 2018. "Evaluating nowcasts of bridge equations with advanced combination schemes for the Turkish unemployment rate," Economic Modelling, Elsevier, vol. 72(C), pages 99-108.
    13. Chan, Felix & Pauwels, Laurent L., 2018. "Some theoretical results on forecast combinations," International Journal of Forecasting, Elsevier, vol. 34(1), pages 64-74.
    14. Chen, Qitong & Hong, Yongmiao & Li, Haiqi, 2024. "Time-varying forecast combination for factor-augmented regressions with smooth structural changes," Journal of Econometrics, Elsevier, vol. 240(1).
    15. Conflitti, Cristina & De Mol, Christine & Giannone, Domenico, 2015. "Optimal combination of survey forecasts," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1096-1103.
    16. Faust, Jon & Wright, Jonathan H., 2013. "Forecasting Inflation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 2-56, Elsevier.
    17. Kim, Hyun Hak & Swanson, Norman R., 2014. "Forecasting financial and macroeconomic variables using data reduction methods: New empirical evidence," Journal of Econometrics, Elsevier, vol. 178(P2), pages 352-367.
    18. Bec, Frédérique & Mogliani, Matteo, 2015. "Nowcasting French GDP in real-time with surveys and “blocked” regressions: Combining forecasts or pooling information?," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1021-1042.
    19. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    20. Bayer, Sebastian, 2018. "Combining Value-at-Risk forecasts using penalized quantile regressions," Econometrics and Statistics, Elsevier, vol. 8(C), pages 56-77.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:48-60. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.