IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v31y2015i3p943-951.html
   My bibliography  Save this article

Limitations of Ensemble Bayesian Model Averaging for forecasting social science problems

Author

Listed:
  • Graefe, Andreas
  • Küchenhoff, Helmut
  • Stierle, Veronika
  • Riedl, Bernhard

Abstract

We compare the accuracies of simple unweighted averages and Ensemble Bayesian Model Averaging (EBMA) for combining forecasts in the social sciences. A review of prior studies from the domain of economic forecasting finds that the simple average was more accurate than EBMA in four studies out of five. On average, the error of EBMA was 5% higher than that of the simple average. A reanalysis and extension of a published study provides further evidence for US presidential election forecasting. The error of EBMA was 33% higher than the corresponding error of the simple average. Simple averages are easy both to describe and to understand, and thus are easy to use. In addition, simple averages provide accurate forecasts in many settings. Researchers who are developing new approaches to combining forecasts need to compare the accuracy of their method to this widely established benchmark. Forecasting practitioners should favor simple averages over more complex methods unless there is strong evidence in support of differential weights.

Suggested Citation

  • Graefe, Andreas & Küchenhoff, Helmut & Stierle, Veronika & Riedl, Bernhard, 2015. "Limitations of Ensemble Bayesian Model Averaging for forecasting social science problems," International Journal of Forecasting, Elsevier, vol. 31(3), pages 943-951.
  • Handle: RePEc:eee:intfor:v:31:y:2015:i:3:p:943-951
    DOI: 10.1016/j.ijforecast.2014.12.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207014001769
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ijforecast.2014.12.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bork, Lasse & Møller, Stig V., 2015. "Forecasting house prices in the 50 states using Dynamic Model Averaging and Dynamic Model Selection," International Journal of Forecasting, Elsevier, vol. 31(1), pages 63-78.
    2. Montgomery, Jacob M. & Hollenbach, Florian M. & Ward, Michael D., 2012. "Improving Predictions using Ensemble Bayesian Model Averaging," Political Analysis, Cambridge University Press, vol. 20(3), pages 271-291, July.
    3. Wright, Jonathan H., 2008. "Bayesian Model Averaging and exchange rate forecasts," Journal of Econometrics, Elsevier, vol. 146(2), pages 329-341, October.
    4. Jonathan H. Wright, 2009. "Forecasting US inflation by Bayesian model averaging," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(2), pages 131-144.
    5. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    6. Genre, Véronique & Kenny, Geoff & Meyler, Aidan & Timmermann, Allan, 2013. "Combining expert forecasts: Can anything beat the simple average?," International Journal of Forecasting, Elsevier, vol. 29(1), pages 108-121.
    7. Todd E. Clark & Michael W. McCracken, 2010. "Averaging forecasts from VARs with uncertain instabilities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(1), pages 5-29, January.
    8. repec:cup:judgdm:v:7:y:2012:i:6:p:746-749 is not listed on IDEAS
    9. Douglas Hibbs, 2000. "Bread and Peace Voting in U.S. Presidential Elections," Public Choice, Springer, vol. 104(1), pages 149-180, July.
    10. Graefe, Andreas, 2015. "Improving forecasts using equally weighted predictors," Journal of Business Research, Elsevier, vol. 68(8), pages 1792-1799.
    11. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    12. Abramowitz, Alan I., 2008. "It's about time: Forecasting the 2008 presidential election with the time-for-change model," International Journal of Forecasting, Elsevier, vol. 24(2), pages 209-217.
    13. Graefe, Andreas & Armstrong, J. Scott & Jones, Randall J. & Cuzán, Alfred G., 2014. "Combining forecasts: An application to elections," International Journal of Forecasting, Elsevier, vol. 30(1), pages 43-54.
    14. Jeremy Smith & Kenneth F. Wallis, 2009. "A Simple Explanation of the Forecast Combination Puzzle," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 71(3), pages 331-355, June.
    15. Mark W. Watson & James H. Stock, 2004. "Combination forecasts of output growth in a seven-country data set," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(6), pages 405-430.
    16. Philip A Schrodt, 2014. "Seven deadly sins of contemporary quantitative political analysis," Journal of Peace Research, Peace Research Institute Oslo, vol. 51(2), pages 287-300, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Easaw, Joshy & Fang, Yongmei & Heravi, Saeed, 2021. "Using Polls to Forecast Popular Vote Share for US Presidential Elections 2016 and 2020: An Optimal Forecast Combination Based on Ensemble Empirical Model," Cardiff Economics Working Papers E2021/34, Cardiff University, Cardiff Business School, Economics Section.
    2. He, Kaijian & Tso, Geoffrey K.F. & Zou, Yingchao & Liu, Jia, 2018. "Crude oil risk forecasting: New evidence from multiscale analysis approach," Energy Economics, Elsevier, vol. 76(C), pages 574-583.
    3. Bunker, Kenneth, 2020. "A two-stage model to forecast elections in new democracies," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1407-1419.
    4. Wang, Jue & Wang, Zhen & Li, Xiang & Zhou, Hao, 2022. "Artificial bee colony-based combination approach to forecasting agricultural commodity prices," International Journal of Forecasting, Elsevier, vol. 38(1), pages 21-34.
    5. Andreas Graefe, 2018. "Predicting elections: Experts, polls, and fundamentals," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(4), pages 334-344, July.
    6. Chen, Guojin & Liu, Yanzhen & Zhang, Yu, 2021. "Systemic risk measures and distribution forecasting of macroeconomic shocks," International Review of Economics & Finance, Elsevier, vol. 75(C), pages 178-196.
    7. Graefe, Andreas, 2023. "Embrace the differences: Revisiting the PollyVote method of combining forecasts for U.S. presidential elections (2004 to 2020)," International Journal of Forecasting, Elsevier, vol. 39(1), pages 170-177.
    8. Deepa Mishra & Angappa Gunasekaran & Thanos Papadopoulos & Stephen J. Childe, 2018. "Big Data and supply chain management: a review and bibliometric analysis," Annals of Operations Research, Springer, vol. 270(1), pages 313-336, November.
    9. repec:cup:judgdm:v:13:y:2018:i:4:p:334-344 is not listed on IDEAS
    10. Graefe, Andreas & Armstrong, J. Scott & Jones, Randall J. & Cuzan, Alfred G., 2017. "Assessing the 2016 U.S. Presidential Election Popular Vote Forecasts," MPRA Paper 83282, University Library of Munich, Germany.
    11. Graefe, Andreas, 2019. "Accuracy of German federal election forecasts, 2013 & 2017," International Journal of Forecasting, Elsevier, vol. 35(3), pages 868-877.
    12. Andreas Graefe & Kesten C Green & J Scott Armstrong, 2019. "Accuracy gains from conservative forecasting: Tests using variations of 19 econometric models to predict 154 elections in 10 countries," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-14, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    2. Graefe, Andreas, 2023. "Embrace the differences: Revisiting the PollyVote method of combining forecasts for U.S. presidential elections (2004 to 2020)," International Journal of Forecasting, Elsevier, vol. 39(1), pages 170-177.
    3. von der Gracht, Heiko A. & Hommel, Ulrich & Prokesch, Tobias & Wohlenberg, Holger, 2016. "Testing weighting approaches for forecasting in a Group Wisdom Support System environment," Journal of Business Research, Elsevier, vol. 69(10), pages 4081-4094.
    4. Blanc, Sebastian M. & Setzer, Thomas, 2016. "When to choose the simple average in forecast combination," Journal of Business Research, Elsevier, vol. 69(10), pages 3951-3962.
    5. Jing Zeng, 2015. "Combining Country-Specific Forecasts when Forecasting Euro Area Macroeconomic Aggregates," Working Paper Series of the Department of Economics, University of Konstanz 2015-11, Department of Economics, University of Konstanz.
    6. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    7. Conflitti, Cristina & De Mol, Christine & Giannone, Domenico, 2015. "Optimal combination of survey forecasts," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1096-1103.
    8. Graefe, Andreas & Armstrong, J. Scott & Jones, Randall J. & Cuzán, Alfred G., 2014. "Combining forecasts: An application to elections," International Journal of Forecasting, Elsevier, vol. 30(1), pages 43-54.
    9. Jing Zeng, 2016. "Combining country-specific forecasts when forecasting Euro area macroeconomic aggregates," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 43(2), pages 415-444, May.
    10. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    11. Montgomery, Jacob M. & Hollenbach, Florian M. & Ward, Michael D., 2015. "Calibrating ensemble forecasting models with sparse data in the social sciences," International Journal of Forecasting, Elsevier, vol. 31(3), pages 930-942.
    12. Samuels, Jon D. & Sekkel, Rodrigo M., 2017. "Model Confidence Sets and forecast combination," International Journal of Forecasting, Elsevier, vol. 33(1), pages 48-60.
    13. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    14. Sebastian M. Blanc & Thomas Setzer, 2020. "Bias–Variance Trade-Off and Shrinkage of Weights in Forecast Combination," Management Science, INFORMS, vol. 66(12), pages 5720-5737, December.
    15. Kim, Hyun Hak & Swanson, Norman R., 2014. "Forecasting financial and macroeconomic variables using data reduction methods: New empirical evidence," Journal of Econometrics, Elsevier, vol. 178(P2), pages 352-367.
    16. Yongchen Zhao, 2021. "The robustness of forecast combination in unstable environments: a Monte Carlo study of advanced algorithms," Empirical Economics, Springer, vol. 61(1), pages 173-199, July.
    17. Martin, Gael M. & Loaiza-Maya, Rubén & Maneesoonthorn, Worapree & Frazier, David T. & Ramírez-Hassan, Andrés, 2022. "Optimal probabilistic forecasts: When do they work?," International Journal of Forecasting, Elsevier, vol. 38(1), pages 384-406.
    18. Kang, Yanfei & Cao, Wei & Petropoulos, Fotios & Li, Feng, 2022. "Forecast with forecasts: Diversity matters," European Journal of Operational Research, Elsevier, vol. 301(1), pages 180-190.
    19. Tao Lin & Yiling Chen, 2022. "Sample Complexity of Forecast Aggregation," Papers 2207.13126, arXiv.org, revised Oct 2023.
    20. Esteban Fernández-Vázquez & Blanca Moreno, 2017. "Entropy Econometrics for combining regional economic forecasts: A Data-Weighted Prior Estimator," Journal of Geographical Systems, Springer, vol. 19(4), pages 349-370, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:31:y:2015:i:3:p:943-951. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.