IDEAS home Printed from https://ideas.repec.org/p/bge/wpaper/688.html
   My bibliography  Save this paper

Conditional Predictive Density Evaluation in the Presence of Instabilities

Author

Listed:
  • Barbara Rossi
  • Tatevik Sehkposyan

Abstract

We propose new methods for evaluating predictive densities. The methods include Kolmogorov-Smirnov and Cramer-von Mises-type tests for the correct specification of predictive densities robust to dynamic mis-specification. The novelty is that the tests can detect mis-specification in the predictive densities even if it appears only over a fraction of the sample, due to the presence of instabilities. Our results indicate that our tests are well sized and have good power in detecting mis-specification in predictive densities, even when it is time-varying. An application to density forecasts of the Survey of Professional Forecasters demonstrates the usefulness of the proposed methodologies.

Suggested Citation

  • Barbara Rossi & Tatevik Sehkposyan, 2013. "Conditional Predictive Density Evaluation in the Presence of Instabilities," Working Papers 688, Barcelona School of Economics.
    • Handle: RePEc:bge:wpaper:688
    as

    Download full text from publisher

    File URL: https://www.barcelonagse.eu/sites/default/files/working_paper_pdfs/688.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Corradi, Valentina & Swanson, Norman R., 2007. "Evaluation of dynamic stochastic general equilibrium models based on distributional comparison of simulated and historical data," Journal of Econometrics, Elsevier, vol. 136(2), pages 699-723, February.
    2. Rossi, Barbara, 2005. "Optimal Tests For Nested Model Selection With Underlying Parameter Instability," Econometric Theory, Cambridge University Press, vol. 21(5), pages 962-990, October.
    3. Dick van Dijk & Philip Hans Franses, 2003. "Selecting a Nonlinear Time Series Model using Weighted Tests of Equal Forecast Accuracy," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 65(s1), pages 727-744, December.
    4. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    5. Corradi, Valentina & Swanson, Norman R., 2006. "Bootstrap conditional distribution tests in the presence of dynamic misspecification," Journal of Econometrics, Elsevier, vol. 133(2), pages 779-806, August.
    6. Ang, Andrew & Bekaert, Geert & Wei, Min, 2007. "Do macro variables, asset markets, or surveys forecast inflation better?," Journal of Monetary Economics, Elsevier, vol. 54(4), pages 1163-1212, May.
    7. Yongmiao Hong, 2005. "Nonparametric Specification Testing for Continuous-Time Models with Applications to Term Structure of Interest Rates," The Review of Financial Studies, Society for Financial Studies, vol. 18(1), pages 37-84.
    8. Andrade, P. & Ghysels, E. & Idier, J., 2012. "Tails of Inflation Forecasts and Tales of Monetary Policy," Working papers 407, Banque de France.
    9. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    10. Inoue, Atsushi, 2001. "Testing For Distributional Change In Time Series," Econometric Theory, Cambridge University Press, vol. 17(1), pages 156-187, February.
    11. Atsushi Inoue, "undated". "Testing Change in Time Series," Computing in Economics and Finance 1997 7, Society for Computational Economics.
    12. Andrews, Donald W K, 1993. "Tests for Parameter Instability and Structural Change with Unknown Change Point," Econometrica, Econometric Society, vol. 61(4), pages 821-856, July.
    13. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    14. Amisano, Gianni & Giacomini, Raffaella, 2007. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 177-190, April.
    15. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    16. Valentina Corradi & Norman Swanson, 2006. "Predictive Density Evaluation. Revised," Departmental Working Papers 200621, Rutgers University, Department of Economics.
    17. Jushan Bai, 2003. "Testing Parametric Conditional Distributions of Dynamic Models," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 531-549, August.
    18. Corradi, Valentina & Swanson, Norman R., 2006. "Predictive density and conditional confidence interval accuracy tests," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 187-228.
    19. Corradi, Valentina & Swanson, Norman R., 2006. "Predictive Density Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 5, pages 197-284, Elsevier.
    20. repec:hal:journl:peer-00834423 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gergely Ganics & Barbara Rossi & Tatevik Sekhposyan, 2019. "From fixed-event to fixed-horizon density forecasts: obtaining measures of multi-horizon uncertainty from survey density forecasts," Working Papers 1947, Banco de España.
    2. Emilio Zanetti Chini, 2018. "Forecasters’ utility and forecast coherence," CREATES Research Papers 2018-23, Department of Economics and Business Economics, Aarhus University.
    3. Davide Delle Monache & Ivan Petrella, 2014. "Adaptive Models and Heavy Tails," Working Papers 720, Queen Mary University of London, School of Economics and Finance.
    4. Zdeněk Zmeškal & Dana Dluhošová & Karolina Lisztwanová & Antonín Pončík & Iveta Ratmanová, 2023. "Distribution Prediction of Decomposed Relative EVA Measure with Levy-Driven Mean-Reversion Processes: The Case of an Automotive Sector of a Small Open Economy," Forecasting, MDPI, vol. 5(2), pages 1-19, May.
    5. Jonas Dovern & Hans Manner, 2020. "Order‐invariant tests for proper calibration of multivariate density forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 440-456, June.
    6. Barbara Rossi, 2013. "Exchange Rate Predictability," Journal of Economic Literature, American Economic Association, vol. 51(4), pages 1063-1119, December.
    7. Barbara Rossi, 2014. "Comment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 32(4), pages 510-514, October.
    8. Clements, Michael P., 2018. "Are macroeconomic density forecasts informative?," International Journal of Forecasting, Elsevier, vol. 34(2), pages 181-198.
    9. Rossi, Barbara & Sekhposyan, Tatevik, 2014. "Evaluating predictive densities of US output growth and inflation in a large macroeconomic data set," International Journal of Forecasting, Elsevier, vol. 30(3), pages 662-682.
    10. González-Rivera, Gloria & Sun, Yingying, 2017. "Density forecast evaluation in unstable environments," International Journal of Forecasting, Elsevier, vol. 33(2), pages 416-432.
    11. João Henrique G. Mazzeu & Gloria González-Rivera & Esther Ruiz & Helena Veiga, 2020. "A bootstrap approach for generalized Autocontour testing Implications for VIX forecast densities," Econometric Reviews, Taylor & Francis Journals, vol. 39(10), pages 971-990, November.
    12. Tommaso Proietti & Martyna Marczak & Gianluigi Mazzi, 2017. "Euromind‐ D : A Density Estimate of Monthly Gross Domestic Product for the Euro Area," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(3), pages 683-703, April.
    13. Gergely Akos Ganics, 2017. "Optimal density forecast combinations," Working Papers 1751, Banco de España.
    14. Gregor Bäurle & Elizabeth Steiner & Gabriel Züllig, 2021. "Forecasting the production side of GDP," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(3), pages 458-480, April.
    15. repec:bny:wpaper:0089 is not listed on IDEAS
    16. Raffaella Giacomini & Barbara Rossi, 2015. "Forecasting in Nonstationary Environments: What Works and What Doesn't in Reduced-Form and Structural Models," Annual Review of Economics, Annual Reviews, vol. 7(1), pages 207-229, August.
    17. Clements, Michael P. & Galvao, Ana Beatriz, 2020. "Density Forecasting with BVAR Models under Macroeconomic Data Uncertainty," EMF Research Papers 36, Economic Modelling and Forecasting Group.
    18. Davide Delle Monache & Ivan Petrella, 2014. "Adaptive Models and Heavy Tails," Working Papers 720, Queen Mary University of London, School of Economics and Finance.
    19. Michael P. Clements & Ana Beatriz Galvão, 2023. "Density forecasting with Bayesian Vector Autoregressive models under macroeconomic data uncertainty," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(2), pages 164-185, March.
    20. Barbara Rossi, 2019. "Forecasting in the Presence of Instabilities: How Do We Know Whether Models Predict Well and How to Improve Them," Working Papers 1162, Barcelona School of Economics.
    21. Rossi, Barbara & Ganics, Gergely & Sekhposyan, Tatevik, 2020. "From Fixed-event to Fixed-horizon Density Forecasts: Obtaining Measures of Multi-horizon Uncertainty from Survey Density Foreca," CEPR Discussion Papers 14267, C.E.P.R. Discussion Papers.
    22. Barbara Rossi, 2018. "Identifying and estimating the effects of unconventional monetary policy in the data: How to do It and what have we learned?," Economics Working Papers 1641, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2020.
    23. Federico Bassetti & Roberto Casarin & Francesco Ravazzolo, 2019. "Density Forecasting," BEMPS - Bozen Economics & Management Paper Series BEMPS59, Faculty of Economics and Management at the Free University of Bozen.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rossi, Barbara & Sekhposyan, Tatevik, 2019. "Alternative tests for correct specification of conditional predictive densities," Journal of Econometrics, Elsevier, vol. 208(2), pages 638-657.
    2. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    3. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    4. Raffaella Giacomini & Barbara Rossi, 2015. "Forecasting in Nonstationary Environments: What Works and What Doesn't in Reduced-Form and Structural Models," Annual Review of Economics, Annual Reviews, vol. 7(1), pages 207-229, August.
    5. Corradi, Valentina & Swanson, Norman R., 2004. "Some recent developments in predictive accuracy testing with nested models and (generic) nonlinear alternatives," International Journal of Forecasting, Elsevier, vol. 20(2), pages 185-199.
    6. González-Rivera, Gloria & Yoldas, Emre, 2012. "Autocontour-based evaluation of multivariate predictive densities," International Journal of Forecasting, Elsevier, vol. 28(2), pages 328-342.
    7. Cees Diks & Valentyn Panchenko & Dick van Dijk, 2008. "Partial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails," Tinbergen Institute Discussion Papers 08-050/4, Tinbergen Institute.
    8. Rossi, Barbara & Sekhposyan, Tatevik, 2014. "Evaluating predictive densities of US output growth and inflation in a large macroeconomic data set," International Journal of Forecasting, Elsevier, vol. 30(3), pages 662-682.
    9. Corradi, Valentina & Swanson, Norman R., 2011. "Predictive density construction and accuracy testing with multiple possibly misspecified diffusion models," Journal of Econometrics, Elsevier, vol. 161(2), pages 304-324, April.
    10. Gordy, Michael B. & McNeil, Alexander J., 2020. "Spectral backtests of forecast distributions with application to risk management," Journal of Banking & Finance, Elsevier, vol. 116(C).
    11. Cai, Lili & Swanson, Norman R., 2011. "In- and out-of-sample specification analysis of spot rate models: Further evidence for the period 1982-2008," Journal of Empirical Finance, Elsevier, vol. 18(4), pages 743-764, September.
    12. Gaglianone, Wagner Piazza & Marins, Jaqueline Terra Moura, 2017. "Evaluation of exchange rate point and density forecasts: An application to Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 707-728.
    13. Corradi, Valentina & Swanson, Norman R., 2004. "A test for the distributional comparison of simulated and historical data," Economics Letters, Elsevier, vol. 85(2), pages 185-193, November.
    14. Hall, Stephen G. & Mitchell, James, 2007. "Combining density forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 1-13.
    15. Norman Swanson & Nii Ayi Armah, 2006. "Predictive Inference Under Model Misspecification with an Application to Assessing the Marginal Predictive Content of Money for Output," Departmental Working Papers 200619, Rutgers University, Department of Economics.
    16. Jonas Dovern & Hans Manner, 2020. "Order‐invariant tests for proper calibration of multivariate density forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 440-456, June.
    17. Valentina Corradi & Norman R. Swanson, 2007. "Nonparametric Bootstrap Procedures For Predictive Inference Based On Recursive Estimation Schemes," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 48(1), pages 67-109, February.
    18. Patton, Andrew, 2013. "Copula Methods for Forecasting Multivariate Time Series," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 899-960, Elsevier.
    19. repec:hal:journl:peer-00834423 is not listed on IDEAS
    20. González-Rivera, Gloria & Sun, Yingying, 2017. "Density forecast evaluation in unstable environments," International Journal of Forecasting, Elsevier, vol. 33(2), pages 416-432.
    21. Jian Wang & Jason J. Wu, 2012. "The Taylor Rule and Forecast Intervals for Exchange Rates," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 44(1), pages 103-144, February.

    More about this item

    Keywords

    predictive density; dynamic mis-specification; instability; structural change; forecast evaluation;
    All these keywords.

    JEL classification:

    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bge:wpaper:688. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Bruno Guallar (email available below). General contact details of provider: https://edirc.repec.org/data/bargses.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.