IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v48y2014i4p2155-2173.html
   My bibliography  Save this article

An unbiased model comparison test using cross-validation

Author

Listed:
  • Bruce Desmarais
  • Jeffrey Harden

Abstract

Social scientists often consider multiple empirical models of the same process. When these models are parametric and non-nested, the null hypothesis that two models fit the data equally well is commonly tested using methods introduced by Vuong (Econometrica 57(2):307–333, 1989 ) and Clarke (Am J Political Sci 45(3):724–744, 2001 ; J Confl Resolut 47(1):72–93, 2003 ; Political Anal 15(3):347–363, 2007 ). The objective of each is to compare the Kullback–Leibler Divergence (KLD) of the two models from the true model that generated the data. Here we show that both of these tests are based upon a biased estimator of the KLD, the individual log-likelihood contributions, and that the Clarke test is not proven to be consistent for the difference in KLDs. As a solution, we derive a test based upon cross-validated log-likelihood contributions, which represent an unbiased KLD estimate. We demonstrate the CVDM test’s superior performance via simulation, then apply it to two empirical examples from political science. We find that the test’s selection can diverge from those of the Vuong and Clarke tests and that this can ultimately lead to differences in substantive conclusions. Copyright Springer Science+Business Media Dordrecht 2014

Suggested Citation

  • Bruce Desmarais & Jeffrey Harden, 2014. "An unbiased model comparison test using cross-validation," Quality & Quantity: International Journal of Methodology, Springer, vol. 48(4), pages 2155-2173, July.
  • Handle: RePEc:spr:qualqt:v:48:y:2014:i:4:p:2155-2173
    DOI: 10.1007/s11135-013-9884-7
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s11135-013-9884-7
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1007/s11135-013-9884-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    2. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    3. Christopher H. Achen, 2005. "Let's Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong," Conflict Management and Peace Science, Peace Science Society (International), vol. 22(4), pages 327-339, September.
    4. Michael A. Bailey, 2007. "Comparable Preference Estimates across Time and Institutions for the Court, Congress, and Presidency," American Journal of Political Science, John Wiley & Sons, vol. 51(3), pages 433-448, July.
    5. Koenker,Roger, 2005. "Quantile Regression," Cambridge Books, Cambridge University Press, number 9780521845731, September.
    6. Mebane, Walter R. & Sekhon, Jasjeet S., 2002. "Coordination and Policy Moderation at Midterm," American Political Science Review, Cambridge University Press, vol. 96(1), pages 141-157, March.
    7. Madhav Joshi & T. David Mason, 2008. "Between Democracy and Revolution: Peasant Support for Insurgency versus Democracy in Nepal," Journal of Peace Research, Peace Research Institute Oslo, vol. 45(6), pages 765-782, November.
    8. Mark Souva, 2005. "Foreign Policy Determinants: Comparing Realist and Domestic-Political Models of Foreign Policy," Conflict Management and Peace Science, Peace Science Society (International), vol. 22(2), pages 149-163, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chuliá, Helena & Garrón, Ignacio & Uribe, Jorge M., 2024. "Daily growth at risk: Financial or real drivers? The answer is not always the same," International Journal of Forecasting, Elsevier, vol. 40(2), pages 762-776.
    2. Diaa Noureldin & Neil Shephard & Kevin Sheppard, 2012. "Multivariate high‐frequency‐based volatility (HEAVY) models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(6), pages 907-933, September.
    3. Yu‐Sheng Lai, 2022. "High‐frequency data and stock–bond investing," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 41(8), pages 1623-1638, December.
    4. Szabolcs Blazsek & Hector Hernández, 2018. "Analysis of electricity prices for Central American countries using dynamic conditional score models," Empirical Economics, Springer, vol. 55(4), pages 1807-1848, December.
    5. Corradi, Valentina & Fosten, Jack & Gutknecht, Daniel, 2024. "Predictive ability tests with possibly overlapping models," Journal of Econometrics, Elsevier, vol. 241(1).
    6. Aslanidis, Nektarios & Christiansen, Charlotte, 2014. "Quantiles of the realized stock–bond correlation and links to the macroeconomy," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 321-331.
    7. Nowotarski, Jakub & Weron, Rafał, 2018. "Recent advances in electricity price forecasting: A review of probabilistic forecasting," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 1548-1568.
    8. West, Kenneth D., 2001. "Encompassing tests when no model is encompassing," Journal of Econometrics, Elsevier, vol. 105(1), pages 287-308, November.
    9. Mayer, Walter J. & Liu, Feng & Dang, Xin, 2017. "Improving the power of the Diebold–Mariano–West test for least squares predictions," International Journal of Forecasting, Elsevier, vol. 33(3), pages 618-626.
    10. Gonzalo, Jesús & Pitarakis, Jean-Yves, 2024. "Out-of-sample predictability in predictive regressions with many predictor candidates," International Journal of Forecasting, Elsevier, vol. 40(3), pages 1166-1178.
    11. Wagner Piazza Gaglianone & Jaqueline Terra Moura Marins, 2014. "Risk Assessment of the Brazilian FX Rate," Working Papers Series 344, Central Bank of Brazil, Research Department.
    12. Todd E. Clark & Michael W. Mccracken, 2014. "Tests Of Equal Forecast Accuracy For Overlapping Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(3), pages 415-430, April.
    13. Susanne M. Schennach & Daniel Wilhelm, 2017. "A Simple Parametric Model Selection Test," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1663-1674, October.
    14. Martin Magris, 2019. "A Vine-copula extension for the HAR model," Papers 1907.08522, arXiv.org.
    15. Szabolcs Blazsek & Anna Downarowicz, 2013. "Forecasting hedge fund volatility: a Markov regime-switching approach," The European Journal of Finance, Taylor & Francis Journals, vol. 19(4), pages 243-275, April.
    16. Fabian Krüger & Ingmar Nolte, 2011. "Disagreement, Uncertainty and the True Predictive Density," Working Paper Series of the Department of Economics, University of Konstanz 2011-43, Department of Economics, University of Konstanz.
    17. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    18. Tae-Hwy Lee & Yong Bao & Burak Saltoğlu, 2007. "Comparing density forecast models Previous versions of this paper have been circulated with the title, 'A Test for Density Forecast Comparison with Applications to Risk Management' since October 2003;," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(3), pages 203-225.
    19. Kalimipalli, Madhu & Susmel, Raul, 2004. "Regime-switching stochastic volatility and short-term interest rates," Journal of Empirical Finance, Elsevier, vol. 11(3), pages 309-329, June.
    20. Hnatkovska, Viktoria & Marmer, Vadim & Tang, Yao, 2012. "Comparison of misspecified calibrated models: The minimum distance approach," Journal of Econometrics, Elsevier, vol. 169(1), pages 131-138.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:48:y:2014:i:4:p:2155-2173. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.