IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/32269.html
   My bibliography  Save this paper

Comprehensive OOS Evaluation of Predictive Algorithms with Statistical Decision Theory

Author

Listed:
  • Jeff Dominitz
  • Charles F. Manski

Abstract

We argue that comprehensive out-of-sample (OOS) evaluation using statistical decision theory (SDT) should replace the current practice of K-fold and Common Task Framework validation in machine learning (ML) research. SDT provides a formal framework for performing comprehensive OOS evaluation across all possible (1) training samples, (2) populations that may generate training data, and (3) populations of prediction interest. Regarding feature (3), we emphasize that SDT requires the practitioner to directly confront the possibility that the future may not look like the past and to account for a possible need to extrapolate from one population to another when building a predictive algorithm. SDT is simple in abstraction, but it is often computationally demanding to implement. We discuss progress in tractable implementation of SDT when prediction accuracy is measured by mean square error or by misclassification rate. We summarize research studying settings in which the training data will be generated from a subpopulation of the population of prediction interest. We consider conditional prediction with alternative restrictions on the state space of possible populations that may generate training data. We present an illustrative application of the methodology to the problem of predicting patient illness to inform clinical decision making. We conclude by calling on ML researchers to join with econometricians and statisticians in expanding the domain within which implementation of SDT is tractable.

Suggested Citation

  • Jeff Dominitz & Charles F. Manski, 2024. "Comprehensive OOS Evaluation of Predictive Algorithms with Statistical Decision Theory," NBER Working Papers 32269, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:32269
    Note: TWP
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w32269.pdf
    Download Restriction: Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bradley Efron, 2020. "Prediction, Estimation, and Attribution," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(530), pages 636-655, April.
    2. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    3. Charles F. Manski, 2019. "Treatment Choice With Trial Data: Statistical Decision Theory Should Supplant Hypothesis Testing," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 296-304, March.
    4. Manski, Charles F., 2023. "Probabilistic prediction for binary treatment choice: With focus on personalized medicine," Journal of Econometrics, Elsevier, vol. 234(2), pages 647-663.
    5. Charles F. Manski, 1989. "Anatomy of the Selection Problem," Journal of Human Resources, University of Wisconsin Press, vol. 24(3), pages 343-360.
    6. Charles F Manski, 2007. "Adaptive Minimax-Regret Treatment Choice, with Application to Drug Approval," Levine's Working Paper Archive 122247000000001404, David K. Levine.
    7. Jeff Dominitz & Charles F. Manski, 2017. "More Data or Better Data? A Statistical Decision Problem," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 84(4), pages 1583-1605.
    8. Charles F. Manski & Aleksey Tetenov, 2023. "Statistical decision theory respecting stochastic dominance," The Japanese Economic Review, Springer, vol. 74(4), pages 447-469, October.
    9. Charles F. Manski & John V. Pepper, 2018. "How Do Right-to-Carry Laws Affect Crime Rates? Coping with Ambiguity Using Bounded-Variation Assumptions," The Review of Economics and Statistics, MIT Press, vol. 100(2), pages 232-244, May.
    10. Stoye, Jörg, 2012. "Minimax regret treatment choice with covariates or with limited validity of experiments," Journal of Econometrics, Elsevier, vol. 166(1), pages 138-156.
    11. Bradley Efron, 2020. "Prediction, Estimation, and Attribution," International Statistical Review, International Statistical Institute, vol. 88(S1), pages 28-59, December.
    12. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    13. Charles F. Manski & John V. Pepper, 2000. "Monotone Instrumental Variables, with an Application to the Returns to Schooling," Econometrica, Econometric Society, vol. 68(4), pages 997-1012, July.
    14. Sawa, Takamitsu & Hiromatsu, Takeshi, 1973. "Minimax Regret Significance Points for a Preliminary Test in Regression Analysis," Econometrica, Econometric Society, vol. 41(6), pages 1093-1101, November.
    15. Charles F. Manski, 2021. "Econometrics for Decision Making: Building Foundations Sketched by Haavelmo and Wald," Econometrica, Econometric Society, vol. 89(6), pages 2827-2853, November.
    16. Charles F. Manski, 2018. "Credible ecological inference for medical decisions with personalized risk assessment," Quantitative Economics, Econometric Society, vol. 9(2), pages 541-569, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Manski, Charles F., 2023. "Probabilistic prediction for binary treatment choice: With focus on personalized medicine," Journal of Econometrics, Elsevier, vol. 234(2), pages 647-663.
    2. Charles F. Manski, 2021. "Econometrics for Decision Making: Building Foundations Sketched by Haavelmo and Wald," Econometrica, Econometric Society, vol. 89(6), pages 2827-2853, November.
    3. Charles F. Manski, 2019. "Statistical inference for statistical decisions," Papers 1909.06853, arXiv.org.
    4. Federico Crippa, 2024. "Regret Analysis in Threshold Policy Design," Papers 2404.11767, arXiv.org.
    5. Timothy Christensen & Hyungsik Roger Moon & Frank Schorfheide, 2022. "Optimal Decision Rules when Payoffs are Partially Identified," Papers 2204.11748, arXiv.org, revised May 2023.
    6. Toru Kitagawa & Sokbae Lee & Chen Qiu, 2022. "Treatment Choice with Nonlinear Regret," Papers 2205.08586, arXiv.org, revised Oct 2024.
    7. Daido Kido, 2023. "Locally Asymptotically Minimax Statistical Treatment Rules Under Partial Identification," Papers 2311.08958, arXiv.org.
    8. Charles F. Manski, 2019. "Meta-Analysis for Medical Decisions," NBER Working Papers 25504, National Bureau of Economic Research, Inc.
    9. Kitagawa, Toru & Wang, Guanyi, 2023. "Who should get vaccinated? Individualized allocation of vaccines over SIR network," Journal of Econometrics, Elsevier, vol. 232(1), pages 109-131.
    10. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    11. Charles F. Manski, 2018. "Reasonable patient care under uncertainty," Health Economics, John Wiley & Sons, Ltd., vol. 27(10), pages 1397-1421, October.
    12. Achim Ahrens & Alessandra Stampi‐Bombelli & Selina Kurer & Dominik Hangartner, 2024. "Optimal multi‐action treatment allocation: A two‐phase field experiment to boost immigrant naturalization," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(7), pages 1379-1395, November.
    13. Charles F. Manski & Aleksey Tetenov, 2023. "Statistical decision theory respecting stochastic dominance," The Japanese Economic Review, Springer, vol. 74(4), pages 447-469, October.
    14. Dimitris Christelis & Dimitris Georgarakos & Tullio Jappelli & Geoff Kenny, 2020. "The Covid-19 Crisis and Consumption: Survey Evidence from Six EU Countries," Working Papers 2020_31, Business School - Economics, University of Glasgow.
    15. Giorgio Brunello & Dimitris Christelis & Anna Sanz‐de‐Galdeano & Anastasia Terskaya, 2024. "Does college selectivity reduce obesity? A partial identification approach," Health Economics, John Wiley & Sons, Ltd., vol. 33(10), pages 2306-2320, October.
    16. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    17. Kock, Anders Bredahl & Preinerstorfer, David & Veliyev, Bezirgen, 2023. "Treatment recommendation with distributional targets," Journal of Econometrics, Elsevier, vol. 234(2), pages 624-646.
    18. Manski, Charles F. & Molinari, Francesca, 2021. "Estimating the COVID-19 infection rate: Anatomy of an inference problem," Journal of Econometrics, Elsevier, vol. 220(1), pages 181-192.
    19. Azevedo, Eduardo M. & Mao, David & Montiel Olea, José Luis & Velez, Amilcar, 2023. "The A/B testing problem with Gaussian priors," Journal of Economic Theory, Elsevier, vol. 210(C).
    20. Neil Christy & Amanda Ellen Kowalski, 2024. "Counting Defiers in Health Care with a Design-Based Likelihood for the Joint Distribution of Potential Outcomes," Papers 2412.16352, arXiv.org.

    More about this item

    JEL classification:

    • C44 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Operations Research; Statistical Decision Theory
    • C45 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Neural Networks and Related Topics
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:32269. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.