IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/25456.html
   My bibliography  Save this paper

Inference on Winners

Author

Listed:
  • Isaiah Andrews
  • Toru Kitagawa
  • Adam McCloskey

Abstract

Many empirical questions concern target parameters selected through optimization. For example, researchers may be interested in the effectiveness of the best policy found in a randomized trial, or the best-performing investment strategy based on historical data. Such settings give rise to a winner’s curse, where conventional estimates are biased and conventional confidence intervals are unreliable. This paper develops optimal confidence intervals and median-unbiased estimators that are valid conditional on the target selected and so overcome this winner’s curse. If one requires validity only on average over targets that might have been selected, we develop hybrid procedures that combine conditional and projection confidence intervals to offer further performance gains relative to existing alternatives.

Suggested Citation

  • Isaiah Andrews & Toru Kitagawa & Adam McCloskey, 2019. "Inference on Winners," NBER Working Papers 25456, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:25456
    Note: AP DEV LS PE TWP
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w25456.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Isaiah Andrews & Jonathan Roth & Ariel Pakes, 2023. "Inference for Linear Conditional Moment Inequalities," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(6), pages 2763-2791.
    2. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    3. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility I: Childhood Exposure Effects," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(3), pages 1107-1162.
    4. Elliott, Graham & Muller, Ulrich K., 2007. "Confidence sets for the date of a single break in linear time series regressions," Journal of Econometrics, Elsevier, vol. 141(2), pages 1196-1218, December.
    5. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    6. Dean Karlan & John A. List, 2007. "Does Price Matter in Charitable Giving? Evidence from a Large-Scale Natural Field Experiment," American Economic Review, American Economic Association, vol. 97(5), pages 1774-1793, December.
    7. Bruce E. Hansen, 2000. "Sample Splitting and Threshold Estimation," Econometrica, Econometric Society, vol. 68(3), pages 575-604, May.
    8. Sarah Baird & Joan Hamory Hicks & Michael Kremer & Edward Miguel, 2016. "Worms at Work: Long-run Impacts of a Child Health Investment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(4), pages 1637-1680.
    9. Bachoc, Francois & Leeb, Hannes & Pötscher, Benedikt M., 2014. "Valid confidence intervals for post-model-selection predictors," MPRA Paper 60643, University Library of Munich, Germany.
    10. Esther Duflo & Michael Greenstone & Rohini Pande & Nicholas Ryan, 2018. "The Value of Regulatory Discretion: Estimates From Environmental Inspections in India," Econometrica, Econometric Society, vol. 86(6), pages 2123-2160, November.
    11. Bruce E. Hansen, 2001. "The New Econometrics of Structural Change: Dating Breaks in U.S. Labour Productivity," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 117-128, Fall.
    12. Keisuke Hirano & Jack R. Porter, 2009. "Asymptotics for Statistical Treatment Rules," Econometrica, Econometric Society, vol. 77(5), pages 1683-1701, September.
    13. David Card & Alexandre Mas & Jesse Rothstein, 2008. "Tipping and the Dynamics of Segregation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(1), pages 177-218.
    14. Yoonseok Lee & Yulong Wang, 2020. "Inference in Threshold Models," Center for Policy Research Working Papers 223, Center for Policy Research, Maxwell School, Syracuse University.
    15. Timothy B. Armstrong & Michal Koles'ar & Mikkel Plagborg-M{o}ller, 2020. "Robust Empirical Bayes Confidence Intervals," Papers 2004.03448, arXiv.org, revised May 2022.
    16. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    17. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    18. Raj Chetty & John N. Friedman & Nathaniel Hendren & Maggie R. Jones & Sonya R. Porter, 2018. "The Opportunity Atlas: Mapping the Childhood Roots of Social Mobility," Working Papers 18-42, Center for Economic Studies, U.S. Census Bureau.
    19. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    20. Abhijit Banerjee & Rema Hanna & Jordan Kyle & Benjamin A. Olken & Sudarno Sumarto, 2018. "Tangible Information and Citizen Empowerment: Identification Cards and Food Subsidy Programs in Indonesia," Journal of Political Economy, University of Chicago Press, vol. 126(2), pages 451-491.
    21. Victor Chernozhukov & Sokbae Lee & Adam M. Rosen, 2013. "Intersection Bounds: Estimation and Inference," Econometrica, Econometric Society, vol. 81(2), pages 667-737, March.
    22. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2014. "A Practical Two‐Step Method for Testing Moment Inequalities," Econometrica, Econometric Society, vol. 82(5), pages 1979-2002, September.
    23. Bhattacharya, Debopam, 2009. "Inferring Optimal Peer Assignment From Experimental Data," Journal of the American Statistical Association, American Statistical Association, vol. 104(486), pages 486-500.
    24. Tetenov, Aleksey, 2012. "Statistical treatment choice based on asymmetric minimax regret criteria," Journal of Econometrics, Elsevier, vol. 166(1), pages 157-165.
    25. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    26. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    27. Andrews, Isaiah & Kitagawa, Toru & McCloskey, Adam, 2021. "Inference after estimation of breaks," Journal of Econometrics, Elsevier, vol. 224(1), pages 39-59.
    28. Adnan Q. Khan & Asim I. Khwaja & Benjamin A. Olken, 2016. "Tax Farming Redux: Experimental Evidence on Performance Pay for Tax Collectors," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(1), pages 219-271.
    29. Elliott, Graham & Müller, Ulrich K., 2014. "Pre and post break parameter inference," Journal of Econometrics, Elsevier, vol. 180(2), pages 141-157.
    30. McCloskey, Adam, 2017. "Bonferroni-based size-correction for nonstandard testing problems," Journal of Econometrics, Elsevier, vol. 200(1), pages 17-35.
    31. Bryan S. Graham & Guido W. Imbens & Geert Ridder, 2014. "Complementarity and aggregate implications of assortative matching: A nonparametric analysis," Quantitative Economics, Econometric Society, vol. 5, pages 29-66, March.
    32. Susanne M. Schennach & Daniel Wilhelm, 2017. "A Simple Parametric Model Selection Test," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1663-1674, October.
    33. Andrews, Donald W K, 1993. "Tests for Parameter Instability and Structural Change with Unknown Change Point," Econometrica, Econometric Society, vol. 61(4), pages 821-856, July.
    34. Laber, Eric B. & Murphy, Susan A., 2011. "Adaptive Confidence Intervals for the Test Error in Classification," Journal of the American Statistical Association, American Statistical Association, vol. 106(495), pages 904-913.
    35. Martina Björkman Nyqvist & Seema Jayachandran, 2017. "Mothers Care More, but Fathers Decide: Educating Parents about Child Health in Uganda," American Economic Review, American Economic Association, vol. 107(5), pages 496-500, May.
    36. Andrews, Donald W.K. & Cheng, Xu & Guggenberger, Patrik, 2020. "Generic results for establishing the asymptotic size of confidence sets and tests," Journal of Econometrics, Elsevier, vol. 218(2), pages 496-531.
    37. Xiaoxia Shi, 2015. "A nondegenerate Vuong test," Quantitative Economics, Econometric Society, vol. 6(1), pages 85-121, March.
    38. Howard S. Bloom & Larry L. Orr & Stephen H. Bell & George Cave & Fred Doolittle & Winston Lin & Johannes M. Bos, 1997. "The Benefits and Costs of JTPA Title II-A Programs: Key Findings from the National Job Training Partnership Act Study," Journal of Human Resources, University of Wisconsin Press, vol. 32(3), pages 549-576.
    39. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility II: County-Level Estimates," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(3), pages 1163-1228.
    40. Gregory Cox, 2018. "Almost Sure Uniqueness of a Global Minimum Without Convexity," Papers 1803.02415, arXiv.org, revised Feb 2019.
    41. Hansen Bruce E., 1997. "Inference in TAR Models," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 2(1), pages 1-16, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    2. Will Davis & Alexander Gordan & Rusty Tchernis, 2021. "Measuring the spatial distribution of health rankings in the United States," Health Economics, John Wiley & Sons, Ltd., vol. 30(11), pages 2921-2936, November.
    3. Dominic Coey & Kenneth Hung, 2022. "Empirical Bayes Selection for Value Maximization," Papers 2210.03905, arXiv.org, revised Jan 2023.
    4. Sergei Bazylik & Magne Mogstad & Joseph P. Romano & Azeem Shaikh & Daniel Wilhelm, 2021. "Finite- and Large-Sample Inference for Ranks using Multinomial Data with an Application to Ranking Political Parties," NBER Working Papers 29519, National Bureau of Economic Research, Inc.
    5. Andrews, Isaiah & Kitagawa, Toru & McCloskey, Adam, 2021. "Inference after estimation of breaks," Journal of Econometrics, Elsevier, vol. 224(1), pages 39-59.
    6. Kohei Yata, 2021. "Optimal Decision Rules Under Partial Identification," Papers 2111.04926, arXiv.org, revised Aug 2023.
    7. Aparajithan Venkateswaran & Anirudh Sankar & Arun G. Chandrasekhar & Tyler H. McCormick, 2024. "Robustly estimating heterogeneity in factorial data using Rashomon Partitions," Papers 2404.02141, arXiv.org, revised Aug 2024.
    8. Davide Viviano & Jelena Bradic, 2020. "Fair Policy Targeting," Papers 2005.12395, arXiv.org, revised Jun 2022.
    9. Isaiah Andrews & Jonathan Roth & Ariel Pakes, 2023. "Inference for Linear Conditional Moment Inequalities," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(6), pages 2763-2791.
    10. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised May 2024.
    11. Hsieh, Yu-Wei & Shi, Xiaoxia & Shum, Matthew, 2022. "Inference on estimators defined by mathematical programming," Journal of Econometrics, Elsevier, vol. 226(2), pages 248-268.
    12. David J. Deming, 2021. "The Growing Importance of Decision-Making on the Job," NBER Working Papers 28733, National Bureau of Economic Research, Inc.
    13. Wei, Waverly & Zhou, Yuqing & Zheng, Zeyu & Wang, Jingshen, 2024. "Inference on the best policies with many covariates," Journal of Econometrics, Elsevier, vol. 239(2).
    14. Jiafeng Chen, 2021. "Nonparametric Treatment Effect Identification in School Choice," Papers 2112.03872, arXiv.org, revised Oct 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrews, Isaiah & Kitagawa, Toru & McCloskey, Adam, 2021. "Inference after estimation of breaks," Journal of Econometrics, Elsevier, vol. 224(1), pages 39-59.
    2. McCloskey, Adam, 2017. "Bonferroni-based size-correction for nonstandard testing problems," Journal of Econometrics, Elsevier, vol. 200(1), pages 17-35.
    3. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    4. Toru Kitagawa & Aleksey Tetenov, 2021. "Equality-Minded Treatment Choice," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(2), pages 561-574, March.
    5. Davide Viviano, 2019. "Policy Targeting under Network Interference," Papers 1906.10258, arXiv.org, revised Apr 2024.
    6. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    7. Eric Mbakop & Max Tabord‐Meehan, 2021. "Model Selection for Treatment Choice: Penalized Welfare Maximization," Econometrica, Econometric Society, vol. 89(2), pages 825-848, March.
    8. Toru Kitagawa & Aleksey Tetenov, 2017. "Equality-minded treatment choice," CeMMAP working papers 10/17, Institute for Fiscal Studies.
    9. Kirill Ponomarev & Vira Semenova, 2024. "On the Lower Confidence Band for the Optimal Welfare," Papers 2410.07443, arXiv.org, revised Oct 2024.
    10. Manski, Charles F., 2023. "Probabilistic prediction for binary treatment choice: With focus on personalized medicine," Journal of Econometrics, Elsevier, vol. 234(2), pages 647-663.
    11. Firpo, Sergio & Galvao, Antonio F. & Kobus, Martyna & Parker, Thomas & Rosa-Dias, Pedro, 2020. "Loss Aversion and the Welfare Ranking of Policy Interventions," IZA Discussion Papers 13176, Institute of Labor Economics (IZA).
    12. Bhattacharya, Debopam & Dupas, Pascaline, 2012. "Inferring welfare maximizing treatment assignment under budget constraints," Journal of Econometrics, Elsevier, vol. 167(1), pages 168-196.
    13. Corradi, Valentina & Fosten, Jack & Gutknecht, Daniel, 2024. "Predictive ability tests with possibly overlapping models," Journal of Econometrics, Elsevier, vol. 241(1).
    14. Toru Kitagawa & Hugo Lopez & Jeff Rowley, 2022. "Stochastic Treatment Choice with Empirical Welfare Updating," Papers 2211.01537, arXiv.org, revised Feb 2023.
    15. Kyle Colangelo & Ying-Ying Lee, 2020. "Double Debiased Machine Learning Nonparametric Inference with Continuous Treatments," Papers 2004.03036, arXiv.org, revised Sep 2023.
    16. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    17. Shosei Sakaguchi, 2021. "Estimation of Optimal Dynamic Treatment Assignment Rules under Policy Constraints," Papers 2106.05031, arXiv.org, revised Aug 2024.
    18. Yaein Baek, 2018. "Estimation of a Structural Break Point in Linear Regression Models," Papers 1811.03720, arXiv.org, revised Jun 2020.
    19. Andros Kourtellos & Chih Ming Tan & Steven N. Durlauf, 2022. "The Great Gatsby Curve," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 571-605, August.
    20. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927RR, Cowles Foundation for Research in Economics, Yale University, revised Apr 2015.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:25456. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.