IDEAS home Printed from https://ideas.repec.org/a/eee/econom/v225y2021i2p148-174.html
   My bibliography  Save this article

Permutation test for heterogeneous treatment effects with a nuisance parameter

Author

Listed:
  • Chung, EunYi
  • Olivares, Mauricio

Abstract

This paper proposes an asymptotically valid permutation test for heterogeneous treatment effects in the presence of an estimated nuisance parameter. Not accounting for the estimation error of the nuisance parameter results in statistics that depend on the particulars of the data generating process, and the resulting permutation test fails to control the Type 1 error, even asymptotically.

Suggested Citation

  • Chung, EunYi & Olivares, Mauricio, 2021. "Permutation test for heterogeneous treatment effects with a nuisance parameter," Journal of Econometrics, Elsevier, vol. 225(2), pages 148-174.
    • Handle: RePEc:eee:econom:v:225:y:2021:i:2:p:148-174
    DOI: 10.1016/j.jeconom.2020.09.015
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304407621001561
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jeconom.2020.09.015?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    2. Bo Li, 2009. "Asymptotically Distribution-Free Goodness-of-Fit Testing: A Unifying View," Econometric Reviews, Taylor & Francis Journals, vol. 28(6), pages 632-657.
    3. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    4. Peng Ding & Avi Feller & Luke Miratrix, 2016. "Randomization inference for treatment effect variation," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 655-671, June.
    5. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    6. Stefan Wager & Susan Athey, 2018. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1228-1242, July.
    7. Goldman, Matt & Kaplan, David M., 2018. "Comparing distributions by multiple testing across quantiles or CDF values," Journal of Econometrics, Elsevier, vol. 206(1), pages 143-166.
    8. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    9. Oliver Linton & Esfandiar Maasoumi & Yoon-Jae Whang, 2005. "Consistent Testing for Stochastic Dominance under General Sampling Schemes," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(3), pages 735-765.
    10. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    11. Xiao, Zhijie & Xu, Lan, 2019. "What do mean impacts miss? Distributional effects of corporate diversification," Journal of Econometrics, Elsevier, vol. 213(1), pages 92-120.
    12. Uri Gneezy & John A List, 2006. "Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments," Econometrica, Econometric Society, vol. 74(5), pages 1365-1384, September.
    13. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    14. Chen, Qiang & Zheng, Xu & Pan, Zhiyuan, 2015. "Asymptotically distribution-free tests for the volatility function of a diffusion," Journal of Econometrics, Elsevier, vol. 184(1), pages 124-144.
    15. Delgado, Miguel A. & Hidalgo, Javier & Velasco, Carlos, 2005. "Distribution free goodness-of-fit tests for linear processes," LSE Research Online Documents on Economics 6840, London School of Economics and Political Science, LSE Library.
    16. Bai, Jushan & Ng, Serena, 2001. "A consistent test for conditional symmetry in time series models," Journal of Econometrics, Elsevier, vol. 103(1-2), pages 225-258, July.
    17. Härdle, W. & Marron, S.J., 1990. "Semiparametric comparison of regression curves," LIDAM Reprints CORE 890, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    18. George A. Akerlof, 1982. "Labor Contracts as Partial Gift Exchange," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 97(4), pages 543-569.
    19. Goldman, Matt & Kaplan, David M., 2018. "Comparing distributions by multiple testing across quantiles or CDF values," Journal of Econometrics, Elsevier, vol. 206(1), pages 143-166.
    20. Jushan Bai, 2003. "Testing Parametric Conditional Distributions of Dynamic Models," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 531-549, August.
    21. Song, Kyungchul, 2010. "Testing semiparametric conditional moment restrictions using conditional martingale transforms," Journal of Econometrics, Elsevier, vol. 154(1), pages 74-84, January.
    22. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2017. "Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment," The Review of Economics and Statistics, MIT Press, vol. 99(4), pages 683-697, July.
    23. Roger Koenker & Zhijie Xiao, 2002. "Inference on the Quantile Regression Process," Econometrica, Econometric Society, vol. 70(4), pages 1583-1612, July.
    24. Chung, EunYi & Romano, Joseph P., 2016. "Multivariate and multiple permutation tests," Journal of Econometrics, Elsevier, vol. 193(1), pages 76-91.
    25. Janssen, Arnold, 1997. "Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem," Statistics & Probability Letters, Elsevier, vol. 36(1), pages 9-21, November.
    26. Parker, Thomas, 2013. "A Comparison Of Alternative Approaches To Supremum-Norm Goodness-Of-Fit Tests With Estimated Parameters," Econometric Theory, Cambridge University Press, vol. 29(5), pages 969-1008, October.
    27. Wei Ma & Yichen Qin & Yang Li & Feifang Hu, 2020. "Statistical Inference for Covariate-Adaptive Randomization Procedures," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(531), pages 1488-1497, July.
    28. Delgado, Miguel A. & Stute, Winfried, 2008. "Distribution-free specification tests of conditional models," Journal of Econometrics, Elsevier, vol. 143(1), pages 37-55, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Young, Alwyn, 2024. "Asymptotically robust permutation-based randomization confidence intervals for parametric OLS regression," European Economic Review, Elsevier, vol. 163(C).
    2. Buhl-Wiggers, Julie & Kerwin, Jason T. & Muñoz-Morales, Juan & Smith, Jeffrey & Thornton, Rebecca, 2024. "Some children left behind: Variation in the effects of an educational intervention," Journal of Econometrics, Elsevier, vol. 243(1).
    3. Stefano Bonnini & Getnet Melak Assegie & Kamila Trzcinska, 2024. "Review about the Permutation Approach in Hypothesis Testing," Mathematics, MDPI, vol. 12(17), pages 1-29, August.
    4. Julius Owusu, 2023. "Randomization Inference of Heterogeneous Treatment Effects under Network Interference," Papers 2308.00202, arXiv.org, revised Jan 2025.
    5. Jeffrey Smith, 2022. "Treatment Effect Heterogeneity," Evaluation Review, , vol. 46(5), pages 652-677, October.
    6. Young, Alwyn, 2024. "Asymptotically robust permutation-based randomization confidence intervals for parametric OLS regression," LSE Research Online Documents on Economics 120933, London School of Economics and Political Science, LSE Library.
    7. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org, revised Feb 2025.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chen, Qiang & Zheng, Xu & Pan, Zhiyuan, 2015. "Asymptotically distribution-free tests for the volatility function of a diffusion," Journal of Econometrics, Elsevier, vol. 184(1), pages 124-144.
    2. Song, Kyungchul, 2010. "Testing semiparametric conditional moment restrictions using conditional martingale transforms," Journal of Econometrics, Elsevier, vol. 154(1), pages 74-84, January.
    3. Jaime Ramirez-Cuellar, 2023. "Testing for idiosyncratic Treatment Effect Heterogeneity," Papers 2304.01141, arXiv.org.
    4. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
    5. Matt Goldman & David M. Kaplan, 2018. "Non‐parametric inference on (conditional) quantile differences and interquantile ranges, using L‐statistics," Econometrics Journal, Royal Economic Society, vol. 21(2), pages 136-169, June.
    6. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    7. Buhl-Wiggers, Julie & Kerwin, Jason T. & Muñoz-Morales, Juan & Smith, Jeffrey & Thornton, Rebecca, 2024. "Some children left behind: Variation in the effects of an educational intervention," Journal of Econometrics, Elsevier, vol. 243(1).
    8. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2022. "Multiple Testing and the Distributional Effects of Accountability Incentives in Education," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1552-1568, October.
    9. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    10. David M. Kaplan & Matt Goldman, 2015. "Nonparametric inference on conditional quantile differences and linear combinations, using L-statistics," Working Papers 1503, Department of Economics, University of Missouri.
    11. Ganesh Karapakula, 2023. "Stable Probability Weighting: Large-Sample and Finite-Sample Estimation and Inference Methods for Heterogeneous Causal Effects of Multivalued Treatments Under Limited Overlap," Papers 2301.05703, arXiv.org, revised Jan 2023.
    12. Thomas Parker, 2010. "A comparison of alternative approaches to sup-norm goodness of fit tests with estimated parameters," CeMMAP working papers CWP34/10, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    13. Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl & Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl, 2017. "Distributional impact analysis: toolkit and illustrations of impacts beyond the average treatment effect," Policy Research Working Paper Series 8139, The World Bank.
    14. Young, Alwyn, 2019. "Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results," LSE Research Online Documents on Economics 101401, London School of Economics and Political Science, LSE Library.
    15. Peng Ding & Avi Feller & Luke Miratrix, 2016. "Randomization inference for treatment effect variation," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 655-671, June.
    16. Qi Li & Jeffrey Scott Racine, 2006. "Nonparametric Econometrics: Theory and Practice," Economics Books, Princeton University Press, edition 1, volume 1, number 8355.
    17. Julius Owusu, 2024. "A Nonparametric Test of Heterogeneous Treatment Effects under Interference," Papers 2410.00733, arXiv.org.
    18. Julius Owusu, 2023. "Randomization Inference of Heterogeneous Treatment Effects under Network Interference," Papers 2308.00202, arXiv.org, revised Jan 2025.
    19. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    20. Jeffrey A. Flory & Andreas Leibbrandt & John A. List, 2016. "The Effects of Wage Contracts on Workplace Misbehaviors: Evidence from a Call Center Natural Field Experiment," NBER Working Papers 22342, National Bureau of Economic Research, Inc.

    More about this item

    Keywords

    Permutation test; Heterogeneous treatment effect; Martingale transformation; Multiple hypothesis testing; Westfall–Young;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C46 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Specific Distributions

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:225:y:2021:i:2:p:148-174. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jeconom .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.