IDEAS home Printed from https://ideas.repec.org/a/spr/stpapr/v65y2024i6d10.1007_s00362-024-01528-2.html
   My bibliography  Save this article

Is Fisher inference inferior to Neyman inference for policy analysis?

Author

Listed:
  • Rauf Ahmad

    (Uppsala University)

  • Per Johansson

    (Uppsala University and YMSC, Tsinghua University)

  • Mårten Schultzberg

    (Uppsala University)

Abstract

The increasing computational power has led to an increasing interest in Fisher’s test in social science. As the Fisher and Neyman inference are based on different principles there is also an increasing interest in understanding the differential features of the two procedures. For example, Young (2018) found that the Fisher test has better size properties than the Neyman test in the situation with influential observations. Ding (2017), on the other hand, showed that the asymptotic variance of the mean-difference estimator (MDE) under Fisher inference is larger than that under Neyman inference, and that the asymptotic Fisher test is less powerful than the t-test even for the simplest case of homogeneous effect. Since MDE plays an important role for policy evaluation, these latter results are a concern for using Fisher’s test as argued in Young (2018). With the aim of providing an understanding of the usefulness of the exact Fisher test for inference to the sample and to the population, this paper clarifies the results in Ding (2017). Using a novel Monte Carlo simulation following the same data generating processes as in Ding (2017), we demonstrate that the Fisher test has no worse power properties than the t-test even with heterogeneous effects.

Suggested Citation

  • Rauf Ahmad & Per Johansson & Mårten Schultzberg, 2024. "Is Fisher inference inferior to Neyman inference for policy analysis?," Statistical Papers, Springer, vol. 65(6), pages 3425-3445, August.
    • Handle: RePEc:spr:stpapr:v:65:y:2024:i:6:d:10.1007_s00362-024-01528-2
    DOI: 10.1007/s00362-024-01528-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00362-024-01528-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00362-024-01528-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Per Johansson & Donald B. Rubin & Mårten Schultzberg, 2021. "On optimal rerandomization designs," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(2), pages 395-403, April.
    2. Xinran Li & Peng Ding, 2017. "General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1759-1769, October.
    3. Jason Wu & Peng Ding, 2021. "Randomization Tests for Weak Null Hypotheses in Randomized Experiments," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1898-1913, October.
    4. Rosenbaum, Paul R., 2007. "Interference Between Units in Randomized Experiments," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 191-200, March.
    5. Peng Ding & Tirthankar Dasgupta, 2018. "A randomization-based perspective on analysis of variance: a test statistic robust to treatment effect heterogeneity," Biometrika, Biometrika Trust, vol. 105(1), pages 45-56.
    6. Adam Kapelner & Abba M. Krieger & Michael Sklar & Uri Shalit & David Azriel, 2021. "Harmonizing Optimized Designs With Classic Randomization in Experiments," The American Statistician, Taylor & Francis Journals, vol. 75(2), pages 195-206, May.
    7. Dimitris Bertsimas & Mac Johnson & Nathan Kallus, 2015. "The Power of Optimization Over Randomization in Designing Experiments Involving Small Samples," Operations Research, INFORMS, vol. 63(4), pages 868-876, August.
    8. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ke Zhu & Hanzhong Liu, 2023. "Pair‐switching rerandomization," Biometrics, The International Biometric Society, vol. 79(3), pages 2127-2142, September.
    2. Zhao, Anqi & Ding, Peng, 2024. "No star is good news: A unified look at rerandomization based on p-values from covariate balance tests," Journal of Econometrics, Elsevier, vol. 241(1).
    3. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org.
    4. Han, Kevin & Basse, Guillaume & Bojinov, Iavor, 2024. "Population interference in panel experiments," Journal of Econometrics, Elsevier, vol. 238(1).
    5. Xiaokang Luo & Tirthankar Dasgupta & Minge Xie & Regina Y. Liu, 2021. "Leveraging the Fisher randomization test using confidence distributions: Inference, combination and fusion learning," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(4), pages 777-797, September.
    6. Shaina J. Alexandria & Michael G. Hudgens & Allison E. Aiello, 2023. "Assessing intervention effects in a randomized trial within a social network," Biometrics, The International Biometric Society, vol. 79(2), pages 1409-1419, June.
    7. Haoge Chang, 2023. "Design-based Estimation Theory for Complex Experiments," Papers 2311.06891, arXiv.org.
    8. Peter L. Cohen & Colin B. Fogarty, 2022. "Gaussian prepivoting for finite population causal inference," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 295-320, April.
    9. Fangzhou Su & Peng Ding, 2021. "Model‐assisted analyses of cluster‐randomized experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(5), pages 994-1015, November.
    10. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    11. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org.
    12. Purevdorj Tuvaandorj, 2024. "A Combinatorial Central Limit Theorem for Stratified Randomization," Papers 2402.14764, arXiv.org, revised Apr 2024.
    13. Nicolaj N. Mühlbach, 2020. "Tree-based Synthetic Control Methods: Consequences of moving the US Embassy," CREATES Research Papers 2020-04, Department of Economics and Business Economics, Aarhus University.
    14. Michael L. Anderson & Fangwen Lu, 2017. "Learning to Manage and Managing to Learn: The Effects of Student Leadership Service," Management Science, INFORMS, vol. 63(10), pages 3246-3261, October.
    15. Giovanni Cerulli, 2014. "ntreatreg: a Stata module for estimation of treatment effects in the presence of neighborhood interactions," United Kingdom Stata Users' Group Meetings 2014 15, Stata Users Group.
    16. Eric C. Brown & John W. Graham & J. David Hawkins & Michael W. Arthur & Megan M. Baldwin & Sabrina Oesterle & John S. Briney & Richard F. Catalano & Robert D. Abbott, 2009. "Design and Analysis of the Community Youth Development Study Longitudinal Cohort Sample," Evaluation Review, , vol. 33(4), pages 311-334, August.
    17. Clément de Chaisemartin, 2022. "Trading-off Bias and Variance in Stratified Experiments and in Staggered Adoption Designs, Under a Boundedness Condition on the Magnitude of the Treatment Effect," Working Papers hal-03873919, HAL.
    18. Ashesh Rambachan & Jonathan Roth, 2020. "Design-Based Uncertainty for Quasi-Experiments," Papers 2008.00602, arXiv.org, revised Oct 2024.
    19. David Escamilla-Guerrero & Edward Kosack & Zachary Ward, 2023. "The Impact of Violence during the Mexican Revolution on Migration to the United States," NBER Working Papers 31531, National Bureau of Economic Research, Inc.
    20. Kiyoyasu Tanaka, 2021. "The European Union's reform in rules of origin and international trade: Evidence from Cambodia," The World Economy, Wiley Blackwell, vol. 44(10), pages 3025-3050, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:stpapr:v:65:y:2024:i:6:d:10.1007_s00362-024-01528-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.