IDEAS home Printed from https://ideas.repec.org/a/spr/testjl/v27y2018i4d10.1007_s11749-017-0571-1.html
   My bibliography  Save this article

Exact testing with random permutations

Author

Listed:
  • Jesse Hemerik

    (Leiden University Medical Center)

  • Jelle Goeman

    (Leiden University Medical Center)

Abstract

When permutation methods are used in practice, often a limited number of random permutations are used to decrease the computational burden. However, most theoretical literature assumes that the whole permutation group is used, and methods based on random permutations tend to be seen as approximate. There exists a very limited amount of literature on exact testing with random permutations, and only recently a thorough proof of exactness was given. In this paper, we provide an alternative proof, viewing the test as a “conditional Monte Carlo test” as it has been called in the literature. We also provide extensions of the result. Importantly, our results can be used to prove properties of various multiple testing procedures based on random permutations.

Suggested Citation

  • Jesse Hemerik & Jelle Goeman, 2018. "Exact testing with random permutations," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 27(4), pages 811-825, December.
    • Handle: RePEc:spr:testjl:v:27:y:2018:i:4:d:10.1007_s11749-017-0571-1
    DOI: 10.1007/s11749-017-0571-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11749-017-0571-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11749-017-0571-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Nicolai Meinshausen, 2006. "False Discovery Control for Multiple Tests of Association Under General Dependence," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 33(2), pages 227-237, June.
    2. Phipson Belinda & Smyth Gordon K, 2010. "Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 9(1), pages 1-16, October.
    3. Youngchao Ge & Sandrine Dudoit & Terence Speed, 2003. "Resampling-based multiple testing for microarray data analysis," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 12(1), pages 1-77, June.
    4. Aldo Solari & Livio Finos & Jelle J. Goeman, 2014. "Rotation-based multiple testing in the multivariate linear model," Biometrics, The International Biometric Society, vol. 70(4), pages 954-961, December.
    5. Dennis D. Cox & Jong Soo Lee, 2008. "Pointwise testing with functional data using the Westfall--Young randomization method," Biometrika, Biometrika Trust, vol. 95(3), pages 621-634.
    6. Fortunato Pesarin, 2015. "Some Elementary Theory of Permutation Tests," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 44(22), pages 4880-4892, November.
    7. Nicolai Meinshausen & Peter Buhlmann, 2005. "Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures," Biometrika, Biometrika Trust, vol. 92(4), pages 893-907, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Aaditya Ramdas & Rina Foygel Barber & Emmanuel J. Candès & Ryan J. Tibshirani, 2023. "Permutation Tests Using Arbitrary Permutation Distributions," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 85(2), pages 1156-1177, August.
    2. Djogbenou, Antoine & Sufana, Razvan, 2024. "Tests for group-specific heterogeneity in high-dimensional factor models," Journal of Multivariate Analysis, Elsevier, vol. 199(C).
    3. Hediger, Simon & Michel, Loris & Näf, Jeffrey, 2022. "On the use of random forest for two-sample testing," Computational Statistics & Data Analysis, Elsevier, vol. 170(C).
    4. Stefano Bonnini & Getnet Melak Assegie & Kamila Trzcinska, 2024. "Review about the Permutation Approach in Hypothesis Testing," Mathematics, MDPI, vol. 12(17), pages 1-29, August.
    5. N W Koning & J Hemerik, 2024. "More efficient exact group invariance testing: using a representative subgroup," Biometrika, Biometrika Trust, vol. 111(2), pages 441-458.
    6. Angel G. Angelov & Magnus Ekström, 2023. "Tests of stochastic dominance with repeated measurements data," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 107(3), pages 443-467, September.
    7. Jesse Hemerik & Jelle J. Goeman, 2021. "Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests," International Statistical Review, International Statistical Institute, vol. 89(2), pages 367-381, August.
    8. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org.
    9. Jesse Hemerik & Jelle J. Goeman & Livio Finos, 2020. "Robust testing in generalized linear models by sign flipping score contributions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 841-864, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jesse Hemerik & Jelle J. Goeman, 2021. "Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests," International Statistical Review, International Statistical Institute, vol. 89(2), pages 367-381, August.
    2. N W Koning & J Hemerik, 2024. "More efficient exact group invariance testing: using a representative subgroup," Biometrika, Biometrika Trust, vol. 111(2), pages 441-458.
    3. Jesse Hemerik & Jelle J. Goeman & Livio Finos, 2020. "Robust testing in generalized linear models by sign flipping score contributions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 841-864, July.
    4. Masha Shunko & Julie Niederhoff & Yaroslav Rosokha, 2018. "Humans Are Not Machines: The Behavioral Impact of Queueing Design on Service Time," Management Science, INFORMS, vol. 64(1), pages 453-473, January.
    5. Van Hanh Nguyen & Catherine Matias, 2014. "On Efficient Estimators of the Proportion of True Null Hypotheses in a Multiple Testing Setup," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 1167-1194, December.
    6. Ebrahimi, Nader, 2008. "Simultaneous control of false positives and false negatives in multiple hypotheses testing," Journal of Multivariate Analysis, Elsevier, vol. 99(3), pages 437-450, March.
    7. X. Jessie Jeng & Huimin Peng & Wenbin Lu, 2021. "Model Selection With Mixed Variables on the Lasso Path," Sankhya B: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 83(1), pages 170-184, May.
    8. Axel Gandy & Georg Hahn, 2016. "A Framework for Monte Carlo based Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 43(4), pages 1046-1063, December.
    9. Konrad Abramowicz & Alessia Pini & Lina Schelin & Sara Sjöstedt de Luna & Aymeric Stamm & Simone Vantini, 2023. "Domain selection and familywise error rate for functional data: A unified framework," Biometrics, The International Biometric Society, vol. 79(2), pages 1119-1132, June.
    10. Romero, Julian & Rosokha, Yaroslav, 2018. "Constructing strategies in the indefinitely repeated prisoner’s dilemma game," European Economic Review, Elsevier, vol. 104(C), pages 185-219.
    11. Ferraccioli, Federico & Sangalli, Laura M. & Finos, Livio, 2022. "Some first inferential tools for spatial regression with differential regularization," Journal of Multivariate Analysis, Elsevier, vol. 189(C).
    12. Silke Janitza & Ender Celik & Anne-Laure Boulesteix, 2018. "A computationally fast variable importance test for random forests for high-dimensional data," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 12(4), pages 885-915, December.
    13. Stefano Bonnini & Michela Borghesi, 2022. "Relationship between Mental Health and Socio-Economic, Demographic and Environmental Factors in the COVID-19 Lockdown Period—A Multivariate Regression Analysis," Mathematics, MDPI, vol. 10(18), pages 1-15, September.
    14. Guillermo Durand & Gilles Blanchard & Pierre Neuvial & Etienne Roquain, 2020. "Post hoc false positive control for structured hypotheses," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(4), pages 1114-1148, December.
    15. Zhang, Jin-Ting & Cheng, Ming-Yen & Wu, Hau-Tieng & Zhou, Bu, 2019. "A new test for functional one-way ANOVA with applications to ischemic heart screening," Computational Statistics & Data Analysis, Elsevier, vol. 132(C), pages 3-17.
    16. Baolin Wu & Zhong Guan & Hongyu Zhao, 2006. "Parametric and Nonparametric FDR Estimation Revisited," Biometrics, The International Biometric Society, vol. 62(3), pages 735-744, September.
    17. Kai Yu & William Wheeler & Qizhai Li & Andrew W. Bergen & Neil Caporaso & Nilanjan Chatterjee & Jinbo Chen, 2010. "A Partially Linear Tree-based Regression Model for Multivariate Outcomes," Biometrics, The International Biometric Society, vol. 66(1), pages 89-96, March.
    18. Angela L. Riffo-Campos & Guillermo Ayala & Juan Domingo, 2021. "Ordering of Omics Features Using Beta Distributions on Montecarlo p -Values," Mathematics, MDPI, vol. 9(11), pages 1-18, June.
    19. Axel Gandy & Georg Hahn, 2014. "MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 1083-1101, December.
    20. Fu, Hsuan & Luger, Richard, 2022. "Multiple testing of the forward rate unbiasedness hypothesis across currencies," Journal of Empirical Finance, Elsevier, vol. 68(C), pages 232-245.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:testjl:v:27:y:2018:i:4:d:10.1007_s11749-017-0571-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.