IDEAS home Printed from https://ideas.repec.org/a/spr/testjl/v27y2018i4d10.1007_s11749-017-0571-1.html
   My bibliography  Save this article

Exact testing with random permutations

Author

Listed:
  • Jesse Hemerik

    (Leiden University Medical Center)

  • Jelle Goeman

    (Leiden University Medical Center)

Abstract

When permutation methods are used in practice, often a limited number of random permutations are used to decrease the computational burden. However, most theoretical literature assumes that the whole permutation group is used, and methods based on random permutations tend to be seen as approximate. There exists a very limited amount of literature on exact testing with random permutations, and only recently a thorough proof of exactness was given. In this paper, we provide an alternative proof, viewing the test as a “conditional Monte Carlo test” as it has been called in the literature. We also provide extensions of the result. Importantly, our results can be used to prove properties of various multiple testing procedures based on random permutations.

Suggested Citation

  • Jesse Hemerik & Jelle Goeman, 2018. "Exact testing with random permutations," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 27(4), pages 811-825, December.
    • Handle: RePEc:spr:testjl:v:27:y:2018:i:4:d:10.1007_s11749-017-0571-1
    DOI: 10.1007/s11749-017-0571-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11749-017-0571-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11749-017-0571-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Youngchao Ge & Sandrine Dudoit & Terence Speed, 2003. "Resampling-based multiple testing for microarray data analysis," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 12(1), pages 1-77, June.
    2. Dennis D. Cox & Jong Soo Lee, 2008. "Pointwise testing with functional data using the Westfall--Young randomization method," Biometrika, Biometrika Trust, vol. 95(3), pages 621-634.
    3. Nicolai Meinshausen, 2006. "False Discovery Control for Multiple Tests of Association Under General Dependence," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 33(2), pages 227-237, June.
    4. Phipson Belinda & Smyth Gordon K, 2010. "Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 9(1), pages 1-16, October.
    5. Aldo Solari & Livio Finos & Jelle J. Goeman, 2014. "Rotation-based multiple testing in the multivariate linear model," Biometrics, The International Biometric Society, vol. 70(4), pages 954-961, December.
    6. Fortunato Pesarin, 2015. "Some Elementary Theory of Permutation Tests," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 44(22), pages 4880-4892, November.
    7. Nicolai Meinshausen & Peter Buhlmann, 2005. "Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures," Biometrika, Biometrika Trust, vol. 92(4), pages 893-907, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Djogbenou, Antoine & Sufana, Razvan, 2024. "Tests for group-specific heterogeneity in high-dimensional factor models," Journal of Multivariate Analysis, Elsevier, vol. 199(C).
    2. Angel G. Angelov & Magnus Ekström, 2023. "Tests of stochastic dominance with repeated measurements data," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 107(3), pages 443-467, September.
    3. Jesse Hemerik & Jelle J. Goeman, 2021. "Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests," International Statistical Review, International Statistical Institute, vol. 89(2), pages 367-381, August.
    4. Jesse Hemerik & Jelle J. Goeman & Livio Finos, 2020. "Robust testing in generalized linear models by sign flipping score contributions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 841-864, July.
    5. Aaditya Ramdas & Rina Foygel Barber & Emmanuel J. Candès & Ryan J. Tibshirani, 2023. "Permutation Tests Using Arbitrary Permutation Distributions," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 85(2), pages 1156-1177, August.
    6. Hediger, Simon & Michel, Loris & Näf, Jeffrey, 2022. "On the use of random forest for two-sample testing," Computational Statistics & Data Analysis, Elsevier, vol. 170(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jesse Hemerik & Jelle J. Goeman, 2021. "Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests," International Statistical Review, International Statistical Institute, vol. 89(2), pages 367-381, August.
    2. Jesse Hemerik & Jelle J. Goeman & Livio Finos, 2020. "Robust testing in generalized linear models by sign flipping score contributions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 841-864, July.
    3. Ferraccioli, Federico & Sangalli, Laura M. & Finos, Livio, 2022. "Some first inferential tools for spatial regression with differential regularization," Journal of Multivariate Analysis, Elsevier, vol. 189(C).
    4. Stefano Bonnini & Michela Borghesi, 2022. "Relationship between Mental Health and Socio-Economic, Demographic and Environmental Factors in the COVID-19 Lockdown Period—A Multivariate Regression Analysis," Mathematics, MDPI, vol. 10(18), pages 1-15, September.
    5. Baolin Wu & Zhong Guan & Hongyu Zhao, 2006. "Parametric and Nonparametric FDR Estimation Revisited," Biometrics, The International Biometric Society, vol. 62(3), pages 735-744, September.
    6. Fu, Hsuan & Luger, Richard, 2022. "Multiple testing of the forward rate unbiasedness hypothesis across currencies," Journal of Empirical Finance, Elsevier, vol. 68(C), pages 232-245.
    7. Chung, Hee Cheol & Ahn, Jeongyoun, 2021. "Subspace rotations for high-dimensional outlier detection," Journal of Multivariate Analysis, Elsevier, vol. 183(C).
    8. Guo Wenge & Peddada Shyamal, 2008. "Adaptive Choice of the Number of Bootstrap Samples in Large Scale Multiple Testing," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 7(1), pages 1-21, March.
    9. Dazard, Jean-Eudes & Sunil Rao, J., 2012. "Joint adaptive mean–variance regularization and variance stabilization of high dimensional data," Computational Statistics & Data Analysis, Elsevier, vol. 56(7), pages 2317-2333.
    10. Ristl, Robin & Xi, Dong & Glimm, Ekkehard & Posch, Martin, 2018. "Optimal exact tests for multiple binary endpoints," Computational Statistics & Data Analysis, Elsevier, vol. 122(C), pages 1-17.
    11. Pini, Alessia & Stamm, Aymeric & Vantini, Simone, 2018. "Hotelling’s T2 in separable Hilbert spaces," Journal of Multivariate Analysis, Elsevier, vol. 167(C), pages 284-305.
    12. Yifan Gu & Yang Qi & Pulin Gong, 2019. "Rich-club connectivity, diverse population coupling, and dynamical activity patterns emerging from local cortical circuits," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-34, April.
    13. Sheng, Xuguang & Yang, Jingyun, 2013. "An adaptive truncated product method for combining dependent p-values," Economics Letters, Elsevier, vol. 119(2), pages 180-182.
    14. Qiu, Zhiping & Fan, Jiangyuan & Zhang, Jin-Ting & Chen, Jianwei, 2024. "Tests for equality of several covariance matrix functions for multivariate functional data," Journal of Multivariate Analysis, Elsevier, vol. 199(C).
    15. Vsevolozhskaya, O.A. & Greenwood, M.C. & Bellante, G.J. & Powell, S.L. & Lawrence, R.L. & Repasky, K.S., 2013. "Combining functions and the closure principle for performing follow-up tests in functional analysis of variance," Computational Statistics & Data Analysis, Elsevier, vol. 67(C), pages 175-184.
    16. Pittelkow Yvonne E & Wilson Susan R, 2003. "Visualisation of Gene Expression Data - the GE-biplot, the Chip-plot and the Gene-plot," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 2(1), pages 1-19, September.
    17. Bergamelli, Michele & Bianchi, Annamaria & Khalaf, Lynda & Urga, Giovanni, 2019. "Combining p-values to test for multiple structural breaks in cointegrated regressions," Journal of Econometrics, Elsevier, vol. 211(2), pages 461-482.
    18. Chaturvedi Nimisha & Menezes Renée X. de & Wieringen Wessel van & Goeman Jelle J., 2018. "A test for detecting differential indirect trans effects between two groups of samples," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 17(5), pages 1-11, October.
    19. Lippmann, Quentin, 2022. "Gender and lawmaking in times of quotas," Journal of Public Economics, Elsevier, vol. 207(C).
    20. Jolene S. Ranek & Wayne Stallaert & J. Justin Milner & Margaret Redick & Samuel C. Wolff & Adriana S. Beltran & Natalie Stanley & Jeremy E. Purvis, 2024. "DELVE: feature selection for preserving biological trajectories in single-cell data," Nature Communications, Nature, vol. 15(1), pages 1-26, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:testjl:v:27:y:2018:i:4:d:10.1007_s11749-017-0571-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.