IDEAS home Printed from https://ideas.repec.org/a/bla/scjsta/v41y2014i4p1083-1101.html
   My bibliography  Save this article

MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests

Author

Listed:
  • Axel Gandy
  • Georg Hahn

Abstract

type="main" xml:id="sjos12085-abs-0001"> Consider testing multiple hypotheses using tests that can only be evaluated by simulation, such as permutation tests or bootstrap tests. This article introduces MMCTest , a sequential algorithm that gives, with arbitrarily high probability, the same classification as a specific multiple testing procedure applied to ideal p-values. The method can be used with a class of multiple testing procedures that include the Benjamini and Hochberg false discovery rate procedure and the Bonferroni correction controlling the familywise error rate. One of the key features of the algorithm is that it stops sampling for all the hypotheses that can already be decided as being rejected or non-rejected. MMCTest can be interrupted at any stage and then returns three sets of hypotheses: the rejected, the non-rejected and the undecided hypotheses. A simulation study motivated by actual biological data shows that MMCTest is usable in practice and that, despite the additional guarantee, it can be computationally more efficient than other methods.

Suggested Citation

  • Axel Gandy & Georg Hahn, 2014. "MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 1083-1101, December.
    • Handle: RePEc:bla:scjsta:v:41:y:2014:i:4:p:1083-1101
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1111/sjos.12085
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Nicolai Meinshausen, 2006. "False Discovery Control for Multiple Tests of Association Under General Dependence," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 33(2), pages 227-237, June.
    2. Guo Wenge & Peddada Shyamal, 2008. "Adaptive Choice of the Number of Bootstrap Samples in Large Scale Multiple Testing," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 7(1), pages 1-21, March.
    3. Helmut Finner & Veronika Gontscharuk & Thorsten Dickhaus, 2012. "False Discovery Rate Control of Step-Up-Down Tests with Special Emphasis on the Asymptotically Optimal Rejection Curve," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 39(2), pages 382-397, June.
    4. Gandy, Axel, 2009. "Sequential Implementation of Monte Carlo Tests With Uniformly Bounded Resampling Risk," Journal of the American Statistical Association, American Statistical Association, vol. 104(488), pages 1504-1511.
    5. Alessio Farcomeni, 2007. "Some Results on the Control of the False Discovery Rate under Dependence," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 34(2), pages 275-297, June.
    6. Hui Jiang & Julia Salzman, 2012. "Statistical properties of an early stopping rule for resampling-based multiple testing," Biometrika, Biometrika Trust, vol. 99(4), pages 973-980.
    7. Jialiang Li & Bee Tai & David Nott, 2009. "Confidence interval for the bootstrap -value and sample size calculation of the bootstrap test," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 21(5), pages 649-661.
    8. Alessio Farcomeni, 2009. "Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 36(3), pages 501-517, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hahn, Georg, 2020. "On the expected runtime of multiple testing algorithms with bounded error," Statistics & Probability Letters, Elsevier, vol. 165(C).
    2. Dong Ding & Axel Gandy & Georg Hahn, 2020. "A simple method for implementing Monte Carlo tests," Computational Statistics, Springer, vol. 35(3), pages 1373-1392, September.
    3. Axel Gandy & Georg Hahn & Dong Ding, 2020. "Implementing Monte Carlo tests with p‐value buckets," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(3), pages 950-967, September.
    4. Hahn, Georg, 2022. "Online multivariate changepoint detection with type I error control and constant time/memory updates per series," Statistics & Probability Letters, Elsevier, vol. 181(C).
    5. Georg Hahn, 2018. "Closure properties of classes of multiple testing procedures," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 102(2), pages 167-178, April.
    6. Axel Gandy & Georg Hahn, 2016. "A Framework for Monte Carlo based Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 43(4), pages 1046-1063, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Axel Gandy & Georg Hahn, 2016. "A Framework for Monte Carlo based Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 43(4), pages 1046-1063, December.
    2. Dickhaus, Thorsten, 2012. "Simultaneous statistical inference in dynamic factor models," SFB 649 Discussion Papers 2012-033, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    3. repec:hum:wpaper:sfb649dp2012-033 is not listed on IDEAS
    4. Cerioli, Andrea & Farcomeni, Alessio, 2011. "Error rates for multivariate outlier detection," Computational Statistics & Data Analysis, Elsevier, vol. 55(1), pages 544-553, January.
    5. Hahn, Georg, 2020. "On the expected runtime of multiple testing algorithms with bounded error," Statistics & Probability Letters, Elsevier, vol. 165(C).
    6. Bajgrowicz, Pierre & Scaillet, Olivier, 2012. "Technical trading revisited: False discoveries, persistence tests, and transaction costs," Journal of Financial Economics, Elsevier, vol. 106(3), pages 473-491.
    7. Dong Ding & Axel Gandy & Georg Hahn, 2020. "A simple method for implementing Monte Carlo tests," Computational Statistics, Springer, vol. 35(3), pages 1373-1392, September.
    8. Dean Palejev & Mladen Savov, 2021. "On the Convergence of the Benjamini–Hochberg Procedure," Mathematics, MDPI, vol. 9(17), pages 1-19, September.
    9. Alessio Farcomeni & Antonio Punzo, 2020. "Robust model-based clustering with mild and gross outliers," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 29(4), pages 989-1007, December.
    10. Guillermo Durand & Gilles Blanchard & Pierre Neuvial & Etienne Roquain, 2020. "Post hoc false positive control for structured hypotheses," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(4), pages 1114-1148, December.
    11. Andrew Y. Chen & Tom Zimmermann, 2022. "Publication Bias in Asset Pricing Research," Papers 2209.13623, arXiv.org, revised Sep 2023.
    12. Chen, Xiongzhi, 2020. "A strong law of large numbers for simultaneously testing parameters of Lancaster bivariate distributions," Statistics & Probability Letters, Elsevier, vol. 167(C).
    13. James, Nicholas A. & Matteson, David S., 2015. "ecp: An R Package for Nonparametric Multiple Change Point Analysis of Multivariate Data," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 62(i07).
    14. Lee, Donghwan & Lee, Youngjo, 2016. "Extended likelihood approach to multiple testing with directional error control under a hidden Markov random field model," Journal of Multivariate Analysis, Elsevier, vol. 151(C), pages 1-13.
    15. Brian D. Segal & Thomas Braun & Michael R. Elliott & Hui Jiang, 2018. "Fast approximation of small p†values in permutation tests by partitioning the permutations," Biometrics, The International Biometric Society, vol. 74(1), pages 196-206, March.
    16. A. Farcomeni & L. Finos, 2013. "FDR Control with Pseudo-Gatekeeping Based on a Possibly Data Driven Order of the Hypotheses," Biometrics, The International Biometric Society, vol. 69(3), pages 606-613, September.
    17. Habiger, Joshua D. & Adekpedjou, Akim, 2014. "Optimal rejection curves for exact false discovery rate control," Statistics & Probability Letters, Elsevier, vol. 94(C), pages 21-28.
    18. Hai Shu & Bin Nan & Robert Koeppe, 2015. "Multiple testing for neuroimaging via hidden Markov random field," Biometrics, The International Biometric Society, vol. 71(3), pages 741-750, September.
    19. Gontscharuk, Veronika & Finner, Helmut, 2013. "Asymptotic FDR control under weak dependence: A counterexample," Statistics & Probability Letters, Elsevier, vol. 83(8), pages 1888-1893.
    20. Wei Sun & Yufeng Liu & James J. Crowley & Ting-Huei Chen & Hua Zhou & Haitao Chu & Shunping Huang & Pei-Fen Kuan & Yuan Li & Darla Miller & Ginger Shaw & Yichao Wu & Vasyl Zhabotynsky & Leonard McMill, 2015. "IsoDOT Detects Differential RNA-Isoform Expression/Usage With Respect to a Categorical or Continuous Covariate With High Sensitivity and Specificity," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(511), pages 975-986, September.
    21. Nicolai Meinshausen & Peter Bühlmann, 2010. "Stability selection," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(4), pages 417-473, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:scjsta:v:41:y:2014:i:4:p:1083-1101. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0303-6898 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.