IDEAS home Printed from https://ideas.repec.org/a/spr/compst/v35y2020i3d10.1007_s00180-019-00927-6.html
   My bibliography  Save this article

A simple method for implementing Monte Carlo tests

Author

Listed:
  • Dong Ding

    (Imperial College London)

  • Axel Gandy

    (Imperial College London)

  • Georg Hahn

    (Lancaster University)

Abstract

We consider a statistical test whose p value can only be approximated using Monte Carlo simulations. We are interested in deciding whether the p value for an observed data set lies above or below a given threshold such as 5%. We want to ensure that the resampling risk, the probability of the (Monte Carlo) decision being different from the true decision, is uniformly bounded. This article introduces a simple open-ended method with this property, the confidence sequence method (CSM). We compare our approach to another algorithm, SIMCTEST, which also guarantees an (asymptotic) uniform bound on the resampling risk, as well as to other Monte Carlo procedures without a uniform bound. CSM is free of tuning parameters and conservative. It has the same theoretical guarantee as SIMCTEST and, in many settings, similar stopping boundaries. As it is much simpler than other methods, CSM is a useful method for practical applications.

Suggested Citation

  • Dong Ding & Axel Gandy & Georg Hahn, 2020. "A simple method for implementing Monte Carlo tests," Computational Statistics, Springer, vol. 35(3), pages 1373-1392, September.
    • Handle: RePEc:spr:compst:v:35:y:2020:i:3:d:10.1007_s00180-019-00927-6
    DOI: 10.1007/s00180-019-00927-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00180-019-00927-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00180-019-00927-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. James G. MacKinnon, 2002. "Bootstrap inference in econometrics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 35(4), pages 615-645, November.
    2. Axel Gandy & Georg Hahn, 2014. "MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 1083-1101, December.
    3. Martin Kulldorff, 2001. "Prospective time periodic geographical disease surveillance using a scan statistic," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 164(1), pages 61-72.
    4. Russell Davidson & James MacKinnon, 2000. "Bootstrap tests: how many bootstraps?," Econometric Reviews, Taylor & Francis Journals, vol. 19(1), pages 55-68.
    5. Ali, Mukhtar M, 1987. "Durbin-Watson and Generalized Durbin-Watson Tests for Autocorrelations and Randomness," Journal of Business & Economic Statistics, American Statistical Association, vol. 5(2), pages 195-203, April.
    6. Ali, Mukhtar M, 1984. "Distributions of the Sample Autocorrelations When Observations Are from a Stationary Autoregressive-Moving-Average Process," Journal of Business & Economic Statistics, American Statistical Association, vol. 2(3), pages 271-278, July.
    7. Silva, Ivair R. & Assunção, Renato M., 2013. "Optimal generalized truncated sequential Monte Carlo test," Journal of Multivariate Analysis, Elsevier, vol. 121(C), pages 33-49.
    8. Axel Gandy & Georg Hahn, 2016. "A Framework for Monte Carlo based Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 43(4), pages 1046-1063, December.
    9. Gandy, Axel, 2009. "Sequential Implementation of Monte Carlo Tests With Uniformly Bounded Resampling Risk," Journal of the American Statistical Association, American Statistical Association, vol. 104(488), pages 1504-1511.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hahn, Georg, 2020. "On the expected runtime of multiple testing algorithms with bounded error," Statistics & Probability Letters, Elsevier, vol. 165(C).
    2. Axel Gandy & Georg Hahn & Dong Ding, 2020. "Implementing Monte Carlo tests with p‐value buckets," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(3), pages 950-967, September.
    3. M.L. Nores & M.P. Díaz, 2016. "Bootstrap hypothesis testing in generalized additive models for comparing curves of treatments in longitudinal studies," Journal of Applied Statistics, Taylor & Francis Journals, vol. 43(5), pages 810-826, April.
    4. repec:ebl:ecbull:v:30:y:2010:i:1:p:55-66 is not listed on IDEAS
    5. A. Melander & G. Sismanidis & D. Grenouilleau, 2007. "The track record of the Commission's forecasts - an update," European Economy - Economic Papers 2008 - 2015 291, Directorate General Economic and Financial Affairs (DG ECFIN), European Commission.
    6. Davidson, Russell & MacKinnon, James G., 2007. "Improving the reliability of bootstrap tests with the fast double bootstrap," Computational Statistics & Data Analysis, Elsevier, vol. 51(7), pages 3259-3281, April.
    7. Trenkler, Carsten, 2009. "Bootstrapping Systems Cointegration Tests With A Prior Adjustment For Deterministic Terms," Econometric Theory, Cambridge University Press, vol. 25(1), pages 243-269, February.
    8. Emmanuel Flachaire, 2005. "More Efficient Tests Robust to Heteroskedasticity of Unknown Form," Econometric Reviews, Taylor & Francis Journals, vol. 24(2), pages 219-241.
    9. JAMES G. MacKINNON, 2006. "Bootstrap Methods in Econometrics," The Economic Record, The Economic Society of Australia, vol. 82(s1), pages 2-18, September.
    10. repec:hum:wpaper:sfb649dp2006-012 is not listed on IDEAS
    11. Silva, Ivair R. & Assunção, Renato M., 2013. "Optimal generalized truncated sequential Monte Carlo test," Journal of Multivariate Analysis, Elsevier, vol. 121(C), pages 33-49.
    12. James G. MacKinnon, 2012. "Thirty Years Of Heteroskedasticity-robust Inference," Working Paper 1268, Economics Department, Queen's University.
    13. Jamel JOUINI & Mohamed Boutahar, 2010. "The finite-sample properties of bootstrap tests in multiple structural change models," Economics Bulletin, AccessEcon, vol. 30(1), pages 55-66.
    14. James Morley & Irina B. Panovska & Tara M. Sinclair, 2014. "Testing Stationarity for Unobserved Components Models," Discussion Papers 2012-41B, School of Economics, The University of New South Wales.
    15. James Morley & Irina B. Panovska & Tara M. Sinclair, 2013. "Testing Stationarity for Unobserved Components Models," Discussion Papers 2012-41A, School of Economics, The University of New South Wales.
    16. Georg Hahn, 2018. "Closure properties of classes of multiple testing procedures," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 102(2), pages 167-178, April.
    17. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    18. James G. MacKinnon, 2007. "Bootstrap Hypothesis Testing," Working Paper 1127, Economics Department, Queen's University.
    19. Dean V. Williamson, 2010. "Financial-Market Contracting," Chapters, in: Peter G. Klein & Michael E. Sykuta (ed.), The Elgar Companion to Transaction Cost Economics, chapter 24, Edward Elgar Publishing.
    20. Li, Ying & Liu, Zhen & Qin, Kuiyuan & Cui, Jiayu & Zeng, Xiaoyu & Ji, Ming & Lan, Jijun & You, Xuqun & Li, Yuan, 2021. "Organizational trust and safety operation behavior in airline pilots: The mediating effects of organizational identification and organizational commitment," Journal of Air Transport Management, Elsevier, vol. 92(C).
    21. Johannesson Magnus & Östling Robert & Ranehill Eva, 2010. "The Effect of Competition on Physical Activity: A Randomized Trial," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 10(1), pages 1-31, September.
    22. Daiki Maki & Yasushi Ota, 2021. "Testing for Time-Varying Properties Under Misspecified Conditional Mean and Variance," Computational Economics, Springer;Society for Computational Economics, vol. 57(4), pages 1167-1182, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:compst:v:35:y:2020:i:3:d:10.1007_s00180-019-00927-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.