IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v165y2022ics0167947321001602.html
   My bibliography  Save this article

Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors

Author

Listed:
  • Kelter, Riko

Abstract

Hypothesis testing is a central statistical method in the biomedical sciences. The ongoing debate about the concept of statistical significance and the reliability of null hypothesis significance tests (NHST) and p-values has brought the advent of various Bayesian hypothesis tests as possible alternatives, which often employ the Bayes factor. However, careful calibration of the prior parameters is necessary for the type I error rates or power of these alternatives to be any better. Also, the availability of various Bayesian tests for the same statistical problem leads to the question which test to choose based on which criteria. Recently proposed Bayesian nonparametric two-sample tests are analyzed with regard to their type I error rates and power to detect an effect. Results show that approaches vary substantially in their ability to control the type I and II errors, and it is shown how to select the prior parameters to balance power and type I error control. This allows for prior elicitation and power analyses based on objective criteria like type I and II error rates even when conducting a Bayesian nonparametric two-sample test. Also, it is shown that existing nonparametric Bayesian two-sample tests are adequate only to test for location-shifts. Together, the results provide guidance how to perform a nonparametric Bayesian two-sample test while simultaneously improving the reliability of research.

Suggested Citation

  • Kelter, Riko, 2022. "Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors," Computational Statistics & Data Analysis, Elsevier, vol. 165(C).
    • Handle: RePEc:eee:csdana:v:165:y:2022:i:c:s0167947321001602
    DOI: 10.1016/j.csda.2021.107326
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947321001602
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2021.107326?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Basu S. & Chib S., 2003. "Marginal Likelihood and Bayes Factors for Dirichlet Process Mixture Models," Journal of the American Statistical Association, American Statistical Association, vol. 98, pages 224-235, January.
    2. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    3. Richard McElreath & Paul E Smaldino, 2015. "Replication, Communication, and the Population Dynamics of Scientific Discovery," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-16, August.
    4. Leonhard Held & Manuela Ott, 2016. "How the Maximal Evidence of -Values Against Point Null Hypotheses Depends on Sample Size," The American Statistician, Taylor & Francis Journals, vol. 70(4), pages 335-341, October.
    5. Florens, J-P & Richard, J-F & Rolin, J-M, 1996. "Bayesian Encompassing Specification Tests of a Parametric Model Against a Non Parametric Alternative," Papers 9608, Catholique de Louvain - Institut de statistique.
    6. Ronald L. Wasserstein & Allen L. Schirm & Nicole A. Lazar, 2019. "Moving to a World Beyond “p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 1-19, March.
    7. Balazs Aczel & Rink Hoekstra & Andrew Gelman & Eric-Jan Wagenmakers & Irene G. Klugkist & Jeffrey N. Rouder & Joachim Vandekerckhove & Michael D. Lee & Richard D. Morey & Wolf Vanpaemel & Zoltan Diene, 2020. "Discussion points for Bayesian inference," Nature Human Behaviour, Nature, vol. 4(6), pages 561-563, June.
    8. J. van Doorn & A. Ly & M. Marsman & E.-J. Wagenmakers, 2020. "Bayesian rank-based hypothesis testing for the rank sum test, the signed rank test, and Spearman's ρ," Journal of Applied Statistics, Taylor & Francis Journals, vol. 47(16), pages 2984-3006, December.
    9. Michael L. Pennell & David B. Dunson, 2008. "Nonparametric Bayes Testing of Changes in a Response Distribution with an Ordinal Predictor," Biometrics, The International Biometric Society, vol. 64(2), pages 413-423, June.
    10. David B. Dunson & Shyamal D. Peddada, 2008. "Bayesian nonparametric inference on stochastic ordering," Biometrika, Biometrika Trust, vol. 95(4), pages 859-874.
    11. Berger J. O & Guglielmi A., 2001. "Bayesian and Conditional Frequentist Testing of a Parametric Model Versus Nonparametric Alternatives," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 174-184, March.
    12. Julia M. Haaf & Alexander Ly & Eric-Jan Wagenmakers, 2019. "Retire significance, but still test hypotheses," Nature, Nature, vol. 567(7749), pages 461-461, March.
    13. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    14. Valen E. Johnson & David Rossell, 2010. "On the use of non‐local prior densities in Bayesian hypothesis tests," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(2), pages 143-170, March.
    15. Bhattacharya, Abhishek & Dunson, David, 2012. "Nonparametric Bayes classification and hypothesis testing on manifolds," Journal of Multivariate Analysis, Elsevier, vol. 111(C), pages 1-19.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Self, Stella & Nolan, Melissa, 2024. "A Bayesian spatial scan statistic for multinomial data," Statistics & Probability Letters, Elsevier, vol. 206(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rafael Carvalho Ceregatti & Rafael Izbicki & Luis Ernesto Bueno Salasar, 2021. "WIKS: a general Bayesian nonparametric index for quantifying differences between two populations," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 30(1), pages 274-291, March.
    2. Argiento, Raffaele & Guglielmi, Alessandra & Pievatolo, Antonio, 2010. "Bayesian density estimation and model selection using nonparametric hierarchical mixtures," Computational Statistics & Data Analysis, Elsevier, vol. 54(4), pages 816-832, April.
    3. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    4. Eric-Jan Wagenmakers & Alexandra Sarafoglou & Sil Aarts & Casper Albers & Johannes Algermissen & Štěpán Bahník & Noah Dongen & Rink Hoekstra & David Moreau & Don Ravenzwaaij & Aljaž Sluga & Franziska , 2021. "Seven steps toward more transparency in statistical practice," Nature Human Behaviour, Nature, vol. 5(11), pages 1473-1480, November.
    5. Surya T. Tokdar & Ryan Martin, 2021. "Bayesian Test of Normality Versus a Dirichlet Process Mixture Alternative," Sankhya B: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 83(1), pages 66-96, May.
    6. Bhattacharya, Abhishek & Dunson, David, 2012. "Nonparametric Bayes classification and hypothesis testing on manifolds," Journal of Multivariate Analysis, Elsevier, vol. 111(C), pages 1-19.
    7. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    8. Simon Berset & Martin Huber & Mark Schelker, 2023. "The fiscal response to revenue shocks," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 30(3), pages 814-848, June.
    9. Barrientos, Andrés F. & Canale, Antonio, 2021. "A Bayesian goodness-of-fit test for regression," Computational Statistics & Data Analysis, Elsevier, vol. 155(C).
    10. Riko Kelter, 2021. "Analysis of type I and II error rates of Bayesian and frequentist parametric and nonparametric two-sample hypothesis tests under preliminary assessment of normality," Computational Statistics, Springer, vol. 36(2), pages 1263-1288, June.
    11. Berset, Simon & Schelker, Mark, 2020. "Fiscal windfall curse," European Economic Review, Elsevier, vol. 130(C).
    12. Christoph Semken & David Rossell, 2022. "Specification analysis for technology use and teenager well‐being: Statistical validity and a Bayesian proposal," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(5), pages 1330-1355, November.
    13. John K. Kruschke, 2021. "Bayesian Analysis Reporting Guidelines," Nature Human Behaviour, Nature, vol. 5(10), pages 1282-1291, October.
    14. Rigdon, Edward E., 2023. "How improper dichotomization and the misrepresentation of uncertainty undermine social science research," Journal of Business Research, Elsevier, vol. 165(C).
    15. D. Vélez & M. E. Pérez & L. R. Pericchi, 2022. "Increasing the replicability for linear models via adaptive significance levels," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 31(3), pages 771-789, September.
    16. Erik W. van Zwet & Eric A. Cator, 2021. "The significance filter, the winner's curse and the need to shrink," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 75(4), pages 437-452, November.
    17. Beau Coker & Cynthia Rudin & Gary King, 2021. "A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results," Management Science, INFORMS, vol. 67(10), pages 6174-6197, October.
    18. Kline, Brendan, 2024. "Classical p-values and the Bayesian posterior probability that the hypothesis is approximately true," Journal of Econometrics, Elsevier, vol. 240(1).
    19. Cinzia Carota, 2006. "Some Faults of the Bayes Factor in Nonparametric Model Selection," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 15(1), pages 37-42, May.
    20. Luai Al-Labadi, 2021. "The two-sample problem via relative belief ratio," Computational Statistics, Springer, vol. 36(3), pages 1791-1808, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:165:y:2022:i:c:s0167947321001602. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.