IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/8h4fe_v1.html
   My bibliography  Save this paper

How do my distributions differ? Significance testing for the Overlapping Index using Permutation Test

Author

Listed:
  • Perugini, Ambra
  • Calignano, Giulia

    (University of Padova)

  • Nucci, Massimo
  • Finos, Livio
  • Pastore, Massimiliano

Abstract

The present study introduces the application of the permutation test to the Overlapping Index, an effect size measure for comparing density distributions of groups or conditions, to estimate effects of interest in psychological science. Starting with common scenarios in psychological science research, the paper highlights the importance of relying on statistical methods that are resilient to the complexities inherent in psychological data, where assumption violations are often inevitable. A Simulation study is presented to illustrate the practical implications and reliability of the proposed test compared to commonly used alternatives. The findings demonstrate the good control of Type I error of the ζ-Overlapping test and how this approach outperforms in terms of power all other tests considered in the simulation, already with small samples. The paper offers practical guidance and demonstrates the advantages of this method, emphasizing its potential to enhance transparency and rigor in psychological data analysis by shifting focus from traditional significance testing to comprehensive distributional evaluations.

Suggested Citation

  • Perugini, Ambra & Calignano, Giulia & Nucci, Massimo & Finos, Livio & Pastore, Massimiliano, 2024. "How do my distributions differ? Significance testing for the Overlapping Index using Permutation Test," OSF Preprints 8h4fe_v1, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:8h4fe_v1
    DOI: 10.31219/osf.io/8h4fe_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/6765449d828257fb594348f5/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/8h4fe_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jesse Hemerik & Jelle Goeman, 2018. "Exact testing with random permutations," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 27(4), pages 811-825, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stefano Bonnini & Getnet Melak Assegie & Kamila Trzcinska, 2024. "Review about the Permutation Approach in Hypothesis Testing," Mathematics, MDPI, vol. 12(17), pages 1-29, August.
    2. Djogbenou, Antoine & Sufana, Razvan, 2024. "Tests for group-specific heterogeneity in high-dimensional factor models," Journal of Multivariate Analysis, Elsevier, vol. 199(C).
    3. Jesse Hemerik & Jelle J. Goeman & Livio Finos, 2020. "Robust testing in generalized linear models by sign flipping score contributions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 841-864, July.
    4. Aaditya Ramdas & Rina Foygel Barber & Emmanuel J. Candès & Ryan J. Tibshirani, 2023. "Permutation Tests Using Arbitrary Permutation Distributions," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 85(2), pages 1156-1177, August.
    5. N W Koning & J Hemerik, 2024. "More efficient exact group invariance testing: using a representative subgroup," Biometrika, Biometrika Trust, vol. 111(2), pages 441-458.
    6. Angel G. Angelov & Magnus Ekström, 2023. "Tests of stochastic dominance with repeated measurements data," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 107(3), pages 443-467, September.
    7. Jesse Hemerik & Jelle J. Goeman, 2021. "Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests," International Statistical Review, International Statistical Institute, vol. 89(2), pages 367-381, August.
    8. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org, revised Feb 2025.
    9. Hediger, Simon & Michel, Loris & Näf, Jeffrey, 2022. "On the use of random forest for two-sample testing," Computational Statistics & Data Analysis, Elsevier, vol. 170(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:8h4fe_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.