IDEAS home Printed from https://ideas.repec.org/a/oup/biomet/v109y2022i3p569-587..html
   My bibliography  Save this article

Multi-scale Fisher’s independence test for multivariate dependence
[A simple measure of conditional dependence]

Author

Listed:
  • S Gorsky
  • L Ma

Abstract

SummaryIdentifying dependency in multivariate data is a common inference task that arises in numerous applications. However, existing nonparametric independence tests typically require computation that scales at least quadratically with the sample size, making it difficult to apply them in the presence of massive sample sizes. Moreover, resampling is usually necessary to evaluate the statistical significance of the resulting test statistics at finite sample sizes, further worsening the computational burden. We introduce a scalable, resampling-free approach to testing the independence between two random vectors by breaking down the task into simple univariate tests of independence on a collection ofcontingency tables constructed through sequential coarse-to-fine discretization of the sample , transforming the inference task into a multiple testing problem that can be completed with almost linear complexity with respect to the sample size. To address increasing dimensionality, we introduce a coarse-to-fine sequential adaptive procedure that exploits the spatial features of dependency structures. We derive a finite-sample theory that guarantees the inferential validity of our adaptive procedure at any given sample size. We show that our approach can achieve strong control of the level of the testing procedure at any sample size without resampling or asymptotic approximation and establish its large-sample consistency. We demonstrate through an extensive simulation study its substantial computational advantage in comparison to existing approaches while achieving robust statistical power under various dependency scenarios, and illustrate how its divide-and-conquer nature can be exploited to not just test independence, but to learn the nature of the underlying dependency. Finally, we demonstrate the use of our method through analysing a dataset from a flow cytometry experiment.

Suggested Citation

  • S Gorsky & L Ma, 2022. "Multi-scale Fisher’s independence test for multivariate dependence [A simple measure of conditional dependence]," Biometrika, Biometrika Trust, vol. 109(3), pages 569-587.
  • Handle: RePEc:oup:biomet:v:109:y:2022:i:3:p:569-587.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/biomet/asac013
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Li Ma & Jialiang Mao, 2019. "Fisher Exact Scanning for Dependency," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(525), pages 245-258, January.
    2. Meintanis, Simos G. & Iliopoulos, George, 2008. "Fourier methods for testing multivariate independence," Computational Statistics & Data Analysis, Elsevier, vol. 52(4), pages 1884-1895, January.
    3. Niklas Pfister & Peter Bühlmann & Bernhard Schölkopf & Jonas Peters, 2018. "Kernel‐based tests for joint independence," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(1), pages 5-31, January.
    4. Székely, Gábor J. & Rizzo, Maria L., 2013. "The distance correlation t-test of independence in high dimension," Journal of Multivariate Analysis, Elsevier, vol. 117(C), pages 193-213.
    5. Kai Zhang, 2019. "BET on Independence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(528), pages 1620-1637, October.
    6. Hongjian Shi & Mathias Drton & Fang Han, 2022. "Distribution-Free Consistent Independence Tests via Center-Outward Ranks and Signs," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(537), pages 395-410, January.
    7. Ruth Heller & Yair Heller & Malka Gorfine, 2013. "A consistent multivariate test of association based on ranks of distances," Biometrika, Biometrika Trust, vol. 100(2), pages 503-510.
    8. L Weihs & M Drton & N Meinshausen, 2018. "Symmetric rank covariances: a generalized framework for nonparametric measures of dependence," Biometrika, Biometrika Trust, vol. 105(3), pages 547-562.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. S Gorsky & L Ma, 2022. "Rejoinder: ‘Multi-scale Fisher’s independence test for multivariate dependence’ [Discussion of ‘Multi-scale Fisher’s independence test for multivariate dependence’]," Biometrika, Biometrika Trust, vol. 109(3), pages 605-609.
    2. Schrab, Antonin & Jitkrittum, Wittawat & Szabo, Zoltan & Sejdinovic, Dino & Gretton, Arthur, 2022. "Discussion of ‘Multi-scale Fisher’s independence test for multivariate dependence’," LSE Research Online Documents on Economics 115629, London School of Economics and Political Science, LSE Library.
    3. D Lee & H El-Zaatari & M R Kosorok & X Li & K Zhang, 2022. "Discussion of ‘Multi-scale Fisher’s independence test for multivariate dependence’ [Multi-scale Fisher’s independence test for multivariate dependence]," Biometrika, Biometrika Trust, vol. 109(3), pages 593-596.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hongjian Shi & Marc Hallin & Mathias Drton & Fang Han, 2020. "Rate-Optimality of Consistent Distribution-Free Tests of Independence Based on Center-Outward Ranks and Signs," Working Papers ECARES 2020-23, ULB -- Universite Libre de Bruxelles.
    2. Zhang, Qingyang, 2019. "Independence test for large sparse contingency tables based on distance correlation," Statistics & Probability Letters, Elsevier, vol. 148(C), pages 17-22.
    3. Hongjian Shi & Mathias Drton & Marc Hallin & Fang Han, 2023. "Semiparametrically Efficient Tests of Multivariate Independence Using Center-Outward Quadrant, Spearman, and Kendall Statistics," Working Papers ECARES 2023-03, ULB -- Universite Libre de Bruxelles.
    4. S Gorsky & L Ma, 2022. "Rejoinder: ‘Multi-scale Fisher’s independence test for multivariate dependence’ [Discussion of ‘Multi-scale Fisher’s independence test for multivariate dependence’]," Biometrika, Biometrika Trust, vol. 109(3), pages 605-609.
    5. Simos G. Meintanis & Joseph Ngatchou-Wandji & James Allison, 2018. "Testing for serial independence in vector autoregressive models," Statistical Papers, Springer, vol. 59(4), pages 1379-1410, December.
    6. Ma, Zichen & Hanson, Timothy E., 2020. "Bayesian nonparametric test for independence between random vectors," Computational Statistics & Data Analysis, Elsevier, vol. 149(C).
    7. Roy, Angshuman & Ghosh, Anil K., 2020. "Some tests of independence based on maximum mean discrepancy and ranks of nearest neighbors," Statistics & Probability Letters, Elsevier, vol. 164(C).
    8. Fan, Yanan & de Micheaux, Pierre Lafaye & Penev, Spiridon & Salopek, Donna, 2017. "Multivariate nonparametric test of independence," Journal of Multivariate Analysis, Elsevier, vol. 153(C), pages 189-210.
    9. Chen, Feifei & Meintanis, Simos G. & Zhu, Lixing, 2019. "On some characterizations and multidimensional criteria for testing homogeneity, symmetry and independence," Journal of Multivariate Analysis, Elsevier, vol. 173(C), pages 125-144.
    10. Tahani Coolen-Maturi, 2016. "New weighted rank correlation coefficients sensitive to agreement on top and bottom rankings," Journal of Applied Statistics, Taylor & Francis Journals, vol. 43(12), pages 2261-2279, September.
    11. Wu, Edmond H.C. & Yu, Philip L.H. & Li, W.K., 2009. "A smoothed bootstrap test for independence based on mutual information," Computational Statistics & Data Analysis, Elsevier, vol. 53(7), pages 2524-2536, May.
    12. Rauf Ahmad, M., 2019. "A significance test of the RV coefficient in high dimensions," Computational Statistics & Data Analysis, Elsevier, vol. 131(C), pages 116-130.
    13. M. D. Jiménez-Gamero & J. L. Moreno-Rebollo & J. A. Mayor-Gallego, 2018. "On the estimation of the characteristic function in finite populations with applications," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 27(1), pages 95-121, March.
    14. Hyodo, Masashi & Nishiyama, Takahiro & Pavlenko, Tatjana, 2020. "Testing for independence of high-dimensional variables: ρV-coefficient based approach," Journal of Multivariate Analysis, Elsevier, vol. 178(C).
    15. David Todem & Wei‐Wen Hsu & KyungMann Kim, 2023. "Nonparametric scanning tests of homogeneity for hierarchical models with continuous covariates," Biometrics, The International Biometric Society, vol. 79(3), pages 2063-2075, September.
    16. Meintanis, Simos G. & Hušková, Marie & Hlávka, Zdeněk, 2022. "Fourier-type tests of mutual independence between functional time series," Journal of Multivariate Analysis, Elsevier, vol. 189(C).
    17. Fang Han, 2024. "An Introduction to Permutation Processes (version 0.5)," Papers 2407.09664, arXiv.org.
    18. Teran Hidalgo, Sebastian J. & Wu, Michael C. & Engel, Stephanie M. & Kosorok, Michael R., 2018. "Goodness-of-fit test for nonparametric regression models: Smoothing spline ANOVA models as example," Computational Statistics & Data Analysis, Elsevier, vol. 122(C), pages 135-155.
    19. Cui, Hengjian & Zhong, Wei, 2019. "A distribution-free test of independence based on mean variance index," Computational Statistics & Data Analysis, Elsevier, vol. 139(C), pages 117-133.
    20. Manuel Febrero-Bande & Wenceslao González-Manteiga & Manuel Oviedo de la Fuente, 2019. "Variable selection in functional additive regression models," Computational Statistics, Springer, vol. 34(2), pages 469-487, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:biomet:v:109:y:2022:i:3:p:569-587.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/biomet .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.