IDEAS home Printed from https://ideas.repec.org/a/nat/nathum/v8y2024i10d10.1038_s41562-024-01931-7.html
   My bibliography  Save this article

Power and reproducibility in the external validation of brain-phenotype predictions

Author

Listed:
  • Matthew Rosenblatt

    (Yale University)

  • Link Tejavibulya

    (Yale University)

  • Huili Sun

    (Yale University)

  • Chris C. Camp

    (Yale University)

  • Milana Khaitova

    (Yale School of Medicine)

  • Brendan D. Adkinson

    (Yale University)

  • Rongtao Jiang

    (Yale School of Medicine)

  • Margaret L. Westwater

    (Yale School of Medicine)

  • Stephanie Noble

    (Yale School of Medicine
    Northeastern University
    Northeastern University)

  • Dustin Scheinost

    (Yale University
    Yale University
    Yale School of Medicine
    Yale School of Medicine)

Abstract

Brain-phenotype predictive models seek to identify reproducible and generalizable brain-phenotype associations. External validation, or the evaluation of a model in external datasets, is the gold standard in evaluating the generalizability of models in neuroimaging. Unlike typical studies, external validation involves two sample sizes: the training and the external sample sizes. Thus, traditional power calculations may not be appropriate. Here we ran over 900 million resampling-based simulations in functional and structural connectivity data to investigate the relationship between training sample size, external sample size, phenotype effect size, theoretical power and simulated power. Our analysis included a wide range of datasets: the Healthy Brain Network, the Adolescent Brain Cognitive Development Study, the Human Connectome Project (Development and Young Adult), the Philadelphia Neurodevelopmental Cohort, the Queensland Twin Adolescent Brain Project, and the Chinese Human Connectome Project; and phenotypes: age, body mass index, matrix reasoning, working memory, attention problems, anxiety/depression symptoms and relational processing. High effect size predictions achieved adequate power with training and external sample sizes of a few hundred individuals, whereas low and medium effect size predictions required hundreds to thousands of training and external samples. In addition, most previous external validation studies used sample sizes prone to low power, and theoretical power curves should be adjusted for the training sample size. Furthermore, model performance in internal validation often informed subsequent external validation performance (Pearson’s r difference

Suggested Citation

  • Matthew Rosenblatt & Link Tejavibulya & Huili Sun & Chris C. Camp & Milana Khaitova & Brendan D. Adkinson & Rongtao Jiang & Margaret L. Westwater & Stephanie Noble & Dustin Scheinost, 2024. "Power and reproducibility in the external validation of brain-phenotype predictions," Nature Human Behaviour, Nature, vol. 8(10), pages 2018-2033, October.
  • Handle: RePEc:nat:nathum:v:8:y:2024:i:10:d:10.1038_s41562-024-01931-7
    DOI: 10.1038/s41562-024-01931-7
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41562-024-01931-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1038/s41562-024-01931-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:nathum:v:8:y:2024:i:10:d:10.1038_s41562-024-01931-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.