IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-53743-y.html
   My bibliography  Save this article

Why experimental variation in neuroimaging should be embraced

Author

Listed:
  • Gregory Kiar

    (Child Mind Institute)

  • Jeanette A. Mumford

    (Stanford University)

  • Ting Xu

    (Child Mind Institute
    Child Mind Institute)

  • Joshua T. Vogelstein

    (Johns Hopkins University)

  • Tristan Glatard

    (The Centre for Addiction and Mental Health)

  • Michael P. Milham

    (Child Mind Institute
    Child Mind Institute)

Abstract

In a perfect world, scientists would develop analyses that are guaranteed to reveal the ground truth of a research question. In reality, there are countless viable workflows that produce distinct, often conflicting, results. Although reproducibility places a necessary bound on the validity of results, it is not sufficient for claiming underlying validity, eventual utility, or generalizability. In this work we focus on how embracing variability in data analysis can improve the generalizability of results. We contextualize how design decisions in brain imaging can be made to capture variation, highlight examples, and discuss how variability capture may improve the quality of results.

Suggested Citation

  • Gregory Kiar & Jeanette A. Mumford & Ting Xu & Joshua T. Vogelstein & Tristan Glatard & Michael P. Milham, 2024. "Why experimental variation in neuroimaging should be embraced," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-53743-y
    DOI: 10.1038/s41467-024-53743-y
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-53743-y
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-53743-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rotem Botvinik-Nezer & Felix Holzmeister & Colin F. Camerer & Anna Dreber & Juergen Huber & Magnus Johannesson & Michael Kirchler & Roni Iwanir & Jeanette A. Mumford & R. Alison Adcock & Paolo Avesani, 2020. "Variability in the analysis of a single neuroimaging dataset by many teams," Nature, Nature, vol. 582(7810), pages 84-88, June.
    2. Eric W Bridgeford & Shangsi Wang & Zeyi Wang & Ting Xu & Cameron Craddock & Jayanta Dey & Gregory Kiar & William Gray-Roncal & Carlo Colantuoni & Christopher Douville & Stephanie Noble & Carey E Prieb, 2021. "Eliminating accidental deviations to minimize generalization error and maximize replicability: Applications in connectomics and genomics," PLOS Computational Biology, Public Library of Science, vol. 17(9), pages 1-20, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dennis Bontempi & Leonard Nuernberg & Suraj Pai & Deepa Krishnaswamy & Vamsi Thiriveedhi & Ahmed Hosny & Raymond H. Mak & Keyvan Farahani & Ron Kikinis & Andrey Fedorov & Hugo J. W. L. Aerts, 2024. "End-to-end reproducible AI pipelines in radiology using the cloud," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    2. Matthew Rosenblatt & Link Tejavibulya & Rongtao Jiang & Stephanie Noble & Dustin Scheinost, 2024. "Data leakage inflates prediction performance in connectome-based machine learning models," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    3. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    4. Felix Holzmeister & Magnus Johannesson & Colin F. Camerer & Yiling Chen & Teck-Hua Ho & Suzanne Hoogeveen & Juergen Huber & Noriko Imai & Taisuke Imai & Lawrence Jin & Michael Kirchler & Alexander Ly , 2025. "Examining the replicability of online experiments selected by a decision market," Nature Human Behaviour, Nature, vol. 9(2), pages 316-330, February.
    5. Christophe Pérignon & Olivier Akmansoy & Christophe Hurlin & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johanneson & Michael Kirchler & Albert Menkveld & Michael Razen & Utz Weitzel, 2022. "Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance," Working Papers hal-03810013, HAL.
    6. Joakim A. Weill & Matthieu Stigler & Olivier Deschenes & Michael R. Springborn, 2021. "Researchers' Degrees-of-Flexibility and the Credibility of Difference-in-Differences Estimates: Evidence From the Pandemic Policy Evaluations," NBER Working Papers 29550, National Bureau of Economic Research, Inc.
    7. Breznau, Nate & Auspurg, Katrin & Brüderl, Josef & Holzmeister, Felix & Nilsonne, Gustav & Aczel, Balazs & Clark, Cory J & Nguyen, Hung Hoang Viet & Varga, Marton Aron & Uhlmann, Eric Luis, 2023. "Pre-Registered Analysis Plan: Is the Inter-Researcher Variability in Social Scientific Results Explicable? An Adversarial Collaboration and Joint Effort to Parse Model and Estimate Dispersion," MetaArXiv h9t5c_v1, Center for Open Science.
    8. Dario Krpan & Jonathan E. Booth & Andreea Damien, 2023. "The positive–negative–competence (PNC) model of psychological responses to representations of robots," Nature Human Behaviour, Nature, vol. 7(11), pages 1933-1954, November.
    9. Andrea I. Luppi & Helena M. Gellersen & Zhen-Qi Liu & Alexander R. D. Peattie & Anne E. Manktelow & Ram Adapa & Adrian M. Owen & Lorina Naci & David K. Menon & Stavros I. Dimitriadis & Emmanuel A. Sta, 2024. "Systematic evaluation of fMRI data-processing pipelines for consistent functional connectomics," Nature Communications, Nature, vol. 15(1), pages 1-24, December.
    10. Gabriel Nova & Sander van Cranenburgh & Stephane Hess, 2024. "Understanding the decision-making process of choice modellers," Papers 2411.01704, arXiv.org.
    11. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    12. Christoph Huber & Anna Dreber & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Utz Weitzel & Miguel Abellán & Xeniya Adayeva & Fehime Ceren Ay & Kai Barron & Zachariah Berry & Werner Bönte , 2023. "Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(23), pages 2215572120-, June.
    13. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    14. Campbell, Douglas & Brodeur, Abel & Dreber, Anna & Johannesson, Magnus & Kopecky, Joseph & Lusher, Lester & Tsoy, Nikita, 2024. "The Robustness Reproducibility of the American Economic Review," I4R Discussion Paper Series 124, The Institute for Replication (I4R).
    15. Sadri, Arash, 2022. "Machine Learning Can Solve the Reproducibility Crisis by Supplanting Reductionist Statistics," MetaArXiv yxba5_v1, Center for Open Science.
    16. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArXiv 2b5k4, Center for Open Science.
    17. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArchive 2b5k4_v1, Center for Open Science.
    18. Farid Aboharb & Pasha A. Davoudian & Ling-Xiao Shao & Clara Liao & Gillian N. Rzepka & Cassandra Wojtasiewicz & Jonathan Indajang & Mark Dibbs & Jocelyne Rondeau & Alexander M. Sherwood & Alfred P. Ka, 2025. "Classification of psychedelics and psychoactive drugs based on brain-wide imaging of cellular c-Fos expression," Nature Communications, Nature, vol. 16(1), pages 1-15, December.
    19. Auer, Tobias & Ulasik, Maria & Holzmeister, Felix, 2024. "A Comment on "Motivated Errors" by Exley and Kessler (2024)," I4R Discussion Paper Series 161, The Institute for Replication (I4R).
    20. Rotem Botvinik-Nezer & Bogdan Petre & Marta Ceko & Martin A. Lindquist & Naomi P. Friedman & Tor D. Wager, 2024. "Placebo treatment affects brain systems related to affective and cognitive processes, but not nociceptive pain," Nature Communications, Nature, vol. 15(1), pages 1-20, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-53743-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.