IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-50310-3.html
   My bibliography  Save this article

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

Author

Listed:
  • Benjamin Lahner

    (MIT)

  • Kshitij Dwivedi

    (Freie Universität Berlin
    Goethe University Frankfurt)

  • Polina Iamshchinina

    (Freie Universität Berlin)

  • Monika Graumann

    (Freie Universität Berlin)

  • Alex Lascelles

    (MIT)

  • Gemma Roig

    (Goethe University Frankfurt
    The Hessian Center for AI (hessian.AI))

  • Alessandro Thomas Gifford

    (Freie Universität Berlin)

  • Bowen Pan

    (MIT)

  • SouYoung Jin

    (MIT)

  • N. Apurva Ratan Murty

    (MIT
    Georgia Institute of Technology)

  • Kendrick Kay

    (University of Minnesota)

  • Aude Oliva

    (MIT)

  • Radoslaw Cichy

    (Freie Universität Berlin)

Abstract

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos’ extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.

Suggested Citation

  • Benjamin Lahner & Kshitij Dwivedi & Polina Iamshchinina & Monika Graumann & Alex Lascelles & Gemma Roig & Alessandro Thomas Gifford & Bowen Pan & SouYoung Jin & N. Apurva Ratan Murty & Kendrick Kay & , 2024. "Modeling short visual events through the BOLD moments video fMRI dataset and metadata," Nature Communications, Nature, vol. 15(1), pages 1-26, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-50310-3
    DOI: 10.1038/s41467-024-50310-3
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-50310-3
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-50310-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Bart Krekelberg & Sabine Dannenberg & Klaus-Peter Hoffmann & Frank Bremmer & John Ross, 2003. "Neural correlates of implied motion," Nature, Nature, vol. 424(6949), pages 674-677, August.
    2. Scott L Fairhall & Angela Albi & David Melcher, 2014. "Temporal Integration Windows for Naturalistic Visual Sequences," PLOS ONE, Public Library of Science, vol. 9(7), pages 1-6, July.
    3. Agustin Lage-Castellanos & Giancarlo Valente & Elia Formisano & Federico De Martino, 2019. "Methods for computing the maximum performance of computational models of fMRI responses," PLOS Computational Biology, Public Library of Science, vol. 15(3), pages 1-25, March.
    4. Jonathan D. Cohen & William M. Perlstein & Todd S. Braver & Leigh E. Nystrom & Douglas C. Noll & John Jonides & Edward E. Smith, 1997. "Temporal dynamics of brain activation during a working memory task," Nature, Nature, vol. 386(6625), pages 604-608, April.
    5. Tomoyasu Horikawa & Yukiyasu Kamitani, 2017. "Generic decoding of seen and imagined objects using hierarchical visual features," Nature Communications, Nature, vol. 8(1), pages 1-15, August.
    6. N. Apurva Ratan Murty & Pouya Bashivan & Alex Abate & James J. DiCarlo & Nancy Kanwisher, 2021. "Computational models of category-selective brain regions enable high-throughput tests of selectivity," Nature Communications, Nature, vol. 12(1), pages 1-14, December.
    7. Matthew F. Glasser & Timothy S. Coalson & Emma C. Robinson & Carl D. Hacker & John Harwell & Essa Yacoub & Kamil Ugurbil & Jesper Andersson & Christian F. Beckmann & Mark Jenkinson & Stephen M. Smith , 2016. "A multi-modal parcellation of human cerebral cortex," Nature, Nature, vol. 536(7615), pages 171-178, August.
    8. Hamed Nili & Cai Wingfield & Alexander Walther & Li Su & William Marslen-Wilson & Nikolaus Kriegeskorte, 2014. "A Toolbox for Representational Similarity Analysis," PLOS Computational Biology, Public Library of Science, vol. 10(4), pages 1-11, April.
    9. Eugenio Piasini & Liviu Soltuzu & Paolo Muratore & Riccardo Caramellino & Kasper Vinken & Hans Op de Beeck & Vijay Balasubramanian & Davide Zoccolan, 2021. "Temporal stability of stimulus representation increases along rodent visual cortical hierarchies," Nature Communications, Nature, vol. 12(1), pages 1-19, December.
    10. Arno Klein & Satrajit S Ghosh & Forrest S Bao & Joachim Giard & Yrjö Häme & Eliezer Stavsky & Noah Lee & Brian Rossa & Martin Reuter & Elias Chaibub Neto & Anisha Keshavan, 2017. "Mindboggling morphometry of human brains," PLOS Computational Biology, Public Library of Science, vol. 13(2), pages 1-40, February.
    11. Russell Epstein & Nancy Kanwisher, 1998. "A cortical representation of the local visual environment," Nature, Nature, vol. 392(6676), pages 598-601, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zachariah M. Reagh & Charan Ranganath, 2023. "Flexible reuse of cortico-hippocampal representations during encoding and recall of naturalistic events," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    2. Valentina Pacella & Victor Nozais & Lia Talozzi & Majd Abdallah & Demian Wassermann & Stephanie J. Forkel & Michel Thiebaut de Schotten, 2024. "The morphospace of the brain-cognition organisation," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    3. Guohua Shen & Tomoyasu Horikawa & Kei Majima & Yukiyasu Kamitani, 2019. "Deep image reconstruction from human brain activity," PLOS Computational Biology, Public Library of Science, vol. 15(1), pages 1-23, January.
    4. Mengna Yao & Bincheng Wen & Mingpo Yang & Jiebin Guo & Haozhou Jiang & Chao Feng & Yilei Cao & Huiguang He & Le Chang, 2023. "High-dimensional topographic organization of visual features in the primate temporal lobe," Nature Communications, Nature, vol. 14(1), pages 1-23, December.
    5. Michael F Bonner & Russell A Epstein, 2018. "Computational mechanisms underlying cortical responses to the affordance properties of visual scenes," PLOS Computational Biology, Public Library of Science, vol. 14(4), pages 1-31, April.
    6. Haider Al-Tahan & Yalda Mohsenzadeh, 2021. "Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder," PLOS Computational Biology, Public Library of Science, vol. 17(3), pages 1-19, March.
    7. Annika Garlichs & Helen Blank, 2024. "Prediction error processing and sharpening of expected information across the face-processing hierarchy," Nature Communications, Nature, vol. 15(1), pages 1-18, December.
    8. Kristjan Kalm & Dennis Norris, 2021. "Sequence learning recodes cortical representations instead of strengthening initial ones," PLOS Computational Biology, Public Library of Science, vol. 17(5), pages 1-34, May.
    9. Alexander J Barnett & Walter Reilly & Halle R Dimsdale-Zucker & Eda Mizrak & Zachariah Reagh & Charan Ranganath, 2021. "Intrinsic connectivity reveals functionally distinct cortico-hippocampal networks in the human brain," PLOS Biology, Public Library of Science, vol. 19(6), pages 1-34, June.
    10. Valentina Krenz & Arjen Alink & Tobias Sommer & Benno Roozendaal & Lars Schwabe, 2023. "Time-dependent memory transformation in hippocampus and neocortex is semantic in nature," Nature Communications, Nature, vol. 14(1), pages 1-17, December.
    11. Marisa Nordt & Jesse Gomez & Vaidehi S. Natu & Alex A. Rezai & Dawn Finzi & Holly Kular & Kalanit Grill-Spector, 2023. "Longitudinal development of category representations in ventral temporal cortex predicts word and face recognition," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    12. Vasiliki Bougou & Michaël Vanhoyland & Alexander Bertrand & Wim Paesschen & Hans Op De Beeck & Peter Janssen & Tom Theys, 2024. "Neuronal tuning and population representations of shape and category in human visual cortex," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    13. Ying Wang & Xue Zhang & Chunhui Wang & Weifen Huang & Qian Xu & Dong Liu & Wen Zhou & Shanguang Chen & Yi Jiang, 2022. "Modulation of biological motion perception in humans by gravity," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    14. Joel Z Leibo & Qianli Liao & Fabio Anselmi & Tomaso Poggio, 2015. "The Invariance Hypothesis Implies Domain-Specific Regions in Visual Cortex," PLOS Computational Biology, Public Library of Science, vol. 11(10), pages 1-29, October.
    15. Isabella C. Wagner & Luise P. Graichen & Boryana Todorova & Andre Lüttig & David B. Omer & Matthias Stangl & Claus Lamm, 2023. "Entorhinal grid-like codes and time-locked network dynamics track others navigating through space," Nature Communications, Nature, vol. 14(1), pages 1-18, December.
    16. Satoko Amemori & Ann M. Graybiel & Ken-ichi Amemori, 2024. "Cingulate microstimulation induces negative decision-making via reduced top-down influence on primate fronto-cingulo-striatal network," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    17. Julia Berezutskaya & Zachary V Freudenburg & Umut Güçlü & Marcel A J van Gerven & Nick F Ramsey, 2020. "Brain-optimized extraction of complex sound features that drive continuous auditory perception," PLOS Computational Biology, Public Library of Science, vol. 16(7), pages 1-34, July.
    18. Leon D. Lotter & Amin Saberi & Justine Y. Hansen & Bratislav Misic & Casey Paquola & Gareth J. Barker & Arun L. W. Bokde & Sylvane Desrivières & Herta Flor & Antoine Grigis & Hugh Garavan & Penny Gowl, 2024. "Regional patterns of human cortex development correlate with underlying neurobiology," Nature Communications, Nature, vol. 15(1), pages 1-21, December.
    19. Haewon Nam & Chongwon Pae & Jinseok Eo & Maeng-Keun Oh & Hae-Jeong Park, 2021. "Inter-species cortical registration between macaques and humans using a functional network property under a spherical demons framework," PLOS ONE, Public Library of Science, vol. 16(10), pages 1-22, October.
    20. Jessica Dafflon & Pedro F. Da Costa & František Váša & Ricardo Pio Monti & Danilo Bzdok & Peter J. Hellyer & Federico Turkheimer & Jonathan Smallwood & Emily Jones & Robert Leech, 2022. "A guided multiverse study of neuroimaging analyses," Nature Communications, Nature, vol. 13(1), pages 1-13, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-50310-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.