IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-50310-3.html
   My bibliography  Save this article

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

Author

Listed:
  • Benjamin Lahner

    (MIT)

  • Kshitij Dwivedi

    (Freie Universität Berlin
    Goethe University Frankfurt)

  • Polina Iamshchinina

    (Freie Universität Berlin)

  • Monika Graumann

    (Freie Universität Berlin)

  • Alex Lascelles

    (MIT)

  • Gemma Roig

    (Goethe University Frankfurt
    The Hessian Center for AI (hessian.AI))

  • Alessandro Thomas Gifford

    (Freie Universität Berlin)

  • Bowen Pan

    (MIT)

  • SouYoung Jin

    (MIT)

  • N. Apurva Ratan Murty

    (MIT
    Georgia Institute of Technology)

  • Kendrick Kay

    (University of Minnesota)

  • Aude Oliva

    (MIT)

  • Radoslaw Cichy

    (Freie Universität Berlin)

Abstract

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos’ extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.

Suggested Citation

  • Benjamin Lahner & Kshitij Dwivedi & Polina Iamshchinina & Monika Graumann & Alex Lascelles & Gemma Roig & Alessandro Thomas Gifford & Bowen Pan & SouYoung Jin & N. Apurva Ratan Murty & Kendrick Kay & , 2024. "Modeling short visual events through the BOLD moments video fMRI dataset and metadata," Nature Communications, Nature, vol. 15(1), pages 1-26, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-50310-3
    DOI: 10.1038/s41467-024-50310-3
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-50310-3
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-50310-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Bart Krekelberg & Sabine Dannenberg & Klaus-Peter Hoffmann & Frank Bremmer & John Ross, 2003. "Neural correlates of implied motion," Nature, Nature, vol. 424(6949), pages 674-677, August.
    2. Agustin Lage-Castellanos & Giancarlo Valente & Elia Formisano & Federico De Martino, 2019. "Methods for computing the maximum performance of computational models of fMRI responses," PLOS Computational Biology, Public Library of Science, vol. 15(3), pages 1-25, March.
    3. Tomoyasu Horikawa & Yukiyasu Kamitani, 2017. "Generic decoding of seen and imagined objects using hierarchical visual features," Nature Communications, Nature, vol. 8(1), pages 1-15, August.
    4. N. Apurva Ratan Murty & Pouya Bashivan & Alex Abate & James J. DiCarlo & Nancy Kanwisher, 2021. "Computational models of category-selective brain regions enable high-throughput tests of selectivity," Nature Communications, Nature, vol. 12(1), pages 1-14, December.
    5. Matthew F. Glasser & Timothy S. Coalson & Emma C. Robinson & Carl D. Hacker & John Harwell & Essa Yacoub & Kamil Ugurbil & Jesper Andersson & Christian F. Beckmann & Mark Jenkinson & Stephen M. Smith , 2016. "A multi-modal parcellation of human cerebral cortex," Nature, Nature, vol. 536(7615), pages 171-178, August.
    6. Hamed Nili & Cai Wingfield & Alexander Walther & Li Su & William Marslen-Wilson & Nikolaus Kriegeskorte, 2014. "A Toolbox for Representational Similarity Analysis," PLOS Computational Biology, Public Library of Science, vol. 10(4), pages 1-11, April.
    7. Arno Klein & Satrajit S Ghosh & Forrest S Bao & Joachim Giard & Yrjö Häme & Eliezer Stavsky & Noah Lee & Brian Rossa & Martin Reuter & Elias Chaibub Neto & Anisha Keshavan, 2017. "Mindboggling morphometry of human brains," PLOS Computational Biology, Public Library of Science, vol. 13(2), pages 1-40, February.
    8. Scott L Fairhall & Angela Albi & David Melcher, 2014. "Temporal Integration Windows for Naturalistic Visual Sequences," PLOS ONE, Public Library of Science, vol. 9(7), pages 1-6, July.
    9. Jonathan D. Cohen & William M. Perlstein & Todd S. Braver & Leigh E. Nystrom & Douglas C. Noll & John Jonides & Edward E. Smith, 1997. "Temporal dynamics of brain activation during a working memory task," Nature, Nature, vol. 386(6625), pages 604-608, April.
    10. Eugenio Piasini & Liviu Soltuzu & Paolo Muratore & Riccardo Caramellino & Kasper Vinken & Hans Op de Beeck & Vijay Balasubramanian & Davide Zoccolan, 2021. "Temporal stability of stimulus representation increases along rodent visual cortical hierarchies," Nature Communications, Nature, vol. 12(1), pages 1-19, December.
    11. Russell Epstein & Nancy Kanwisher, 1998. "A cortical representation of the local visual environment," Nature, Nature, vol. 392(6676), pages 598-601, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zachariah M. Reagh & Charan Ranganath, 2023. "Flexible reuse of cortico-hippocampal representations during encoding and recall of naturalistic events," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    2. Guohua Shen & Tomoyasu Horikawa & Kei Majima & Yukiyasu Kamitani, 2019. "Deep image reconstruction from human brain activity," PLOS Computational Biology, Public Library of Science, vol. 15(1), pages 1-23, January.
    3. Michael F Bonner & Russell A Epstein, 2018. "Computational mechanisms underlying cortical responses to the affordance properties of visual scenes," PLOS Computational Biology, Public Library of Science, vol. 14(4), pages 1-31, April.
    4. Mengna Yao & Bincheng Wen & Mingpo Yang & Jiebin Guo & Haozhou Jiang & Chao Feng & Yilei Cao & Huiguang He & Le Chang, 2023. "High-dimensional topographic organization of visual features in the primate temporal lobe," Nature Communications, Nature, vol. 14(1), pages 1-23, December.
    5. Haider Al-Tahan & Yalda Mohsenzadeh, 2021. "Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder," PLOS Computational Biology, Public Library of Science, vol. 17(3), pages 1-19, March.
    6. Annika Garlichs & Helen Blank, 2024. "Prediction error processing and sharpening of expected information across the face-processing hierarchy," Nature Communications, Nature, vol. 15(1), pages 1-18, December.
    7. Kristjan Kalm & Dennis Norris, 2021. "Sequence learning recodes cortical representations instead of strengthening initial ones," PLOS Computational Biology, Public Library of Science, vol. 17(5), pages 1-34, May.
    8. Alexander J Barnett & Walter Reilly & Halle R Dimsdale-Zucker & Eda Mizrak & Zachariah Reagh & Charan Ranganath, 2021. "Intrinsic connectivity reveals functionally distinct cortico-hippocampal networks in the human brain," PLOS Biology, Public Library of Science, vol. 19(6), pages 1-34, June.
    9. Valentina Krenz & Arjen Alink & Tobias Sommer & Benno Roozendaal & Lars Schwabe, 2023. "Time-dependent memory transformation in hippocampus and neocortex is semantic in nature," Nature Communications, Nature, vol. 14(1), pages 1-17, December.
    10. Marisa Nordt & Jesse Gomez & Vaidehi S. Natu & Alex A. Rezai & Dawn Finzi & Holly Kular & Kalanit Grill-Spector, 2023. "Longitudinal development of category representations in ventral temporal cortex predicts word and face recognition," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    11. Vasiliki Bougou & Michaël Vanhoyland & Alexander Bertrand & Wim Paesschen & Hans Op De Beeck & Peter Janssen & Tom Theys, 2024. "Neuronal tuning and population representations of shape and category in human visual cortex," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    12. Isabella C. Wagner & Luise P. Graichen & Boryana Todorova & Andre Lüttig & David B. Omer & Matthias Stangl & Claus Lamm, 2023. "Entorhinal grid-like codes and time-locked network dynamics track others navigating through space," Nature Communications, Nature, vol. 14(1), pages 1-18, December.
    13. Leon D. Lotter & Amin Saberi & Justine Y. Hansen & Bratislav Misic & Casey Paquola & Gareth J. Barker & Arun L. W. Bokde & Sylvane Desrivières & Herta Flor & Antoine Grigis & Hugh Garavan & Penny Gowl, 2024. "Regional patterns of human cortex development correlate with underlying neurobiology," Nature Communications, Nature, vol. 15(1), pages 1-21, December.
    14. Haewon Nam & Chongwon Pae & Jinseok Eo & Maeng-Keun Oh & Hae-Jeong Park, 2021. "Inter-species cortical registration between macaques and humans using a functional network property under a spherical demons framework," PLOS ONE, Public Library of Science, vol. 16(10), pages 1-22, October.
    15. Arno Klein & Satrajit S Ghosh & Forrest S Bao & Joachim Giard & Yrjö Häme & Eliezer Stavsky & Noah Lee & Brian Rossa & Martin Reuter & Elias Chaibub Neto & Anisha Keshavan, 2017. "Mindboggling morphometry of human brains," PLOS Computational Biology, Public Library of Science, vol. 13(2), pages 1-40, February.
    16. Ann Hillier & Ryan P Kelly & Terrie Klinger, 2016. "Narrative Style Influences Citation Frequency in Climate Change Science," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-12, December.
    17. Hamed Nili & Alexander Walther & Arjen Alink & Nikolaus Kriegeskorte, 2020. "Inferring exemplar discriminability in brain representations," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-28, June.
    18. Manish Saggar & James M. Shine & Raphaël Liégeois & Nico U. F. Dosenbach & Damien Fair, 2022. "Precision dynamical mapping using topological data analysis reveals a hub-like transition state at rest," Nature Communications, Nature, vol. 13(1), pages 1-19, December.
    19. Devetag, Giovanna & Warglien, Massimo, 2003. "Games and phone numbers: Do short-term memory bounds affect strategic behavior?," Journal of Economic Psychology, Elsevier, vol. 24(2), pages 189-202, April.
    20. Deck, Cary & Jahedi, Salar & Sheremeta, Roman, 2021. "On the consistency of cognitive load," European Economic Review, Elsevier, vol. 134(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-50310-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.