IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-48114-6.html
   My bibliography  Save this article

Computational reconstruction of mental representations using human behavior

Author

Listed:
  • Laurent Caplette

    (Yale University)

  • Nicholas B. Turk-Browne

    (Yale University
    Yale University)

Abstract

Revealing how the mind represents information is a longstanding goal of cognitive science. However, there is currently no framework for reconstructing the broad range of mental representations that humans possess. Here, we ask participants to indicate what they perceive in images made of random visual features in a deep neural network. We then infer associations between the semantic features of their responses and the visual features of the images. This allows us to reconstruct the mental representations of multiple visual concepts, both those supplied by participants and other concepts extrapolated from the same semantic space. We validate these reconstructions in separate participants and further generalize our approach to predict behavior for new stimuli and in a new task. Finally, we reconstruct the mental representations of individual observers and of a neural network. This framework enables a large-scale investigation of conceptual representations.

Suggested Citation

  • Laurent Caplette & Nicholas B. Turk-Browne, 2024. "Computational reconstruction of mental representations using human behavior," Nature Communications, Nature, vol. 15(1), pages 1-19, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-48114-6
    DOI: 10.1038/s41467-024-48114-6
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-48114-6
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-48114-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jiayu Zhan & Oliver G. B. Garrod & Nicola van Rijsbergen & Philippe G. Schyns, 2019. "Modelling face memory reveals task-generalizable representations," Nature Human Behaviour, Nature, vol. 3(8), pages 817-826, August.
    2. Martin N. Hebart & Charles Y. Zheng & Francisco Pereira & Chris I. Baker, 2020. "Revealing the multidimensional mental representations of natural objects underlying human similarity judgements," Nature Human Behaviour, Nature, vol. 4(11), pages 1173-1185, November.
    3. Satoshi Nishida & Antione Blanc & Naoya Maeda & Masataka Kado & Shinji Nishimoto, 2021. "Behavioral correlates of cortical semantic representations modeled by word vectors," PLOS Computational Biology, Public Library of Science, vol. 17(6), pages 1-35, June.
    4. Alexander G. Huth & Wendy A. de Heer & Thomas L. Griffiths & Frédéric E. Theunissen & Jack L. Gallant, 2016. "Natural speech reveals the semantic maps that tile human cerebral cortex," Nature, Nature, vol. 532(7600), pages 453-458, April.
    5. Pinglei Bao & Liang She & Mason McGill & Doris Y. Tsao, 2020. "A map of object space in primate inferotemporal cortex," Nature, Nature, vol. 583(7814), pages 103-108, July.
    6. Charles R. Harris & K. Jarrod Millman & Stéfan J. Walt & Ralf Gommers & Pauli Virtanen & David Cournapeau & Eric Wieser & Julian Taylor & Sebastian Berg & Nathaniel J. Smith & Robert Kern & Matti Picu, 2020. "Array programming with NumPy," Nature, Nature, vol. 585(7825), pages 357-362, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tan Wang & L. Jeff Hong, 2023. "Large-Scale Inventory Optimization: A Recurrent Neural Networks–Inspired Simulation Approach," INFORMS Journal on Computing, INFORMS, vol. 35(1), pages 196-215, January.
    2. Léon Faure & Bastien Mollet & Wolfram Liebermeister & Jean-Loup Faulon, 2023. "A neural-mechanistic hybrid approach improving the predictive power of genome-scale metabolic models," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    3. Claudia Quinteros-Cartaya & Guillermo Solorio-Magaña & Francisco Javier Núñez-Cornú & Felipe de Jesús Escalona-Alcázar & Diana Núñez, 2023. "Microearthquakes in the Guadalajara Metropolitan Zone, Mexico: evidence from buried active faults in Tesistán Valley, Zapopan," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 116(3), pages 2797-2818, April.
    4. Furqan Dar & Samuel R. Cohen & Diana M. Mitrea & Aaron H. Phillips & Gergely Nagy & Wellington C. Leite & Christopher B. Stanley & Jeong-Mo Choi & Richard W. Kriwacki & Rohit V. Pappu, 2024. "Biomolecular condensates form spatially inhomogeneous network fluids," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    5. Vasiliki Bougou & Michaël Vanhoyland & Alexander Bertrand & Wim Paesschen & Hans Op De Beeck & Peter Janssen & Tom Theys, 2024. "Neuronal tuning and population representations of shape and category in human visual cortex," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    6. Nina Tiel & Fabian Fopp & Philipp Brun & Johan Hoogen & Dirk Nikolaus Karger & Cecilia M. Casadei & Lisha Lyu & Devis Tuia & Niklaus E. Zimmermann & Thomas W. Crowther & Loïc Pellissier, 2024. "Regional uniqueness of tree species composition and response to forest loss and climate change," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    7. López Pérez, Mario & Mansilla Corona, Ricardo, 2022. "Ordinal synchronization and typical states in high-frequency digital markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 598(C).
    8. Jessica M. Vanslambrouck & Sean B. Wilson & Ker Sin Tan & Ella Groenewegen & Rajeev Rudraraju & Jessica Neil & Kynan T. Lawlor & Sophia Mah & Michelle Scurr & Sara E. Howden & Kanta Subbarao & Melissa, 2022. "Enhanced metanephric specification to functional proximal tubule enables toxicity screening and infectious disease modelling in kidney organoids," Nature Communications, Nature, vol. 13(1), pages 1-23, December.
    9. Kiran Krishnamachari & Dylan Lu & Alexander Swift-Scott & Anuar Yeraliyev & Kayla Lee & Weitai Huang & Sim Ngak Leng & Anders Jacobsen Skanderup, 2022. "Accurate somatic variant detection using weakly supervised deep learning," Nature Communications, Nature, vol. 13(1), pages 1-8, December.
    10. Larissa Samaan & Leonie Klock & Sandra Weber & Mirjam Reidick & Leonie Ascone & Simone Kühn, 2024. "Low-Level Visual Features of Window Views Contribute to Perceived Naturalness and Mental Health Outcomes," IJERPH, MDPI, vol. 21(5), pages 1-35, May.
    11. Dennis Bontempi & Leonard Nuernberg & Suraj Pai & Deepa Krishnaswamy & Vamsi Thiriveedhi & Ahmed Hosny & Raymond H. Mak & Keyvan Farahani & Ron Kikinis & Andrey Fedorov & Hugo J. W. L. Aerts, 2024. "End-to-end reproducible AI pipelines in radiology using the cloud," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    12. Lauren L. Porter & Allen K. Kim & Swechha Rimal & Loren L. Looger & Ananya Majumdar & Brett D. Mensh & Mary R. Starich & Marie-Paule Strub, 2022. "Many dissimilar NusG protein domains switch between α-helix and β-sheet folds," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    13. Oren Amsalem & Hidehiko Inagaki & Jianing Yu & Karel Svoboda & Ran Darshan, 2024. "Sub-threshold neuronal activity and the dynamical regime of cerebral cortex," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    14. Matthew Rosenblatt & Link Tejavibulya & Rongtao Jiang & Stephanie Noble & Dustin Scheinost, 2024. "Data leakage inflates prediction performance in connectome-based machine learning models," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    15. Jackie Grant & Mark Hindmarsh & Sergey E. Koposov, 2022. "The distribution of loss to future USS pensions due to the UUK cuts of April 2022," Papers 2206.06201, arXiv.org.
    16. Sayedali Shetab Boushehri & Katharina Essig & Nikolaos-Kosmas Chlis & Sylvia Herter & Marina Bacac & Fabian J. Theis & Elke Glasmacher & Carsten Marr & Fabian Schmich, 2023. "Explainable machine learning for profiling the immunological synapse and functional characterization of therapeutic antibodies," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    17. Henry Webel & Lili Niu & Annelaura Bach Nielsen & Marie Locard-Paulet & Matthias Mann & Lars Juhl Jensen & Simon Rasmussen, 2024. "Imputation of label-free quantitative mass spectrometry-based proteomics data using self-supervised deep learning," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    18. Chandan Singh & Armin Askari & Rich Caruana & Jianfeng Gao, 2023. "Augmenting interpretable models with large language models during training," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    19. Shukla, Mohak & Thakur, Ajay D., 2022. "An Enquiry on similarities between Renormalization Group and Auto-Encoders using Transfer Learning," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 608(P1).
    20. Borofsky, Talia & Feldman, Marcus W. & Ram, Yoav, 2024. "Cultural transmission, competition for prey, and the evolution of cooperative hunting," Theoretical Population Biology, Elsevier, vol. 156(C), pages 12-21.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-48114-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.