IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-48114-6.html
   My bibliography  Save this article

Computational reconstruction of mental representations using human behavior

Author

Listed:
  • Laurent Caplette

    (Yale University)

  • Nicholas B. Turk-Browne

    (Yale University
    Yale University)

Abstract

Revealing how the mind represents information is a longstanding goal of cognitive science. However, there is currently no framework for reconstructing the broad range of mental representations that humans possess. Here, we ask participants to indicate what they perceive in images made of random visual features in a deep neural network. We then infer associations between the semantic features of their responses and the visual features of the images. This allows us to reconstruct the mental representations of multiple visual concepts, both those supplied by participants and other concepts extrapolated from the same semantic space. We validate these reconstructions in separate participants and further generalize our approach to predict behavior for new stimuli and in a new task. Finally, we reconstruct the mental representations of individual observers and of a neural network. This framework enables a large-scale investigation of conceptual representations.

Suggested Citation

  • Laurent Caplette & Nicholas B. Turk-Browne, 2024. "Computational reconstruction of mental representations using human behavior," Nature Communications, Nature, vol. 15(1), pages 1-19, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-48114-6
    DOI: 10.1038/s41467-024-48114-6
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-48114-6
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-48114-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Alexander G. Huth & Wendy A. de Heer & Thomas L. Griffiths & Frédéric E. Theunissen & Jack L. Gallant, 2016. "Natural speech reveals the semantic maps that tile human cerebral cortex," Nature, Nature, vol. 532(7600), pages 453-458, April.
    2. Jiayu Zhan & Oliver G. B. Garrod & Nicola van Rijsbergen & Philippe G. Schyns, 2019. "Modelling face memory reveals task-generalizable representations," Nature Human Behaviour, Nature, vol. 3(8), pages 817-826, August.
    3. Martin N. Hebart & Charles Y. Zheng & Francisco Pereira & Chris I. Baker, 2020. "Revealing the multidimensional mental representations of natural objects underlying human similarity judgements," Nature Human Behaviour, Nature, vol. 4(11), pages 1173-1185, November.
    4. Pinglei Bao & Liang She & Mason McGill & Doris Y. Tsao, 2020. "A map of object space in primate inferotemporal cortex," Nature, Nature, vol. 583(7814), pages 103-108, July.
    5. Satoshi Nishida & Antione Blanc & Naoya Maeda & Masataka Kado & Shinji Nishimoto, 2021. "Behavioral correlates of cortical semantic representations modeled by word vectors," PLOS Computational Biology, Public Library of Science, vol. 17(6), pages 1-35, June.
    6. Charles R. Harris & K. Jarrod Millman & Stéfan J. Walt & Ralf Gommers & Pauli Virtanen & David Cournapeau & Eric Wieser & Julian Taylor & Sebastian Berg & Nathaniel J. Smith & Robert Kern & Matti Picu, 2020. "Array programming with NumPy," Nature, Nature, vol. 585(7825), pages 357-362, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Furqan Dar & Samuel R. Cohen & Diana M. Mitrea & Aaron H. Phillips & Gergely Nagy & Wellington C. Leite & Christopher B. Stanley & Jeong-Mo Choi & Richard W. Kriwacki & Rohit V. Pappu, 2024. "Biomolecular condensates form spatially inhomogeneous network fluids," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    2. López Pérez, Mario & Mansilla Corona, Ricardo, 2022. "Ordinal synchronization and typical states in high-frequency digital markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 598(C).
    3. Jessica M. Vanslambrouck & Sean B. Wilson & Ker Sin Tan & Ella Groenewegen & Rajeev Rudraraju & Jessica Neil & Kynan T. Lawlor & Sophia Mah & Michelle Scurr & Sara E. Howden & Kanta Subbarao & Melissa, 2022. "Enhanced metanephric specification to functional proximal tubule enables toxicity screening and infectious disease modelling in kidney organoids," Nature Communications, Nature, vol. 13(1), pages 1-23, December.
    4. Lauren L. Porter & Allen K. Kim & Swechha Rimal & Loren L. Looger & Ananya Majumdar & Brett D. Mensh & Mary R. Starich & Marie-Paule Strub, 2022. "Many dissimilar NusG protein domains switch between α-helix and β-sheet folds," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    5. Matthew Rosenblatt & Link Tejavibulya & Rongtao Jiang & Stephanie Noble & Dustin Scheinost, 2024. "Data leakage inflates prediction performance in connectome-based machine learning models," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    6. Sayedali Shetab Boushehri & Katharina Essig & Nikolaos-Kosmas Chlis & Sylvia Herter & Marina Bacac & Fabian J. Theis & Elke Glasmacher & Carsten Marr & Fabian Schmich, 2023. "Explainable machine learning for profiling the immunological synapse and functional characterization of therapeutic antibodies," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    7. Chandan Singh & Armin Askari & Rich Caruana & Jianfeng Gao, 2023. "Augmenting interpretable models with large language models during training," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    8. Khaled Akkad & David He, 2023. "A dynamic mode decomposition based deep learning technique for prognostics," Journal of Intelligent Manufacturing, Springer, vol. 34(5), pages 2207-2224, June.
    9. Romain Fournier & Zoi Tsangalidou & David Reich & Pier Francesco Palamara, 2023. "Haplotype-based inference of recent effective population size in modern and ancient DNA samples," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    10. Laura Portell & Sergi Morera & Helena Ramalhinho, 2022. "Door-to-Door Transportation Services for Reduced Mobility Population: A Descriptive Analytics of the City of Barcelona," IJERPH, MDPI, vol. 19(8), pages 1-20, April.
    11. Caroline Haimerl & Douglas A. Ruff & Marlene R. Cohen & Cristina Savin & Eero P. Simoncelli, 2023. "Targeted V1 comodulation supports task-adaptive sensory decisions," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    12. Matthias Wagener & Andriette Bekker & Mohammad Arashi, 2021. "Mastering the Body and Tail Shape of a Distribution," Mathematics, MDPI, vol. 9(21), pages 1-22, October.
    13. Gallo Cassarino, Tiziano & Barrett, Mark, 2022. "Meeting UK heat demands in zero emission renewable energy systems using storage and interconnectors," Applied Energy, Elsevier, vol. 306(PB).
    14. Maren Schnieder, 2023. "Ebike Sharing vs. Bike Sharing: Demand Prediction Using Deep Neural Networks and Random Forests," Sustainability, MDPI, vol. 15(18), pages 1-15, September.
    15. Gabriele Orlando & Daniele Raimondi & Ramon Duran-Romaña & Yves Moreau & Joost Schymkowitz & Frederic Rousseau, 2022. "PyUUL provides an interface between biological structures and deep learning algorithms," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    16. Hazal Colak Oz & Çiçek Güven & Gonzalo Nápoles, 2023. "School dropout prediction and feature importance exploration in Malawi using household panel data: machine learning approach," Journal of Computational Social Science, Springer, vol. 6(1), pages 245-287, April.
    17. Vincent Wagner & Nicole Erika Radde, 2021. "SiCaSMA: An Alternative Stochastic Description via Concatenation of Markov Processes for a Class of Catalytic Systems," Mathematics, MDPI, vol. 9(10), pages 1-13, May.
    18. L. Mathur & B. Szalai & N. H. Du & R. Utharala & M. Ballinger & J. J. M. Landry & M. Ryckelynck & V. Benes & J. Saez-Rodriguez & C. A. Merten, 2022. "Combi-seq for multiplexed transcriptome-based profiling of drug combinations using deterministic barcoding in single-cell droplets," Nature Communications, Nature, vol. 13(1), pages 1-15, December.
    19. Samuel G. Fadel & Sebastian Mair & Ricardo da Silva Torres & Ulf Brefeld, 2023. "Contextual movement models based on normalizing flows," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 107(1), pages 51-72, March.
    20. Ivan Brandić & Lato Pezo & Nikola Bilandžija & Anamarija Peter & Jona Šurić & Neven Voća, 2023. "Comparison of Different Machine Learning Models for Modelling the Higher Heating Value of Biomass," Mathematics, MDPI, vol. 11(9), pages 1-14, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-48114-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.