IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1006397.html
   My bibliography  Save this article

Methods for computing the maximum performance of computational models of fMRI responses

Author

Listed:
  • Agustin Lage-Castellanos
  • Giancarlo Valente
  • Elia Formisano
  • Federico De Martino

Abstract

Computational neuroimaging methods aim to predict brain responses (measured e.g. with functional magnetic resonance imaging [fMRI]) on the basis of stimulus features obtained through computational models. The accuracy of such prediction is used as an indicator of how well the model describes the computations underlying the brain function that is being considered. However, the prediction accuracy is bounded by the proportion of the variance of the brain response which is related to the measurement noise and not to the stimuli (or cognitive functions). This bound to the performance of a computational model has been referred to as the noise ceiling. In previous fMRI applications two methods have been proposed to estimate the noise ceiling based on either a split-half procedure or Monte Carlo simulations. These methods make different assumptions over the nature of the effects underlying the data, and, importantly, their relation has not been clarified yet. Here, we derive an analytical form for the noise ceiling that does not require computationally expensive simulations or a splitting procedure that reduce the amount of data. The validity of this analytical definition is proved in simulations, we show that the analytical solution results in the same estimate of the noise ceiling as the Monte Carlo method. Considering different simulated noise structure, we evaluate different estimators of the variance of the responses and their impact on the estimation of the noise ceiling. We furthermore evaluate the interplay between regularization (often used to estimate model fits to the data when the number of computational features in the model is large) and model complexity on the performance with respect to the noise ceiling. Our results indicate that when considering the variance of the responses across runs, computing the noise ceiling analytically results in similar estimates as the split half estimator and approaches the true noise ceiling under a variety of simulated noise scenarios. Finally, the methods are tested on real fMRI data acquired at 7 Tesla.Author summary: Encoding computational models in brain responses measured with fMRI allows testing the algorithmic representations carried out by the neural population within voxels. The accuracy of a model in predicting new responses is used as a measure of the brain validity of the computational model being tested, but the result of this analysis is determined not only by how precisely the model describes the responses but also by the quality of the data. In this article, we evaluate existing approaches to estimate the best possible accuracy that any computational model can achieve conditioned to the amount of measurement noise that is present in the experimental data (i.e. the noise ceiling). Additionally we introduce a close form estimation of the noise ceiling that does not require computationally or data expensive procedures. All the methods are compared using simulated and real fMRI data. We draw conclusions over the impact of regularization procedures and make practical recommendations on how to report the results of computational models in neuroimaging.

Suggested Citation

  • Agustin Lage-Castellanos & Giancarlo Valente & Elia Formisano & Federico De Martino, 2019. "Methods for computing the maximum performance of computational models of fMRI responses," PLOS Computational Biology, Public Library of Science, vol. 15(3), pages 1-25, March.
  • Handle: RePEc:plo:pcbi00:1006397
    DOI: 10.1371/journal.pcbi.1006397
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006397
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1006397&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1006397?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jonathan Rubin & Nachum Ulanovsky & Israel Nelken & Naftali Tishby, 2016. "The Representation of Prediction Error in Auditory Cortex," PLOS Computational Biology, Public Library of Science, vol. 12(8), pages 1-28, August.
    2. Hamed Nili & Cai Wingfield & Alexander Walther & Li Su & William Marslen-Wilson & Nikolaus Kriegeskorte, 2014. "A Toolbox for Representational Similarity Analysis," PLOS Computational Biology, Public Library of Science, vol. 10(4), pages 1-11, April.
    3. Kendrick N. Kay & Thomas Naselaris & Ryan J. Prenger & Jack L. Gallant, 2008. "Identifying natural images from human brain activity," Nature, Nature, vol. 452(7185), pages 352-355, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benjamin Lahner & Kshitij Dwivedi & Polina Iamshchinina & Monika Graumann & Alex Lascelles & Gemma Roig & Alessandro Thomas Gifford & Bowen Pan & SouYoung Jin & N. Apurva Ratan Murty & Kendrick Kay & , 2024. "Modeling short visual events through the BOLD moments video fMRI dataset and metadata," Nature Communications, Nature, vol. 15(1), pages 1-26, December.
    2. Andrew J. Stier & Sina Sajjadi & Fariba Karimi & Luís M. A. Bettencourt & Marc G. Berman, 2024. "Implicit racial biases are lower in more populous more diverse and less segregated US cities," Nature Communications, Nature, vol. 15(1), pages 1-10, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hamed Nili & Alexander Walther & Arjen Alink & Nikolaus Kriegeskorte, 2020. "Inferring exemplar discriminability in brain representations," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-28, June.
    2. Ming Bo Cai & Nicolas W Schuck & Jonathan W Pillow & Yael Niv, 2019. "Representational structure or task structure? Bias in neural representational similarity analysis and a Bayesian method for reducing bias," PLOS Computational Biology, Public Library of Science, vol. 15(5), pages 1-30, May.
    3. Jörn Diedrichsen & Nikolaus Kriegeskorte, 2017. "Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis," PLOS Computational Biology, Public Library of Science, vol. 13(4), pages 1-33, April.
    4. Teruaki Kido & Yuko Yotsumoto & Masamichi J. Hayashi, 2025. "Hierarchical representations of relative numerical magnitudes in the human frontoparietal cortex," Nature Communications, Nature, vol. 16(1), pages 1-15, December.
    5. Valentina Krenz & Arjen Alink & Tobias Sommer & Benno Roozendaal & Lars Schwabe, 2023. "Time-dependent memory transformation in hippocampus and neocortex is semantic in nature," Nature Communications, Nature, vol. 14(1), pages 1-17, December.
    6. Zvi N. Roth & Kendrick Kay & Elisha P. Merriam, 2022. "Natural scene sampling reveals reliable coarse-scale orientation tuning in human V1," Nature Communications, Nature, vol. 13(1), pages 1-13, December.
    7. Hamed Nili & Cai Wingfield & Alexander Walther & Li Su & William Marslen-Wilson & Nikolaus Kriegeskorte, 2014. "A Toolbox for Representational Similarity Analysis," PLOS Computational Biology, Public Library of Science, vol. 10(4), pages 1-11, April.
    8. Jacob M. Paul & Martijn Ackooij & Tuomas C. Cate & Ben M. Harvey, 2022. "Numerosity tuning in human association cortices and local image contrast representations in early visual cortex," Nature Communications, Nature, vol. 13(1), pages 1-15, December.
    9. Kay H Brodersen & Thomas M Schofield & Alexander P Leff & Cheng Soon Ong & Ekaterina I Lomakina & Joachim M Buhmann & Klaas E Stephan, 2011. "Generative Embedding for Model-Based Classification of fMRI Data," PLOS Computational Biology, Public Library of Science, vol. 7(6), pages 1-19, June.
    10. Sreejan Kumar & Theodore R. Sumers & Takateru Yamakoshi & Ariel Goldstein & Uri Hasson & Kenneth A. Norman & Thomas L. Griffiths & Robert D. Hawkins & Samuel A. Nastase, 2024. "Shared functional specialization in transformer-based language models and the human brain," Nature Communications, Nature, vol. 15(1), pages 1-19, December.
    11. Ghislain St-Yves & Emily J. Allen & Yihan Wu & Kendrick Kay & Thomas Naselaris, 2023. "Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    12. Raheel Zafar & Sarat C Dass & Aamir Saeed Malik, 2017. "Electroencephalogram-based decoding cognitive states using convolutional neural network and likelihood ratio based score fusion," PLOS ONE, Public Library of Science, vol. 12(5), pages 1-23, May.
    13. Cai Wingfield & Li Su & Xunying Liu & Chao Zhang & Phil Woodland & Andrew Thwaites & Elisabeth Fonteneau & William D Marslen-Wilson, 2017. "Relating dynamic brain states to dynamic machine states: Human and machine solutions to the speech recognition problem," PLOS Computational Biology, Public Library of Science, vol. 13(9), pages 1-25, September.
    14. Tarana Nigam & Caspar M. Schwiedrzik, 2024. "Predictions enable top-down pattern separation in the macaque face-processing hierarchy," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    15. Michael F Bonner & Russell A Epstein, 2018. "Computational mechanisms underlying cortical responses to the affordance properties of visual scenes," PLOS Computational Biology, Public Library of Science, vol. 14(4), pages 1-31, April.
    16. Sebastian P. H. Speer & Laetitia Mwilambwe-Tshilobo & Lily Tsoi & Shannon M. Burns & Emily B. Falk & Diana I. Tamir, 2024. "Hyperscanning shows friends explore and strangers converge in conversation," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    17. Hadar Levi-Aharoni & Oren Shriki & Naftali Tishby, 2020. "Surprise response as a probe for compressed memory states," PLOS Computational Biology, Public Library of Science, vol. 16(2), pages 1-21, February.
    18. Samy A. Abdel-Ghaffar & Alexander G. Huth & Mark D. Lescroart & Dustin Stansbury & Jack L. Gallant & Sonia J. Bishop, 2024. "Occipital-temporal cortical tuning to semantic and affective features of natural images predicts associated behavioral responses," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    19. Máté Aller & Agoston Mihalik & Uta Noppeney, 2022. "Audiovisual adaptation is expressed in spatial and decisional codes," Nature Communications, Nature, vol. 13(1), pages 1-17, December.
    20. Marcel Adam Just & Vladimir L Cherkassky & Augusto Buchweitz & Timothy A Keller & Tom M Mitchell, 2014. "Identifying Autism from Neural Representations of Social Interactions: Neurocognitive Markers of Autism," PLOS ONE, Public Library of Science, vol. 9(12), pages 1-22, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1006397. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.