Neural responses to natural and model-matched stimuli reveal distinct computations in primary and nonprimary auditory cortex

A central goal of sensory neuroscience is to construct models that can explain neural responses to natural stimuli. As a consequence, sensory models are often tested by comparing neural responses to natural stimuli with model responses to those stimuli. One challenge is that distinct model features are often correlated across natural stimuli, and thus model features can predict neural responses even if they do not in fact drive them. Here, we propose a simple alternative for testing a sensory model: we synthesize a stimulus that yields the same model response as each of a set of natural stimuli, and test whether the natural and “model-matched” stimuli elicit the same neural responses. We used this approach to test whether a common model of auditory cortex—in which spectrogram-like peripheral input is processed by linear spectrotemporal filters—can explain fMRI responses in humans to natural sounds. Prior studies have that shown that this model has good predictive power throughout auditory cortex, but this finding could reflect feature correlations in natural stimuli. We observed that fMRI responses to natural and model-matched stimuli were nearly equivalent in primary auditory cortex (PAC) but that nonprimary regions, including those selective for music or speech, showed highly divergent responses to the two sound sets. This dissociation between primary and nonprimary regions was less clear from model predictions due to the influence of feature correlations across natural stimuli. Our results provide a signature of hierarchical organization in human auditory cortex, and suggest that nonprimary regions compute higher-order stimulus properties that are not well captured by traditional models. Our methodology enables stronger tests of sensory models and could be broadly applied in other domains.Author summary: Modeling neural responses to natural stimuli is a core goal of sensory neuroscience. A standard way to test sensory models is to predict responses to natural stimuli. One challenge with this approach is that different features are often correlated across natural stimuli, making their contributions hard to tease apart. We propose an alternative in which we compare neural responses to a natural stimulus and a “model-matched” synthetic stimulus designed to yield the same responses as the natural stimulus. We tested whether a standard model of auditory cortex can explain human cortical responses measured with fMRI. Model-matched and natural stimuli produced nearly equivalent responses in primary auditory cortex, but highly divergent responses in nonprimary regions, including those selective for music or speech. This dissociation was not evident using model predictions because of the influence of feature correlations in natural stimuli. Our results provide a novel signature of hierarchical organization in human auditory cortex, and suggest that nonprimary regions compute higher-order stimulus properties that are not captured by traditional models. The model-matching methodology could be broadly applied in other domains.