Behavioral correlates of cortical semantic representations modeled by word vectors

The quantitative modeling of semantic representations in the brain plays a key role in understanding the neural basis of semantic processing. Previous studies have demonstrated that word vectors, which were originally developed for use in the field of natural language processing, provide a powerful tool for such quantitative modeling. However, whether semantic representations in the brain revealed by the word vector-based models actually capture our perception of semantic information remains unclear, as there has been no study explicitly examining the behavioral correlates of the modeled brain semantic representations. To address this issue, we compared the semantic structure of nouns and adjectives in the brain estimated from word vector-based brain models with that evaluated from human behavior. The brain models were constructed using voxelwise modeling to predict the functional magnetic resonance imaging (fMRI) response to natural movies from semantic contents in each movie scene through a word vector space. The semantic dissimilarity of brain word representations was then evaluated using the brain models. Meanwhile, data on human behavior reflecting the perception of semantic dissimilarity between words were collected in psychological experiments. We found a significant correlation between brain model- and behavior-derived semantic dissimilarities of words. This finding suggests that semantic representations in the brain modeled via word vectors appropriately capture our perception of word meanings.Author summary: Word vectors, which have been originally developed in the field of engineering (natural language processing), have been extensively leveraged in neuroscience studies to model semantic representations in the human brain. These studies have attempted to model brain semantic representations by associating them with the meanings of thousands of words via a word vector space. However, there has been no study explicitly examining whether the brain semantic representations modeled by word vectors actually capture our perception of semantic information. To address this issue, we compared the semantic representational structure of words in the brain estimated from word vector-based brain models with that evaluated from behavioral data in psychological experiments. The results revealed a significant correlation between these model- and behavior-derived semantic representational structures of words. This indicates that the brain semantic representations modeled using word vectors actually reflect the human perception of word meanings. Our findings contribute to the establishment of word vector-based brain modeling as a useful tool in studying human semantic processing.