Learning Reward Uncertainty in the Basal Ganglia

Learning the reliability of different sources of rewards is critical for making optimal choices. However, despite the existence of detailed theory describing how the expected reward is learned in the basal ganglia, it is not known how reward uncertainty is estimated in these circuits. This paper presents a class of models that encode both the mean reward and the spread of the rewards, the former in the difference between the synaptic weights of D1 and D2 neurons, and the latter in their sum. In the models, the tendency to seek (or avoid) options with variable reward can be controlled by increasing (or decreasing) the tonic level of dopamine. The models are consistent with the physiology of and synaptic plasticity in the basal ganglia, they explain the effects of dopaminergic manipulations on choices involving risks, and they make multiple experimental predictions.Author Summary: To maximize their chances for survival, animals need to base their decisions not only on the average consequences of chosen actions, but also on the variability of the rewards resulting from these actions. For example, when an animal’s food reserves are depleted, it should prefer to forage in an area where food is guaranteed over an area where the amount of food is higher on average but variable, thus avoiding the risk of starvation. To implement such policies, animals need to be able to learn about variability of rewards resulting from taking different actions. This paper proposes how such learning may be implemented in a circuit of subcortical nuclei called the basal ganglia. It also suggests how the information about reward uncertainty can be used during decision making, so that animals can make choices that not only maximize expected rewards but also minimize risks.