Model based planners reflect on their model-free propensities

Dual-reinforcement learning theory proposes behaviour is under the tutelage of a retrospective, value-caching, model-free (MF) system and a prospective-planning, model-based (MB), system. This architecture raises a question as to the degree to which, when devising a plan, a MB controller takes account of influences from its MF counterpart. We present evidence that such a sophisticated self-reflective MB planner incorporates an anticipation of the influences its own MF-proclivities exerts on the execution of its planned future actions. Using a novel bandit task, wherein subjects were periodically allowed to design their environment, we show that reward-assignments were constructed in a manner consistent with a MB system taking account of its MF propensities. Thus, in the task participants assigned higher rewards to bandits that were momentarily associated with stronger MF tendencies. Our findings have implications for a range of decision making domains that includes drug abuse, pre-commitment, and the tension between short and long-term decision horizons in economics.Author summary: Extant theories of model-based (MB) planning usually ignore whether one’s model-free (MF) tendencies cohere, or conflict, with one’s desired goal-directed actions. From the perspective of these theories, one’s MF disposition to abstinence or indulgent alcohol-drinking is irrelevant for a planner who intends to avoid alcohol consumption at a forthcoming party. However, MF tendencies can hijack one’s actions such that plans consistent with MF tendencies are often adhered to, whereas plans that conflict with MF tendencies are prone to be disregarded. Here, we show that planners consider whether their MF proclivities facilitate or hinder goal-pursuit. We show that people are disposed to design a task-environment in such a way that renders MB goals consistent (rather than conflicting) with their MF dispositions.