Action-Oriented Minds

While most theories of behavior assume that animals maximize rewards, much natural behavior, such as curiosity and play, is intrinsically originated, with no reference to extrinsic tasks. In this talk, I will introduce the maximum-occupancy-principle (MOP), an intrinsic motivation signal that adopts as principle the diversity and variability of natural behavior. By maximizing the occupancy (entropy) of action-state paths, agents endowed with this intrinsic motivation can generate all sorts of behaviors that are capable of, in a close-loop, open-ended manner, solely shaped by terminal states and cognitive and body constraints. Behaviors spontaneously and dynamically reorganize into goals and subgoals. Applied to neural networks, MOP promotes the visitation of the full repertoire of activity patterns, avoiding dynamical collapse. All in all, MOP provides the first open-ended generative model of behavior in embodied agents.

The free-energy principle, and its corollary active inference, suggest that both action and perception can be understood as minimisation of variational free energy constrained by a probabilistic generative model of the agent's environment. On slower timescales, it provides an account of learning as the optimisation of the parameters of the agent's generative model. However, the dynamic interaction of action, perception, and learning under this framework has not been well studied.

The open-ended complexity of natural environments means that it is generally infeasible for agents to model their sensorimotor environment comprehensively. However, the presence of adaptive behaviour constrains agents to behaviourally relevant trajectories, reducing the diversity of the sensorimotor interactions they experience in their lifetime. Thus, agents need not possess a comprehensive and veridical model of the environment, but can operate on much simpler “action-oriented” models that are tailored towards their sensorimotor trajectory. Yet how such action-oriented models are learned remains unclear.

Specifically, at the onset of learning, agents must learn from limited sensory experience, and this can lead to models that initially overfit the environment, making suboptimal predictions about the consequences of action. Subsequently, using these models to determine goal-oriented actions can result in biased and suboptimal sensory samples from the environment, further compounding the model's inefficiencies and ultimately entrenching maladaptive cycles of learning and control, a process we refer to as a “bad bootstrap”.

Here, we exploit active inference to show in a simple model that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample-efficient learning algorithms.