Abstract
The default mode network (DMN) is believed to subserve the baseline mental activity in humans. Its highest energy consumption compared to other brain networks and its intimate coupling with conscious awareness are both pointing to an overarching function. Many research streams support an evolutionarily adaptive role in envisioning experience to anticipate the future.
The present paper proposes a process model that tries to explain how the DMN may implement continuous evaluation and prediction of the environment to guide behavior. DMN function is recast in mathematical terms of control theory and reinforcement learning based on Markov decision processes. We argue that our formal account of DMN function naturally accommodates as special cases the previously proposed cognitive accounts on (1) predictive coding, (2) semantic associations, and (3) a “sentinel” role. Moreover, this process model for the neural optimization of complex behavior in the DMN offers parsimonious explanations for recent experimental findings in animals and humans.
1 Introduction
In the absence of external stimulation, the human brain is not at rest. In the beginning of the 21st century, brain imaging may have been the first technique to allow for the discovery of a unique brain network that probably subserves baseline mental activities (Raichle et al., 2001; Buckner et al., 2008; Bzdok and Eickhoff, 2015). The “default mode network” (DMN) continues to metabolize large quantities of oxygen and glucose energy to maintain neuronal computation during free-ranging thought (Kenet et al., 2003; Fiser et al., 2004). The baseline energy demand is only weakly modulated at the onset of defined psychological tasks (Gusnard and Raichle, 2001). At its opposite, during sleep, the decoupling of brain structures discarded the idea of the DMN being only a passive network resonance and rather supported an important role in sustaining conscious awareness (Horovitz et al., 2009).
This dark matter of brain physiology (Raichle, 2006) begs the question of the biological purpose underlying DMN activity. What has early been described as the “stream of consciousness” in psychology (James, 1890) found a potential neurobiological manifestation in the DMN (Shulman et al., 1997; Raichle et al., 2001). We propose that this set of some of the most advanced regions in the association cortex (Mesulam, 1998; Margulies et al., 2016b) are responsible for higher-order control of human behavior. Our functional account follows the notion of “a hierarchy of brain systems with the DMN at the top and the salience and dorsal attention systems at intermediate levels, above thalamic and unimodal sensory cortex” (Carhart-Harris and Friston, 2010).
1.1 Towards a formal account of default mode function: higher-order control of the organism
The human DMN is responsible for extended parts of the baseline neural activity, which typically decreases when engaged in psychological experiments (Gusnard and Raichle, 2001). The standard mode of neural information maintenance and manipulation has been argued to mediate evolutionarily conserved functions (Brown, 1914; Binder et al., 1999; Buzsáki, 2006). Today, many psychologists and neuroscientists believe that the DMN implements some form of probabilistic estimation of past, hypothetical, and future events (Fox et al., 2005; Hassabis et al., 2007; Schacter et al., 2007; Binder et al., 2009; Buckner et al., 2008; Spreng et al., 2009). This brain network might have emerged to continuously predict the environment using mental imagery as an evolutionary advantage (Suddendorf and Corballis, 2007). However, information processing in the DMN has also repeatedly been shown to directly impact human behavior.Goal-directed task performance improved with decreased activity in default mode regions (Weissman et al., 2006) and increased DMN activity was linked to more task-independent, yet sometimes useful thoughts (Mason et al., 2007; Seli et al., 2016). Gaining insight into DMN function is particularly challenging because this network appears to simultaneously modulate perception-action cycles and entertain probabilistic contemplations across time, space, and content domains (Boyer, 2008).
The present work adopts the perspective of a human agent faced with the choice of the next actions and guided by outcomes of really happened, hypothetically imagined, and expected futures to optimize behavioral performance. Formally, a particularly attractive framework to describe, quantify, and predict intelligent systems, such as the brain, is proposed to be the combination of control theory and reinforcement learning (RL). An intelligent agent improves the interaction with the environment by continuously updating its computation of value estimates and action predispositions through integration of feedback outcomes. Henceforth, control refers to the influence that an agent exerts when interacting with the environment to reach preferred states.
Psychologically, the more the ongoing executed task is unknown and unpracticed, the less stimulus-independent thoughts occur (Filler and Giambra, 1973; Teasdale et al., 1995; Christoff et al., 2016). Conversely, it is known that, the more the world is easy to predict, the more human mental activity becomes detached from the actual sensory environment (Antrobus et al., 1966; Pope and Singer, 1978). Without requiring explicit awareness, these “offline” processes may contribute to optimizing control of the organism. We formalize a policy matrix to capture the space of possible actions that the agent can perform on the environment given the current state. A value function maps environmental objects and events (i.e., states) to expected rewards. Switching between states reduces to a sequential processing model. Informed by outcomes of performed actions, neural computation reflected in DMN dynamics could be constantly shaped by prediction error through feedback loops. Such an RL account of DMN function can naturally embed human behavior into the tension between exploitative action with immediate gains and exploratory action with longer-term gratification.
We argue that DMN implication in many advanced cognitive processes in humans can be recast as prediction error minimization based on probabilistic mental simulations, thus maximizing action outcome across multiple time scales. Such a purposeful optimization objective may be solved by a stochastic approximation based on a brain implementation of Markov Chain Monte Carlo (MCMC) sampling (Tenenbaum et al., 2011). Even necessarily imperfect memory recall, random day-time mind-wandering, and seemingly arbitrary dreams during sleep may provide blocks of pseudo-experience to iteratively optimize the behavior of the organism. It has indeed been proposed that the human brain’s energy budget is largely dedicated to “the development and maintenance of [a] probabilistic model of anticipated events”(Raichle and Gusnard, 2005). This idea is invigorated by empirical evidence from neuroscience experiments (Körding and Wolpert, 2004; Fiser et al., 2004). The present paper proposes a process model that satisfies this contention.
2 Known neurobiological properties of the default mode network
We begin by a neurobiological deconstruction of the DMN based on experimental findings in the neuroscience literature. This walkthrough across each node of the DMN will outline the individual functional profiles, paving the way for their algorithmic interpretation in our formal account (section 3).
2.1 The posteromedial cortex: global monitoring and information integration
The midline structures of the human DMN, including the posteromedial cortex (PMC) and the medial prefrontal cortex (mPFC), are probably responsible for the highest turn-over of energy consumption (Raichle et al., 2001; Gusnard and Raichle, 2001). These metabolic characteristics go hand-in-hand with neuroimaging analyses that suggested the PMC and mPFC to potentially represent the functional backbone of the DMN (Andrews-Hanna et al., 2010; Hagmann et al., 2008).
Normal and disturbed metabolic fluctuations in the human PMC have been closely related to changes of conscious awareness (Cavanna and Trimble, 2006). Indeed, the PMC matures relatively late (i.e., myelination) during postnatal development in monkeys (Goldman-Rakic, 1987), which is generally considered to be a sign of evolutionary sophistication. This DMN node has long been speculated to reflect constant computation of environmental statistics and its internal representation as an inner “mind’s eye” (Cavanna and Trimble, 2006; Leech and Sharp, 2014). For instance, Bálint’s syndrome is a neurological disorder of conscious awareness that results from medial damage in the parietal cortex (Bálint et al., 1909). Neurological patients are plagued by an inability to combine various individual features of the visual environment into an integrated whole (i.e., simultanagnosia) as well as an inability to direct action towards currently unattended environmental objects (i.e., optic ataxia). This can be viewed as a high-level impairment in gathering information about alternative objects (i.e., exploration) as well as leveraging these environmental opportunities (i.e., exploitation). Congruently, the human PMC was coupled in two functional connectivity analyses (Bzdok et al., 2015) with the amygdala, involved in significance evaluation, and the nucleus accumbens (NAc), involved in reward evaluation. Specifically, among all parts of the PMC, the ventral posterior cingulate cortex was most connected to the laterobasal nuclei group of the amygdala (Bzdok et al., 2015). This amygdalar subregion has been proposed to continuously scan environmental input for biological relevance assessment (Bzdok et al., 2013a).
The putative role of the PMC in continuous abstract integration of environmental relevance and ensuing top-level guidance of action on the environment is supported by many neuroscience experiments. Electrophysiological recordings in animals implicated PMC neurons in strategic decision making (Pearson et al., 2009), risk assessment (McCoy and Platt, 2005), outcome-dependent behavioral modulation (Hayden et al., 2009), as well as approach-avoidance behavior (Vann et al., 2009). Neuron spiking activity in the PMC allowed distinguishing whether a monkey would pursue an exploratory or exploitative behavioral strategy during food foraging (Pearson et al., 2009). Further, single-cell recordings in the monkey PMC demonstrated this brain region’s sensitivity to subjective target utility (McCoy and Platt, 2005) and integration across individual decision-making instances (Pearson et al., 2009). This DMN node encoded the preference for or aversion to options with uncertain reward outcomes and its spiking activity was more associated with subjectively perceived relevance of a chosen object than by its actual value, based on an “internal currency of value” (McCoy and Platt, 2005). In fact, direct stimulation of PMC neurons promoted exploratory actions, which would otherwise be shunned (Hayden et al., 2008). Graded changes in firing rates of PMC neurons indicated changes in upcoming choice trials, while their neural patterns were distinct from neuronal spike firings that indicated choosing either option. Similarly in humans, the DMN has been shown to gather and integrate information over different parts of auditory narratives in an fMRI study (Simony et al., 2016).
Moreover, the retrosplenial portion of the PMC could support representation of action possibilities and evaluation of reward outcomes by integrating information from memory recall and different perspective frames. Regarding memory recall, retrosplenial damage has been consistently associated with anterograde and retrograde memory impairments of various kinds of sensory information in animals and humans (Vann et al., 2009). Regarding perspective frames, the retrosplenial subregion of the PMC has been proposed to mediate between the organism’s egocentric (i.e., focused on sensory environment) and allocentric (i.e., focused on world knowledge) viewpoints in animals and humans (Epstein, 2008; Burgess, 2008; Valiquette and McNamara, 2007).
Consequently, the PMC may contribute to overall DMN function by monitoring the subjective outcomes of possible actions and integrating that information with memory and perspective frames into short- and longer-term behavioral agendas. Estimated value, that differs across individuals, probably enriches statistical assessment of the environment to map and predict delayed reward opportunities in the future. In doing so, the PMC may continuously adapt the organism to changes in both the external environment and its internal representation to enable strategic behavior.
2.2 The prefrontal cortex: action consideration and stimulus-value association
Analogous to the PMC, the dorsomedial PFC (dmPFC) of the DMN is believed to subserve multi-sensory processes across time, space, and content domains to exert top-level control on behavior. Comparing to the PMC, however, dmPFC function may be closer to a “mental sketchpad” (Goldman-Rakic et al., 1996), as it potentially subserves the de-novo construction and manipulation of meaning representations instructed by stored semantics and memories (Bzdok et al., 2013c).
The dmPFC may subserve inference, representation, and assessment of one’s own and other individuals’ action considerations. Generally, neurological patients with tissue damage in the prefrontal cortex are known to struggle with adaptation to new stimuli and events (Stuss and Benson, 1986). Specifically, neural activity in the human dmPFC reflected expectations about other peoples’ actions and outcomes of these predictions. Neural activity in the dmPFC indeed explained the performance decline of inferring other peoples’ thoughts in aging humans (Moran et al., 2012). Certain dmPFC neurons in macaque monkeys exhibited a preference for processing others’, rather than own, behavior with fine-grained adjustment of contextual circumstances (Yoshida et al., 2010). Such highly abstract neural computations necessarily rely on the construction of probabilistic internal information drawing from memory recall, mental scene elaboration processes, and stored knowledge of world regularities. Moreover, in a computational neuroimaging experiment, dorsomedial PFC activity preferentially processed the consequences of action choices that were considered but not actually executed, whereas ventromedial PFC (vmPFC) activity processed especially value outcomes of actions actually performed on the environment (Nicolle et al., 2012).
Comparing to the dmPFC, the vmPFC probably subserves subjective value evaluation and risk estimation of relevant environmental stimuli (Fig. 2). The ventromedial prefrontal DMN may subserve adaptive behavior by bottom-up-driven processing of what matters now, probably drawing on sophisticated value representations (Kringelbach and Rolls, 2004; O’Doherty et al., 2015). Quantitative lesion findings across 344 human individuals confirmed a substantial impairment in value-based action choice (Gläscheret al., 2012). Indeed, this DMN node is preferentially connected with reward-related and limbic regions. The vmPFC is well known to have direct connections with the NAc in axonal tracing studies in monkeys (Haber et al., 1995). Congruently, the gray-matter volume of the vmPFC and NAc correlated with indices of value-guided behavior and reward attitudes in humans (Lebreton et al., 2009). NAc activity is thought to reflect reward prediction signals from dopaminergic neurotransmitter pathways (Schultz, 1998) that not only channel action towards basic survival needs but also enable more abstract reward processings, and thus perhaps RL, in humans (O’Doherty et al., 2015). Consistently, diffusion MRI tractography in humans and monkeys (Croxson et al., 2005) quantified the NAc to be more connected to the vmPFC than dmPFC in both species. Two different functional connectivity analyses in humans also strongly connected the vmPFC with the NAc, hippocampus (HC), and PMC (Bzdok et al., 2015). In line with these connectivity findings in animals and humans, the vmPFC is often proposed to represent triggered emotional and motivational states (Damasio et al., 1996). Such real or imagined arousal states could be mapped in the vmPFC as a bioregulatory disposition influencing cognition and decision making. In neuroeconomic studies of human decision making, the vmPFC consistently reflects an individuals subjective value estimates (Behrens et al., 2008). This may be why performance within and across participants was related to state encoding in the vmPFC (Schuck et al., 2016). Such a “cognitive map” of the action space was argued to encode the current task state even when states are unobservable from the sensory environment.
2.3 The hippocampus: memory, space, and experience replay
The DMN midline has close functional links with the HC in the medial temporal lobe (Vincent et al., 2006; Shannon et al., 2013) —a region long known to be involved in memory operations and spatial navigation in animals and humans.
While the HC is traditionally believed to allow recalling past experience, there is now increasing evidence for an important role in constructing mental models in general (Zeidman and Maguire, 2016; Schacter et al., 2007; Gelbard-Sagiv et al., 2008; Javadi et al., 2017; Boyer, 2008). Its recursive anatomical architecture may be specifically designed to allow reconstructing entire episodes of experience from memory fragments. Indeed, hippocampal damage is not only associated with an impairment in re-experiencing the past (i.e., amnesia), but also imagination dedicated to one’s own future and imagination of experiences more broadly (Hassabis et al., 2007). Mental scenes created by neurological patients with HC lesion exposed a lack of spatial integrity, richness in detail, and overall coherence. Single-cell recordings in the animal HC revealed constantly active neuronal populations whose firing coincided with specific locations in space during environmental navigation. Indeed, when an animal is choosing between alternative paths, the corresponding neuronal populations in the HC spike one after another (Johnson and Redish, 2007). Such neuronal patterns in the HC appear to directly indicate upcoming behavior, such as in planning navigational trajectories (Pfeiffer and Foster, 2013) and memory consolidation of choice relevance (De Lavilléon et al., 2015). Congruently, London taxi drivers, humans with high performance in spatial navigation, were shown to exhibit increased gray-matter volume in the HC (Maguire et al., 2000).
There is hence increasing evidence that HC function extends beyond simple forms of encoding and reconstruction of memory and space information. Based on spike recordings of hippocampal neuronal populations, complex spiking patterns can be followed across extended periods including their modification of input-free self-generated patterns after environmental events (Buzsáki, 2004). Specific spiking sequences, which were elicited by experimental task design, have been shown to be re-enacted spontaneously during quiet wakefulness and sleep (Hartley et al., 2014; ONeill et al., 2010). Moreover, neuronal spike sequences measured in hippocampal place cells of rats featured re-occurrence directly after experimental trials as well as directly before upcoming experimental trials (Diba and Buzsáki, 2007). Similar spiking patterns in hippocampal neurons during rest and sleep have been proposed to be critical in communicating local information to the neocortex for long-term storage, potentially also in the nodes of the DMN. Moreover, in mice, invasively triggering spatial experience recall in the HC during sleep has been demonstrated to subsequently alter action choice during wakefulness (De Lavilléon et al., 2015). These HC-subserved mechanisms probably contribute to advanced cognitive processes that require re-experiencing or newly constructed mental scenarios, such as in recalling autobiographical memory episodes (Hassabis et al., 2007).
Thus, the HC probably orchestrates re-experience of environmental aspects for consolidations based on re-enactment and for integration into rich mental scene construction (Deuker et al., 2016; Bird et al., 2010). As such, the HC may impact ongoing perception of and action on the environment (Zeidman and Maguire, 2016; De Lavilléon et al., 2015).
2.4 The right and left TPJ: prediction error signaling and world semantics
The DMN emerges with its midline structures early in human development (Doria et al., 2010), while the right and left TPJs may become fully integrated into this major brain network only after birth. The TPJs are known to exhibit hemispheric differences based on microanatomical properties and gyrification patterns (Seghier, 2013). Globally, neuroscientific investigations on hemispheric functional specialization have highlighted the right versus left cerebral hemisphere as dominant for attentional versus semantic functions (Seghier, 2013; Bzdok et al., 2013b,2016a; Stephan et al., 2007).
The TPJ in the right-hemispheric DMN (RTPJ) has been shown to be closely related to multi-sensory prediction and prediction error signaling. It is probably central for action initiation during goal-directed psychological tasks and for sensorimotor behavior by integrating multi-sensory attention (Corbetta and Shulman, 2002). Involvement of this DMN node was repeatedly reported in multi-step action execution (Hartmann et al., 2005), visuo-proprioceptive conflict (Balslev et al., 2005), and detection of environmental changes across visual, auditory, or tactile stimulation (Downar et al., 2000). Direct electrical stimulation of the human RTPJ during neurosurgery was associated with altered perception and stimulus awareness (Blanke et al., 2002). It was argued that the RTPJ encodes actions and ensuing outcomes, without necessarily relating those to value estimation (Liljeholm et al., 2013; Hamilton and Grafton, 2008; Jakobs et al., 2009). Additionally, neural activity in the RTPJ has been proposed to reflect stimulus-driven attentional reallocation to self-relevant and unexpected sources of information as a circuit breaker that recalibrates control and maintenance brain networks (Bzdok et al., 2013b; Corbetta et al., 2008). Indeed, neurological patients with RTPJ damage have particular difficulties with multi-step actions (Hartmann et al., 2005). In the face of large discrepancies between actual and previously predicted environmental events the RTPJ acts as a potential switch between externally-oriented mind sets focussed on the sensory environment and internally-oriented mind sets focussed on mental scene construction. For instance, temporally induced RTPJ damage in humans diminished the impact of predicted intentions of other individuals (Young et al., 2010), a capacity believed to be enabled by the DMN. The RTPJ might hence be an important relay that shifts away from the internally directed baseline processes to, instead, deal with unexpected environmental stimuli and events.
The TPJ in the left-hemispheric DMN (LTPJ), in turn, has a close relationship to Wernicke’s area involved in semantic processes, such as in spoken and written language. Neurological patients with damage in Wernicke’s area have a major impairment of language comprehension when listening to others or reading a book. Patient speech preserves natural rhythm and normal syntax, yet the voiced sentences lack meaning (i.e., aphasia). Abstracting from speech interpretations in linguistics and neuropsychology, the LTPJ probably mediates access to and integration of world knowlege, such as required during action considerations (Binder and Desai, 2011; Seghier, 2013). Consistent with this view, LTPJ damage in humans also entails problems in recognizing others’ pantomimed action towards objects without obvious relation to processing explicit language content (Varney and Damasio, 1987). Inner speech also hinges on knowledge recall about the physical and social world. Indeed, the internal production of verbalized thought (“language of the mind”) was closely related to the LTPJ in a pattern analysis of brain volume (Geva et al., 2011). Further, episodic memory recall and mental imagery strongly draw on re-assembling world knowledge. Isolated building blocks of world structure probably get rebuilt in internally constructed mental scenarios that guide present action choice, weigh hypothetical possibilities, and forecast future events. The LTPJ may hence facilitate the automated environmental predictions by incorporating experience-derived building blocks of world regularities into ongoing action, planning, and problem solving.
3 Reinforcement learning: a process model for DMN function
We now argue the outlined neurobiological properties of the DMN nodes to be sufficient for implementing all components of a full-fledged RL system. Recalling past experience, considering candidate actions, random sampling of possible experiences, as well as estimation of instantaneous and expected delayed reward outcomes are key components of intelligent RL agents that are plausible to functionally intersect in the DMN.
RL is a problem-solving technique in which, through repeated interactions with an environment, an agent learns to reach goals and optimize reward signals in an iterative trial-and-error fashion (Fig. 3). At a given moment, each taken action a triggers a change in the state of the environment s → s′, accompanied by environmental feedback signals as reward r = r(s, a, s′) collected by the agent. If the collected reward outcome yields a negative value it can be more naturally interpreted as punishment. In this view, the environment is partially controlled by the action of the agent and the reward can be thought of as satisfaction —or aversion —accompanying the execution of a particular action.
The environment is generally taken as stochastic, that is, changing in random ways. In addition, the environment is only partially observable in the sense that only limited aspects of the environment’s state are accessible to the agent’s sensory perception. (Starkweather et al., 2017). We assume that volatility of the environment is realistic in a computational model which sets out to explain DMN functions of the human brain. We argue that a functional account of the DMN based on RL can naturally embed human behavior in the tension between exploitative action with immediate gains and explorative action with longer-term reward outcomes (Dayan and Daw, 2008).
In short, DMN implication in a diversity of particularly advanced cognitive processes can be parsimoniously explained as probabilistic mental scene of experience simulations coupled with prediction error minimization to calibrate action trajectories for reward outcome maximization at different time scales. Such a purposeful optimization objective may be solved by a stochastic approximation based on a brain implementation of MCMC sampling (Tenenbaum et al., 2011).
3.1 Markov decision processes
Model-free RL has had great success in many real-world problems, including robotics (Ng et al., 2004; Abbeel and Ng, 2004), super-human performance in complex video games (Mnih et al., 2015), and strategic board games like the breakthrough results upon recently on the game of Go (Silver et al., 2016), considered to be a golden benchmark problem in artificial intelligence. We emphasize that the brain in general, and the DMN in particular, is a physical system governed by the laws of physics and can be formally described by Markov processes at a sufficiently coarse scale. It has indeed been previously proposed (Tegmark, 2016) that any system obeying the laws of classical physics can be accurately modeled as a Markov process as long as the time step is sufficiently short. The process has memory if the next state depends not only on the current state but also on a finite number of past states. Rational probabilistic planning can be reformulated as a standard memoryless Markov process by simply expanding the definition of the state s to include experience episodes of the past.
In artificial intelligence and machine learning, a popular computational model for multi-step decision processes in such an environment are Markov decision processes (MDPs) (Sutton and Barto, 1998). An MDP operationalizes a sequential decision process in which it is assumed that environment dynamics are determined by a Markov process, but the agent cannot directly observe the underlying state. Instead, the agent tries to optimize a subjective reward signal (i.e., is likely to be different for another agent in the same state), by maintaining probability distributions over actions according to their expected utility. This is a minimal set of assumptions that can be made about an environment faced by an agent engaged in active learning.
Model-free RL can be plausibly realized in the human brain (O’Doherty et al., 2015). Indeed, it has been proposed (Gershman et al., 2015) that a core property of human intelligence underlie improvement of expected utility outcomes as a strategy for action choice in uncertain environments, a situation perfectly captured by the formalism of MDPs. It has also long been proposed (Dayan and Daw, 2008) that there is a rather direct mapping of model-free RL learning algorithms onto aspects of the brain. The neurotransmitter dopamine could serve as a “teaching signal” to better estimate value associations and action policies by controlling synaptic plasticity in the reward-processing circuitry, including the NAc. In contrast, model-based RL would start off with some mechanistic assumptions about the dynamics of the world. These can be assumptions about the physical laws governing the agent’s environment or constraints on the state space and transitions between states.
In our adopted model-free RL framework, an agent might represent such knowledge about the world as follows:
r(s, “stand still”) = 0 if s does not correspond to a location offering relevant resources.
p(s′|s, “stand still”) = 1 if s′ = s and 0 otherwise.
etc.
Definition
Mathematically, an MDP is simply a quintuplet (, , r, p) where
is the set of states, such as .
is the set of actions, such as .
is the reward function, so that r(s, a, s′) is the instant reward for taking action a in state s followed by a state-transition s → s′.
, (s, a, s′) ↦ p(s′|s, a), the probability of moving to state s′ if action a is taken from state s. In addition, one requires that such transitions be Markovian. Consequently, the future states are independent of past states and only depend on the present state and action taken.
3.1.1 Accumulated rewards and policies
The behavior of the agent is governed by a policy, which maps states of the world to probability distributions over actions. Starting at time t = 0, following a policy π generates a trajectory of action choices as follows:
We assume time-invariance in that we expect the dynamics of the process to be equivalent over sufficiently long time windows of equal length (i.e., stationarity). Since an action executed in the present moment might have repercussions in the far future, it turns out that the quantity to optimize is not the instantaneous rewards r(s, a), but a cumulative reward estimate which takes into account expected reward from action choices in the future. A common approach to modeling this accumulation is the time-discounted cumulative reward
This random variable1 measures the cumulative reward of following an action policy π. Note that value buffering may be realized in the vmPFC. This DMN node has direct connections to to the NAc, known to be involved in reward evaluation.
The goal of the RL agent is then to update this action policy in order to maximize Gπ on average (cf. below). In (1), the definition of cumulative reward Gπ, the constant γ (0 ≤ γ < 1) is the reward discount factor, viewed to be characteristic for a certain agent. On the one hand, setting γ = 0 yields perfectly hedonistic behavior. An agent with such a shortsighted time horizon is exclusively concerned with immediate rewards. This is however not compatible with coordinated planning of long-term goal that is potentially subserved by neural activity in the DMN. On the other hand, setting 0 < γ < 1 allows a learning process to arise. A positive γ can be seen as calibrating risk-seeking trait of the intelligent agent, that is, the behavioral predispositions related to trading longer delays for higher reward outcomes. Such an agent puts relatively more emphasis on rewards expected in a longer-term future. More specifically, rewards that are not expected to come within τ:= 1/(1 − γ) time steps from the present point are disregarded. This reduces the variance of expected rewards accumulated across considered action cascades by limiting the depth of the search tree. Given that there is more uncertainty in the farsighted future, it is important to appreciate that a stochastic policy estimation is more advantageous in many RL settings.
3.2 The components of reinforcement learning in the DMN
Given only the limited information available from an MDP, at a state s the average utility of choosing an action a under a policy π can be captured by the single number called the Q-value for the state-action pair (s, a). In other words, Qπ (s, a) corresponds to the expected reward over all considered action trajectories, in which the agent sets out in the environment in state s, chooses action a, and then follows the policy π to select future actions. For the brain, Qπ (s, a) defined in (2) provides the subjective utility of executing a specific action. It thus answers the question “What is the expected utility of taking action a in this situation ?”. Qπ(s, a) offers a formalization of optimal behavior that may well capture processing aspects of the DMN in human agents.
3.2.1 Optimal behavior and the Bellman equation
Optimal behavior of the agent corresponds to a strategy π* for choosing actions such that, for every state, the chosen action guarantees the best possible reward on average. Formally,
The learning goal is to approach the policy π* as close as possible, that is to solve the MDP. Note that (3) presents merely a definition and does not lend itself as a candidate schema for solving MDPs with even moderately-sized action and state spaces (i.e., intractability). Fortunately, the Bellman equation (Sutton and Barto, 1998) provides a fixed-point relation which defines Q* implicitly via a sampling procedure, without querying the entire space of policies, with the form where the so-called Bellman transform Bel(Q) of an arbitrary Q-value function is another Q-value function defined by
The Bellman equation (4) is a temporal consistency equation which provides a dynamic decomposition of optimal behavior by dividing the Q-value function into the immediate reward and the discounted rewards of the upcoming states. The optimal Q-value operator Q* is a fixed point for this equation. As a consequence of this decomposition, the complicated dynamic programming problem (3) is broken down into simpler sub-problems at different time points. Indeed, exploitation of hierarchical structure in action considerations has previously been related to the medial prefrontal part of the DMN (Koechlin et al., 1999; Braver and Bongiolatti, 2002). Using the Bellman equation, each state can be associated with a certain value to guide action towards a preferred state, thus improving on the current action policy of the agent. Note that in (4) the random sampling is performed only over quantities which depend on the environment. This aspect of the learning process can unroll off-policy by observing state transitions triggered by another (possibly stochastic) behavioral policy.
Neural correlates of the Bellman equation in the DMN
Relating decomposition of consecutive action choices by the Bellman equation to neuroscience, specific neural activity in the dorsal prefrontal cortex (BA9) was linked to processing “goal-tree sequences” in human neuroimaging experiments (Koechlin et al., 1999, 2000). Sub-goal exploration may require multi-task switching between cognitive processes as later parts of a solution frequently depend on respective earlier steps in a given solution path, which necessitates storage of expected intermediate outcomes. As such, “cognitive branching” operations for nested processing of behavioral strategies are likely to entail secondary reallocation of attention and working-memory resources. Further neuroimaging experiments (Braver and Bongiolatti, 2002) corroborated the prefrontal DMN to subserve “processes related to the management and monitoring of sub-goals while maintaining information in working memory”. Moreover, neurological patients with lesions in this DMN node were reported to be impaired in aspects of realizing “multiple sub-goal scheduling” (Burgess et al., 2000). Hence, the various advanced human mental abilities subserved by the DMN, such as planning and abstract reasoning, can be viewed to involve some form of action-decision branching to enable higher-order executive control.
3.2.2 Value approximation and the policy matrix
As already mentioned in the previous section, Q-learning optimizes over the class of deterministic policies of the form (3). State spaces may be extremely large, and tracking all possible states and actions may require prohibitively excessive computation and memory resources. The need of maintaining an explicit table of states can be eliminated by instead using of an approximate Q-value function by keeping track of an approximating parameter θ of much lower dimension than the number of states. At a given time step, the world is in a state , and the agent takes an action which it expects to be the most valuable on average, namely
This defines a mapping from states directly to actions. For instance, a simple linear model with a kernel ɸ would be of the form , where ɸ(s, a) would represent a high-level representation of the state-action pairs (s, a), as was previously proposed (Song et al., 2016), or artificial neural-network models as demonstrated in recent seminal investigations (Mnih et al., 2015; Silver et al., 2016) for playing complex games (atari, Go, etc.) at super-human levels. In the DMN, the dmPFC would implement such a hard-max lookup over the action space. The model parameters θ would correspond to synaptic weights and connection strengths within and between brain regions. It is a time-varying neuronal program which dictates how to move from world states s to actions a via the hard-max policy (6). The approximating Q-value function would tell the DMN the (expected) usefulness of taking an action a in state s. The DMN, and in particular its dmPFC node, could then contribute to the choice, at a given state s, of an action a which maximizes these approximate Q-values. This mapping from states to actions that is conventionally called policy matrix (Mnih et al., 2015; Silver et al., 2016). Learning consists in starting from a given table and updating it during action choices, which take the agent to different table entries.
An additional layer of learning concerns the addition of new entries in the table (i.e., state and action). The framing of the policy matrix is not described here but could be supported by synaptic epigenesis (Gisiger et al., 2005). Indeed, the tuning of synaptic weights through learning can stabilize additional patterns of activity by creating new attractors in the neural dynamics landscape (Takeuchi et al., 2014). Those attractors will then constrain both the number of factors taken into account by decision processes and the number of behaviors at reach by the agent (Wang, 2008).
3.2.3 Self-training and the loss function
Successful learning in brains and computer algorithms is not possible without a defined learning goal —the loss function. The action a chosen in state s according to the policy matrix defined in (6) yields a reward r collected by the agent, after which the environment transitions to a new state . One such cycle yields a new experience e = (s, a, r, s′). Each cycle represents a behavior unit of the agent and is recorded in replay memory buffer —which we hypothesize to be subserved by the HC —, possibly discarding the oldest entries to make space: . At time step k, the agent seeks an update θk ← θk−1 + δθk of the parameters for its approximate model of the Q-value function. This warrants a learning process and definition of a loss function. The Bellman equation (4) provides a way to obtain such a loss function (9) as we outline in the following. Experience replay consists in sampling batches of experiences from the replay memory . The agent then tries to approximate the would-be Q-value for the state-action pair (s, a) as predicted by the Bellman equation (4), namely with the prediction of a parametrized regression model . From a neurobiological perspective, experience replay can be manifested as the re-occurrence of neuron spiking sequences that have also occurred during specific prior actions and environmental states. The HC is a strong candidate to contribute to such a mechanism because neuroscience experiments have repeatedly indicated in rats, mice, cats, rabbits, songbirds, and monkeys (Buhry et al., 2011; Nokia et al., 2010; Dave and Margoliash, 2000; Skaggs et al., 2007).
At the current step k, computing an optimal parameter update then corresponds to finding the model parameters θk which minimize the following mean-squared loss function where yk is defined in (4). A recently proposed, practically successful alternative approach (Mnih et al., 2015; Silver et al., 2016) is to learn this representation using a deep neural-network model, leading to the so-called deep Q-learning family of methods which likely are state-of-the-art in RL. The set of model parameters θ that instantiate the non-linear interactions between layers of the artifical neural network may find a neurobiological correspondence in the adaptive strengths of axonal connections between neurons from the different levels of the neural processing hierarchy (Mesulam, 1998; Taylor et al., 2015).
3.2.4 Optimal control via stochastic gradient descent in the DMN
Efficient learning of the entire set of model parameters can effectively be achieved via stochastic gradient descent, a universal algorithm for finding local minima based on the first derivative of the optimization objective. Stochastic here means that the true gradient is estimated from batches of training examples samples, which, in our case, corresponds to blocks of experience from the replay memory: where the positive constants α1, α2,… are learning rates. Thus, the next action is taken to drive reward prediction errors to percolate from lower to higher processing layers to modulate the choice of future actions. It is a standard result that under special conditions on the learning rates αk –namely that the learning rates are neither too large nor too small, or more precisely that the sum diverges while the thus generated approximating sequence of Q-value functions are attracted and absorbed by the optimal Q-value function Q* defined implicitly by the Bellman equation (4).
3.2.5 Does the hippocampus subserve MCMC sampling?
In RL, MCMC simulation is a common means to update the agent’s belief state (Silver and Veness, 2010). MCMC simulation provides a simple method for evaluating the value of a state. They provide an effective mechanism both for tree search (of the considered action trajectories) and for belief state updates, breaking the curse of dimensionality and allowing much greater scalability than an RL agent without stochastic resampling procedures. Such methods have scaling as a function of available data (i.e., sample complexity) that is determined only by the underlying difficulty of the MDP, rather than the size of the state space or observation space, which can be prohibitively large.
In the human brain, the HC could contribute to synthesizing imagined sequences of world states, actions and rewards (Aronov et al., 2017; Chao et al., 2017; Boyer, 2008). These simulations of experience batches would be used to update the value function, without ever looking inside the black box describing the model’s dynamics (De Lavilléon et al., 2015). This would be a simple control algorithm by evaluating all legal actions and selecting the action with highest expected cumulative rewards. In MDPs, MCMC simulation provides an effective mechanism both for tree search and for belief-based state updates, breaking the curse of dimensionality and allowing much greater scalability than has previously been possible (Silver et al., 2016).
3.3 Putting everything together
The DMN is today known to consistently increase in neural activity when humans engage in cognitive processes that are detached from the current sensory environment and this network was proposed to be situated at the top of the brain network hierarchy (Carhart-Harris and Friston, 2010; Margulies et al., 2016b). Its putative involvement in thinking about the past, hypothetical experiences, and the future appears to tie in with the implicit computation of action and state cascades as a function of what happened in the past. A policy matrix encapsulates the repertoire of possible actions on the world given a current state. The policy matrix encodes the probabilites of choosing actions to be executed in a certain situation. The DMN may subserve constant exploration of possible action trajectories and nested estimaton of their cumulative reward outcomes. Implicit computation of future choices provides an explanation for the evolutionary emergence and practical usefulness of mind-wandering at day-time and dreams during sleep in humans.
The HC may contribute to generation of perturbed action-transition-state-reward samples as batches of pseudo-experience (i.e., imagined, hypothesized, and recalled mental scenarios). The small variations in these experience samplings allow searching a larger space of model parameters and possible experiences. Taken to its extreme, stochastic recombination of experience building blocks can further optimize the behavior of the RL agent by model learning from scenarios in the environment that the agent might only very rarely or never encounter. An explanation is thus offered for experiencing seemingly familiar situations that a human has however never actually encountered (i.e., déjà vu effect). While such a situation may not have been experienced in the physical world, the DMN may have previously stochastically generated, evaluated, and adapted to such a randomly synthesized situation. In the absence of environmental input and feedback (e.g., mind-wandering or sleep), mental scene construction allows for pseudo-experiencing possible future scenarios and action outcomes. Our formal account of DMN function thus acknowledges the unavoidable stochasticity of computation in neural systems (Faisal et al., 2008).
From the perspective of a model-free RL agent, inference in the human brain reduces to generalization of policy and value computations from sampled experiences to successful action choices and reward predictions in future states. As such, plasticity in the DMN arises naturally. If an agent behaving optimally in a certain environment moves to new, yet unexperienced environment, reward prediction errors will largely increase. This feedback will lead to adaptation of policy considerations and value estimations until the intelligent system converges to a new steady state of optimal action decisions in a volatile world.
4 Relation to existing accounts
4.1 Predictive coding hypothesis
Predictive coding mechanisms (Clark, 2013; Friston, 2008) are a frequently evoked idea in the context of default mode function (Bar et al., 2007). Cortical responses are explained as emerging from continuous functional interaction between higher and lower levels of the neural processing hierarchy. Feed-forward sensory processing is constantly calibrated by top-down modulation from more multi-sensory and associative brain regions further away from primary sensory cortical regions. The dynamic interplay between cortical processing levels may enable learning about world aspects by reconciling gaps between fresh sensory input and expectations computed based on stored prior information. At each stage of neural processing, an internally generated prediction of aspects of environmental sensations is directly compared against the actual environmental input. A prediction error at one of the processing levels incurs plasticity changes of neuronal projections (i.e., adapting model parameters) to allow for gradually improved future prediction of the environment. In this way, the predictive coding hypothesis offers explanations for the constructive, non-deterministic nature of sensory perception (Friston, 2010; Buzsáki, 2006) and the intimate relation of motor action to sensory expectations (Wolpert et al., 1995; Körding and Wolpert, 2004). Contextual integration of sensorimotor perception-action cycles may be maintained by top-down modulation using a-priori information about the environment.
In short, predictive coding processes conceptualize updates of the internal representation of the environment to best accommodate and prepare the organism for processing the constant influx of sensory stimulation and performing action on the environment. There are hence a number of common properties between the predictive coding account and the proposed formal account of DMN function based on MDPs. Importantly, a generative model of how perceived sensory cues arise in the world would be incorporated into the current neuronal wiring. Further, both functional accounts are plausibilized by neuroscientific evidence that suggest the human brain to be a “statistical organ” (Friston et al., 2014) with the biological purpose to generalize from the past to new experiences. Neuroanatomically, axonal back projections indeed outnumber by far the axonal input projections existing in the monkey and probably also human brain (Salin and Bullier, 1995). These many and diverse modulatory influences from higher onto downstream cortical areas can inject prior knowledge at every stage of processing environmental information. Moreover, both accounts provide a parsimonious explanation why the human brain’s processing load devoted to incoming information decreases when the environment becomes predictable. This is because the internal generative model only requires updates after discrepancies have occurred between environmental reality and its internally reinstantiated representation. Increased computation resources are however allocated when unknown stimuli or unexpected events are encountered by the organism. The predictive coding and MDP account hence naturally evoke a mechanism of brain plasticity in that neuronal wiring gets increasingly adapted when faced by unanticipated environmental challenges.
While sensory experience is a constructive process from both views, the predictive coding account frames sensory perception of the external world as a generative experience due to the modulatory top-down influence at various stages of sensory input processing. This generative top-down design is replaced in our MDP view of the DMN by a sequential decision-making framework. Further, the hierarchical processing aspect from predictive coding is re-expressed in our account in form of nested prediction of probable upcoming actions, states, and outcomes. While both accounts capture the consequences of action, the predictive coding account is typically explained without explicit parameterization of the agent’s time horizon and has a tendency to be presented as emphasizing prediction about the immediate future. In the present account, the horizon of that look into the future is made explicit in the γ parameter of the Bellman equation. Finally, the process of adapting the neuronal connections for improved top-down modulation takes the concrete form of stochastic gradient computation and back-propagation in our MDP implementation. It is however important to note that the neurobiological plausibility of the back-propagation procedure is controversal (Goodfellow et al., 2016).
In sum, recasting DMN function in terms of MDPs therefore naturally incorporates a majority of aspects from the prediction coding hypothesis. The present MDP account of DMN function may therefore serve as a concrete implementation of predictive coding ideas from cognitive neuroscience. MDPs have the advantage of exposing an explicit mechanism for the horizon of future considerations and for how the internal representation of the world is updated, as well as why certain predictions may be more relevant to the agent than others.
4.2 Semantic hypothesis
Another frequently proposed cognitive account to explain DMN function revolves around forming logical associations and abstract analogies between experiences and conceptual knowledge derived from past behavior (Bar, 2007; Binder et al., 1999; Constantinescu et al., 2016). Analogies might naturally tie incoming new sensory stimuli to explicit world knowledge (i.e., semantics) (Bar, 2009). The encoding of complex environmental features could thus be facilitated by association to known similar states. Going beyond isolated meaning and concepts extracted from the world, semantic building blocks may need to get recombined to enable mental imagery of non-existing scenarios. As such, semantic knowledge would be a prerequisite for optimizing behavior by constantly simulating possible future scenarios (Boyer, 2008; Binder and Desai, 2011). Such cognitive processes can afford the internal construction and elaboration of necessary information that is not presented in the surrounding sensory environment by recombining building blocks of concept knowledge and episodic memories (Hassabis and Maguire, 2009). Indeed, in aging humans, remembering the past and imaging the future equally decreased in the level of detail and were associated with concurrent deficits in forming and integrating relationships between items (Addis et al., 2008; Spreng and Levine, 2006). Further, episodic memory, language, problem solving, planning, estimating other others’ thoughts, and spatial navigation represent neural processes that are likely to build on abstract world knowledge and logical associations for integrating the constituent elements in rich and coherent mental scenes (Schacter et al., 2007). Such scene construction processes could contribute to interpreting the present and foretelling the future. Further, mental scene construction has been proposed to imply a distinction between engagement in the sensory environment and internally generated mind-wandering (Buckner and Carroll, 2007). These investigators stated that “A computational model […] will probably require a form of regulation by which perception of the current world is suppressed while simulation of possible alternatives are constructed, followed by a return to perception of the present.”.
In comparison, both the semantic hypothesis and the present formal account based on MDPs expose mechanisms of how action considerations could be mentally explored. In both accounts, there is also no reason to assume that predictions of various levels of complexity, abstraction, timescale, and purpose rely on mechanisms that are qualitatively different. This concurs with DMN activity increases across time, space, and content domains demonstrated in many neuroimaging studies (Spreng et al., 2009; Laird et al., 2009; Bzdok et al., 2012; Binder et al., 2009). Further, the semantic hypothesis and MDP account provide explanations why HC damage does not only impair recalling memories, but also hypothetical and future thinking (Hassabis et al., 2007). While both semantic hypothesis and our formal account propose memory-based internally generated information for probabilistic mental models of action outcomes, MDPs render explicit the grounds on which an action is eventually chosen, namely, the estimated cumulative reward. In contrast to many versions of the semantic hypothesis, the MDPs naturally integrate the egocentric view (more related to current action, state, and reward) and the world view (more related to past and future actions, states, and rewards) on the world in a same optimization problem. Finally, the semantic account of DMN function does not offer a mechanistic explanation how explicit world knowledge and semantic analogies thereof lead to prediction of future actions and states. The semantic hypothesis does also not explain why memory recall for scene construction in humans is typically fragmentary and noisy instead of accurate and reliable. In contrast to existing accounts on semantics and mental scene construction, the random and creative aspects of DMN function are explained in MDPs by the advantages of stochastic optimization. Our MDP account provides an algorithmic explanation in that stochasticity of the parameter space exploration by MCMC approximation achieves better fine-tuning of the action policies and estimation of expected reward outcomes. That is, the purposeful stochasticity of policy and value estimation in MDPs provides a candidate explanation for why humans have evolved imperfect noisy memories as the more advantageous adaptation. In sum, mental scene construction according to the semantic account is lacking an explicit time and incentive model, both of which are integral parts of the MDP interpretation of DMN function.
4.3 Sentinel hypothesis
The DMN regions have been associated with processing the experienced or expected relevance of environment cues (Montague et al., 2006). Processing self-relevant information was perhaps the first cognitive account that was proposed for DMN function (Gusnard et al., 2001; Raichle et al., 2001). Since then, many investigators have speculated that neural activity in the DMN may reflect the brain’s continuous tracking of relevance in the environment, such as spotting predators, as an advantageous evolutionary adaptation (Buckner et al., 2008; Hahn et al., 2007). According to this cognitive account, the human brain’s baseline maintains a “radar” function to detect subjectively relevant cues and unexpected events in the environment. Propositions of a sentinel function to underlie DMN activity have however seldom detailed the mechanisms of how attention and memory resources are exactly reallocated when encountering a self-relevant environmental stimulus. However, in the present MDP account, promising action trajectories are recursively explored by the human DMN. Conversely, certain branches of candidate action trajectories are detected to be less worthy to become mentally explored. This mechanism,expressed by the Bellman equation, directly implies stratified allocation of attention and working memory load over relevant cues and events in the environment. Further, our account provides a parsimonious explanation for the consistently observed DMN implication in certain goal-directed experimental tasks and in task-unconstrained mind-wandering (Smith et al., 2009; Bzdok et al., 2016b). Both environment-detached and environment-engaged cognitive processes may entail DMN recruitment if real or imagined experience is processed, manipulated, and used for predictions. During tasks, the policy and value estimates may be updated to optimize especially short-term action. At rest, these parameter updates may improve especially mid-and long-term action. This horizon of the agent is expressed in the γ parameter in the MDP account. We thus provide answers for the currently unsettled question why the involvement of the same neurobiological brain circuit (i.e., DMN) has been documented for specific task performances and baseline house-keeping functions.
In particular, environmental stimuli especially important for humans are frequently of social nature. This is probably unsurprising given that the complexity of the social systems is a likely to be a human-defining property (Tomasello, 2009). According to the “social brain hypothesis”, the human brain has especially been shaped for forming and maintaining increasingly complex social systems, which allows solving ecological problems by means of social relationships (Whiten and Byrne, 1988). Indeed, social topics amounted to roughly two thirds of human everyday communication (Dunbar et al., 1997), while mind-wandering at daytime and dreams during sleep are rich in stories about people and the complex relationships between them. In line with this, the DMN was argued to be specialized in continuous processing of social information as a physiological baseline of human brain function (Schilbach et al., 2008). This view was later challenged by observing analogues of the DMN in monkeys (Mantini et al., 2011), cats (Popa et al., 2009), and rats (Lu et al., 2012), three species with social-cognitive capacities that are supposedly less advanced than in humans.
Further, the principal connectivity gradient in the cortex appears to be greatly expanded in humans compared to monkeys, suggesting a phylogenetically conserved axis of cortical expansion with the DMN emerging at the extreme end in humans (Margulies et al., 2016a). Neurocomputational models of dyadic whole-brain dynamics demonstrated how the human connectivity topology, on top of facilitating processing at the intra-individual level, can explain our propensity to coordinate through sensorimotor loops with others at the inter-individual level (Dumas et al., 2012). The DMN is moreover largely overlapping with neural networks associated with higher level social cognition (Schilbach et al., 2012). For instance, the vmPFC, PMC, and RTPJ together play a key role in bridging the gap between self and other by integrating low-level embodied processes within higher level inference-based mentalizing (Lombardo et al., 2009).
Rather than functional specificity for processing social information, the present MDP account can parsimoniously incorporate the dominance of social content in human mental activity as high value function estimates for information about humans (Baker et al., 2009; Kampe et al., 2001; Krienen et al., 2010). The DMN may thus modulate reward processing in the human agent in a way that prioritizes appraisal of and action towards social contexts, without excluding relevance of environmental cues of the physical world. In sum, our account on the DMN directly implies its previously proposed “sentinel” function of monitoring the environment for self-relevant information in general and inherently accommodates the importance of social environmental cues as a special case.
4.4 The free-energy principle and active inference
According to theories of the free-energy principle (FEP) and active inference (Friston, 2010; Friston et al., 2009), the brain corresponds to a biomechanical reasoning engine. It is dedicated to minimizing the long-term average of surprise: the log-likelihood of the observed sensory input –more precisely, an upper bound thereof– relative to the expectations about the external world derived from internal representations. The brain would continuously generate hypothetical explanations of the world and predict its sensory input. Precursors of this interpretation can be traced back to Dayan and colleagues (Dayan et al., 1995) who introduced the Helmholtz machine, a hierarchical factorial directional deep belief network. According to the FEP account, the goal of the brain is to optimize over a generative model G of sensations: to iteratively modify its internal representation pG(z|x) of objects in the world, their interactions and dynamics. The model then allows to minimize surprise when these representations are confronted with sensory input x during action-perception cycles (i.e., the generative model). The FEP also proposes a companion model called the recognition model, which works in tandem with the generative model pR(z|x) to accomplish approximate inference. Put differently, the recognition model dreams imaginary worlds z, while the generative model tries to produce sensory sensations x which match these imagined mental scenarios. Howewer, surprise is challenging to optimize numerically because we need to sum over all hidden causes of the sensations. Instead, FEP therefore minimizes an upper-bound of surprise, namely the free energy given by
Despite its popularity, criticism against the FEP has arisen over the years, some of which outlined in the following. The main algorithm for minimizing free energy is the wake-sleep algorithm (Dayan et al., 1995). As these authors noted, a crucial drawback of the wake-sleep algorithm is that it involves a pair of forward (generation) and backward (recognition) models that together do not correspond to optimization of (a bound of) the marginal likelihood. The brain may therefore be unlikely to implement a variant of the wake-sleep algorithm. The recent theory of variational auto-encoders (VAEs) (Kingma and Welling, 2013) might provide an efficient alternative to the wake-sleep algorithm. VAEs overcome a number of the technical limits of the wake-sleep algorithm by using a reparametrization trick. For instance, unlike the wake-sleep algorithm for minimizing free energy, VAEs can be efficiently trained via back-propagation of prediction errors.
On another front, since theories based on the FEP (Friston, 2010; Friston et al., 2009) conceptualize ongoing behavior in an organism to be geared towards the surprise-minimizing goal, an organism entering a dark room (Fig. 5) would strive to remain in this location because its sensory inputs are perfectly predictable given the environmental state (Friston et al., 2012). However, such a behavioral tendency is seldom observed in animals or humans in the real world: In a dark room, intelligent agents would search for light sources to explore its surroundings or leave it. Defenders of the FEP have retorted by advancing the “full package” (Friston et al., 2012): FEP is proposed to be multi-scale and there would be a meta-scale at which the organism would be surprised by such a lack of surprise. According to this argument, a dark room would paradoxically correspond to a state of particularly high surprise. Driven by surprise-minimization objective, the FEP agent would eventually bootstrap itself out of such saddle points to explore more interesting parts of the environment. In contrast, an organism operating under our RL-based theory would inevitably identify the sensory-stimulus-deprived room as a local minimum. Indeed, hippocampal experience replay (see 3.2.3) would serve to sample memories or fantasies of alternative situations with reward structure. Such artificially generated internal sensory input subserved by the DMN can entice the organism to explore the room, for instance by looking for and using the light switch or simply finding the room exit.
Finally, we note that FEP and active inference can be reframed in terms of our model-free RL framework. This becomes possible by recasting the Q-value function (i.e expected long-term reward) maximized by the DMN to correspond to negative surprise, that is, the log-likehood of current sensory priors the agent has about the world. More explicitly, this corresponds to
Such a surprise-guided reinforcement learning scheme has previously been advocated under the equivalent framework of information compression (Schmidhuber, 2010; Mohamed and Rezende, 2015). Nevertheless, minimization of surprise quantities alone may be insufficient to explain the diversity of behaviors observed in humans and other intelligent animals.
5 Conclusion
Which brain function could be important enough for the existence and survival of the human species to justify constantly high energy costs? MDPs motivate an attractive formal account how the human association cortex might implement multi-sensory representation and control of the environment to optimize the organism’s interaction with the world. This idealized process model explains a number of previous experimental observations in the DMN by simple but non-trivial mechanisms. From the view of a Markovian sequential decision process, human behavior unfolds by integrating action outcomes from stored past events and extrapolation to upcoming events for guiding action choice in the present context. MDPs also provide a mathematical formalism how opportunity in the environment can be recursively exploited when confronted with challenging decisions. This functional interpretation may be well compatible with the DMN’s poorly understood involvement across autobiographical memory recall, problem solving, abstract reasoning, social cognition, as well as delay discounting and self-related prospection. Improvement of the internal world representation by injecting stochasticity into the recall of past actions and the estimation of action outcomes may explain why highly accurate memories have been disfavored in human evolution and why human creativity might be adaptive.
It is an important feature of the proposed artificial intelligence perspective on DMN biology that it is practically computable and yields falsifiable neuroscientific hypotheses. Neuroscience experiments could be designed that operationalize the set of action, value, and state variables that govern the behavior of intelligent RL agents. At the least, we propose an alternative vocabulary to describe, contextualize, and interpret experimental findings in neuroscience studies on the DMN. Ultimately, DMN activity may instantiate a holistic integration ranging from real experience over purposeful dreams to anticipated futures for continued refinement of the organism’s fate.
Footnotes
↵1 Random as it depends both on the environment’s dynamic and the policy π being played (which can be stochastic).