Abstract
While forming and updating beliefs about future life outcomes, people tend to consider good news and to disregard bad news. This tendency is supposed to support the optimism bias. Whether learning bias is specific to “high-level” abstract belief update or a particular expression of a more general “low-level” reinforcement learning process is unknown. Here we report evidence in favor of the second hypothesis. In a simple instrumental learning task, participants incorporated worse-than-expected outcomes at a lower rate compared to better-than-expected ones. This asymmetry was correlated across subjects with standard measure of dispositional optimism. Multimodal imaging indicated that inter-individual variability in the expression of asymmetric update relies on the dorsomedial prefrontal cortex at both morphological and functional levels. Our results constitute a new step in the understanding of the genesis of optimism bias at the neurocomputational level.
Introduction
“It is the peculiar and perpetual error of the human understanding to be more moved and excited by affirmatives than negatives; whereas it ought properly to hold itself indifferently disposed towards both alike” (p. 36)a.
People typically underestimate the likelihood of negative events and overestimate the likelihood of positive events. This cognitive trait in (healthy) humans is known as the optimism bias and has been repeatedly evidenced in many different guises and populations (Shepperd et al., 2013, 2015; Weinstein, 1980) such as students projecting their salary after graduation (Shepperd et al., 1996), women estimating their risk of getting breast cancer (Waters et al., 2011) or heavy smokers assessing their risk of premature mortality (Schoenbaum, 1997). One mechanism hypothesized to underlie this phenomenon is an asymmetry in belief updating, colloquially referred to as “the good news / bad news effect” (Eil and Rao, 2011; Sharot et al., 2011). Indeed, preferentially revising one’s beliefs when provided with favorable compared to unfavorable information constitutes a learning bias which could, in principle, generates and sustains an overestimation of the likelihood of desired events and a concomitant underestimation of the likelihood of undesired events (optimism bias) (Sharot and Garrett, 2016).
This good news/bad news effect has recently been demonstrated in the case where outcomes are hypothetical future prospects associated with a strong a priori desirability or undesirability (estimation of post-graduation salary or the probability of getting cancer) (Shepperd et al., 1996; Waters et al., 2011). In this experimental context, belief formation triggers complex interactions between episodic, affective and executive cognitive functions (Eil and Rao, 2011; Sharot et al., 2011, 2007), and belief updating relies on a learning process involving abstract probabilistic information (Garrett et al., 2014; Moutsiana et al., 2015, 2013; Sharot et al., 2011). However, it remains unclear whether this learning asymmetry also applies to immediate reinforcement events driving instrumental learning directed to affectively neutral options (i.e. with no a priori desirability or undesirability). If an asymmetric update is also found in a task involving neutral items and direct feedbacks, then the good news/bad news effect could be considered as a specific – cognitive – manifestation of a general reinforcement learning asymmetry. If the asymmetry is not found at the basic reinforcement learning level, this would mean that the asymmetry is specific of abstract belief updating, and this would require a theory explaining this discrepancy.
To arbitrate between these two alternative possibilities, we analyzed instrumental behavior of subjects performing a simple two-armed bandit task, involving neutral stimuli and actual and immediate monetary outcomes, with two learning models. The first model (a standard RL algorithm) confounded individual learning rates for positive and negative feedbacks and the second one differentiated them, potentially accounting for learning asymmetries.
Over two experiments, we found subjects’ behavior was better explained by the asymmetric model, with an overall difference in learning rates consistent with preferential learning from positive, compared to negative, prediction errors. Furthermore, this tendency of using the asymmetric model covaried with optimistic trait as measured with standard psychometric scale (LOT-R). The good news/bad news effect can then be considered as a particular consequence of a more general asymmetry in reinforcement learning, accounting for the optimism bias.
Our and previous studies suggest that the good news/bad news effect is highly variable across subjects. Behavioral differences in optimistic beliefs and optimistic update have been shown to be reflected by differences in brain activation in the prefrontal cortex (Sharot et al., 2011). However the question remains, whether or not such interindividual differences are related to specific anatomical, supposedly developmental, variability as revealed by voxel-based morphometry (VBM). Our imaging results indicate that the interindividual variability in the tendency in optimistic learning correlates with grey matter density in the dorsal prefrontal cortices. Finally, our fMRI results confirmed at the functional level the assumptions of our computational model as well as the VBM results.
Results
Behavioral task and dependent variables
Healthy subjects performed a probabilistic instrumental learning task with monetary feedbacks, previously used in brain imaging, pharmacological and clinical studies13-15 (Figure 1A). In this task, options (abstract cues) were presented in fixed pairs (i.e. conditions). In all conditions each cue was associated with a stationary probability of reward. In asymmetric conditions, the two reward probabilities differed between cues (25/75%). From asymmetric conditions we extracted the rate of “correct” response (selection of the best option) as a measure of performance (Figure 1B, left). In symmetric conditions, both cues had the same reward probabilities (25/25% or 75/75%), such that there was no intrinsic “correct response”. In symmetric conditions we extracted for each subject and each symmetric pair, a “preferred response” rate, defined as the choice rate of the option most frequently selected by a given subject (i.e.by definition in more than 50% of trials). The preferred response rate in the 25/25% conditions should be taken as a measure of the tendency to overestimate the value of one instrumental cue compared to the other, in absence of actual outcome-based evidence. (Figure 1B, right). In a first experiment (N=50) that subjects performed while being fMRI scanned, the task involved reward (+0.5€) and reward omission (0.0€), as the best and worst outcome respectively. In a second purely behavioral experiment (N=35), the task involved reward (+0.5€) and punishment (-0.5€), as the best and worst outcome respectively. In addition to performing the instrumental learning task, subjects of this second experiment filled the Life Orientation Test – Revised (LOT-R), a standard assessment of optimistic trait.
Computational models
We fitted the behavioral data with two reinforcement-learning models (Sutton and Barto, 1998). The “reference” model was represented by a standard Rescorla-Wagner (Rescorla and Wagner, 1972), thereafter referred to as RW model. The RW model learns option values by minimizing reward perdition errors. It uses a single learning rate (alpha: a) to learn from positive and negative prediction errors. The “target” model was represented by a modified version of the RW model, thereafter referred to as RW± model. In the RW± model, learning from positive and negative prediction errors is governed by different learning rates (alpha: a+ and alpha: a-respectively). For a+ > a- the RW± model instantiates optimistic reinforcement learning (i.e. the good news/bad news effect); for a+ = a-, the RW± instantiates realistic (or unbiased) reinforcement learning, just as in the RW model (the RW model is thus nested in the RW± model); finally for a+ < a- the RW± instantiates pessimistic reinforcement learning. In both models the choices are taken by feeding the option values into a softmax decision rule, whose exploration/exploitation trade-off is governed by a parameter (b).
Model comparison and model parameters analysis
We implemented Bayesian model comparison to establish which model better accounted for the behavioral data. For each model we estimated the optimal free parameters by maximization the likelihood of the participants’ choices, given the models and sets of parameters. For each model and each subject, we calculated the Bayesian Information Criterion (BIC) by penalizing the maximum likelihood with the number of free parameters in the model. Group-level BIC analysis indicated that the RW± model better explains the behavioral data compared to the RW model (BICrw=99.4±4.4, BICrw±=93.6±4.7; t(49)= 2.9, p=0.006, paired t-test), even accounting for its additional degrees of freedom (Table 1). RW± being the best fitting model we compared the learning rates fitted for positive (good news: a+) and negative (bad news: a-) prediction errors. We found a+ significantly higher compared to a- (t(49)= 3.8, p<0.001 paired t-test). To summarize, model comparison indicated that, in our simple instrumental learning task, the best fitting model is the model with different learning rates for learning from positive and negative predictions errors (RW±). Crucially, learning rates comparison indicated that instrumental values are preferentially updated following positive prediction errors, which is consistent with an optimistic bias operating when learning from immediate feedback (optimistic reinforcement learning).
Computational characterization of inter-individual variability
To explore inter-individual variability, we computed for each subject the between-model BIC difference (DBIC=BICRW - BICRW±). The ΔBIC quantifies at the individual level the goodness of fit improvement moving from the RW to the RW± models, hence approximates a measure of optimism. Subjects with a positive DBIC (N=25, in the first experiment) are subjects whose behavior is better explained by the RW± model (thereafter refereed as RW± subjects); subjects with a negative DBIC (N=25, in the first experiment) are subjects whose behavior is better explained by the RW model (thereafter refereed as RW subjects) (Figure 2A). Importantly the maximum likelihood of reference model (RW) was not different between the two groups of subjects, indicating similar baseline quality of fit (t(48)= -1.0, p=0.314, two-sample t-test).
To characterize the computational profiles of these two groups of subjects and the validity of the ΔBIC as a measure of optimism, we analyzed and compared their free parameters (a+ and a-and 1/ß) (figure 2C). Learning rates fitted with the RW± model entered a two-way ANOVA with group (RW and RW±) and learning rates type (a+ and a-) as respectively between-and within-subjects factors. The AnOVA showed a main effect of learning rate type (F(1,48)=16.5, P<0.001) with a+ higher than a-. We also found a main effect of group (F(1,48)=10.48, P=0.002) and a significant group x learning type interaction (F(1,48)= 7.8, P=0.007). Post-hoc tests revealed that average learning rates for positive prediction errors were not different among the two groups, a+RW=0.45 ± 0.08 and a+RW±= 0.27 ± 0.06 (t(48) = 1.7, p=0.086, two-sample t-test). On the contrary, average learning rates for negative prediction errors were significantly different between groups a-RW= 0.41 ± 0.08 and a- RW±= 0.04 ± 0.02 (t(48)= 4.6, p<0.001, two-sample t-test). In addition, an asymmetry in learning rates was detected within the RW± group, where a+ was higher than a- (t(24)=5.1, p<0.001, paired t-test) but not within RW group (t(24)=0.9, p=0.399, paired t-test). Thus, RW± subjects specifically drove the learning rates asymmetry found in whole population. On the other side the RW subject display “realistic” (as opposed to “optimistic”) instrumental learning (Figure 2B and 2C).
Interestingly, the choice randomness (captured by the 1/ß, “temperature” parameter) was also found to be significantly different between the two groups of subject, 1/ßRW=0.20 ± 0.05 and 1/ßRW±=0.06 ± 0.01. (t(48)= 2.9, p=0.006, two-sample t-test). This suggests that optimistic reinforcement learning, observed in RW± subjects, is also associated with exploitative, as opposite to explorative, behavior (Figure 2C). Finally, the ΔBIC (our classification variable) was found significantly correlated significantly with the normalized learning rate asymmetry ([a+ - a-]/[a+ + a-]; R=0.5455, p<0.001) and with the temperature (R=-0.3343, p=0.0177). To summarize, RW± subjects tend to weight more positive feedbacks and, as a consequence, to exploit more consistently the previously rewarded options (optimism). Both computational features of this optimistic computational phenotype are quantitatively captured by the ΔBIC index.
Behavioral signature distinguishing optimistic from realistic subjects
In order to analyze the behavioral consequences of optimistic, as opposed to realistic, learning and to confirm our model-based results with model-free behavioral observations, we compared the task’s dependent variables between our two groups of subjects (Figure 2D, Table 2). Correct response rate did not differ between groups (t(48)=-0.7323, p=0.467, two-sample t-tests). However, the preferred response rate in the 25/25% condition was significantly higher for RW± group in comparison to RW group (t(48)= -3.4, p=0.001, two-sample t-test).
In order to validate the ability of RW± model to capture this difference, we performed simulations using both models and submitted them to the same statistical analysis as actual choices (Figure 2D). The RW± model simulated preferred response rate was significantly higher for RW± group compared to the RW group (t(48)= -5.4496, p<0.001, two-sample t-test), which replicated human behavior. However, the simulated preferred response rates from the RW model were similar in the two groups (t(48)= -0.6, p=0.566, two-sample t-test), which departed from our observations in real subjects. We found that a higher preferred response rate in the 25/25% condition was a specific signature of optimistic learning: in poorly rewarding environment (and where there is no intrinsic correct response), optimistic subjects tend to overestimate the value of one of the two options (Supplementary Figure 1). Finally, we found that the preferred response rate in the 25/25% condition was significantly correlated with our classification computational variable ΔBIC (R=0.6366, p<0.001). The preferred response rate thus provides a model-free signature of optimistic reinforcement learning that is congruent with our computational analysis: the preferred response rate was higher in RW± group in comparison to RW group and only simulations realized with RW± model were able to replicate this pattern of responses.
Anatomical signature distinguishing optimistic from realistic subjects
To investigate the neuroanatomical basis of the interindividual computational variability observed in our task, we used voxel-based morphometry (VBM). We devised a multiple regression model with our computational classification variable ΔBIC as a continuous variable of interest. We used theΔBIC since it captures two the computational features distinguishing the RW± from the RW subjects: the learning rate asymmetry and the higher tendency to explore. This analysis showed a significant positive correlation between the ΔBIC and grey matter density in only two brain regions; the dorsolateral and dorsomedial prefrontal cortices (DLPFC and the DMPFC; Figure 3A and B). This result implies that the better the behavioral fit by the RW± (optimistic learning), the higher is the grey matter density in DLPFC and DMPFC. Moreover, in order to validate this model-based result with a model-free analysis, we also analyzed the correlation between the grey matter density in these regions and preferred response rate in the 25/25% condition. We found significant correlations in both the DMPFC and the DLPFC (respectively R=0.45, p<0.001, and R=0.42, p=0.003; Figure 3C and D). Our VBM then indicates that the computational variability observed in our task is linked to structural difference in the dorsal prefrontal cortex (both in the medial and the lateral part). More precisely, the more a subject’s computational strategy diverges from an unbiased - “realistic” - model, the higher is the grey matter density in these areas. These results lend further support, based on biological data, to the computational phenotyping based on the learning task.
Functional signature distinguishing optimistic from realistic subjects
To investigate the functional consequence of the anatomical difference between the RW± and RW subjects (i.e. different grey matter density in in DLPFC/DMPFC), we analyzed the brain activity in these regions, using functional Magnetic Resonance Imagining (fMRI). We devised a general linear model in which we modeled as separated events the choice and the outcome onset, each modulated by different parametric modulators. The choice onset was modulated by the reaction time (RT), which counts as a proxy of decision value and complexity and reflects the decision process(Kolling et al., 2012; Shenhav et al., 2014). The outcome onset was modulated by the outcome obtained (0.5€ or 0.0€) in a given trial. Overall, we found both the DMPFC and the DLPFC as significantly encoding reaction times (t(49)=5.9751, P<0.001 and t(49)=2.0873, P=0.042 respectively, one-sample t-tests). However regression coefficients were not different between the two groups in either region (t(48)=-0.0253, P=0.9799 and t(48)=0.1658, P=0.869 respectively, two-sample t-tests). On the contrary, when all subjects were analyzed together, the activity in the DMPFC and DLPFC did not correlate with the outcome value (t(49)=0.5967, P=0.5535 and t(49)=0.648, P=0.52 respectively, one-sample t-tests). We tested whether the two computational phenotypes differed in the way their DMPFC and DLPFC react to the outcome value. We found a significant difference in the DMPFC (t(48)=-2.4919, p=0.016, two-sample t-test); post-hoc comparison showed that the DMPFC activity covaried positively with the outcome value in the RW± group (t(24)=2.1326, P=0.043, one-sample t-test), but not in the RW group (t(24)=-1.3669, P=0.1843, one-sample t-test). We found a similar trend in DLPFC, but the effect did not reach significance (Figure 4E). It therefore seems that the difference in grey matter density identified with VBM between the RW and the RW± groups has a functional counterpart. More precisely, the two phenotypes differ in that only in the RW± the DMPFC responds to the outcome value. On the other side, decision process related activations (RT) were found to be similar in the two groups.
Optimistic life attitude correlates with optimistic reinforcement learning
In order to validate our “low level” (computational and instrumental) measures of optimistic reinforcement learning in respect to classical “high level” (psychological and attitudinal) measure of optimism, we ran an additional experiment. In this second behavioral experiment (N=35), subjects completed the Life Orientation Test revised (LOT-R) assessing optimism trait, in addition to our task. Importantly the computational and behavioral results obtained in this second experiment fully replicated the behavioral and computational findings reported above (see Supplementary Materials and Supplementary FigureS2). We derived from the LOT-R scale the optimism score that we found correlated with the ΔBIC (our modelbased index of optimistic learning: R=0.3814, p=0.024) (Figure 4A and C). We also found a positive and significant correlation between the LOT-R and the preferred response rate in the 25/25% condition (our model-free signature of optimistic behavior: R=0.3350, p=0.049). Thus, the more the subjects are optimist according the “high level” standard psychological measure of optimism, the more they show “low level” computational and behavioral signatures of the RW± phenotype.
Discussion
We found that, in a simple instrumental learning task involving neutral visual stimuli associated to actual monetary rewards, participants preferentially updated option values following better-than-expected, compared to worse-than-expected, outcomes. This learning asymmetry was replicated in two experiments and proved to be robust across different conditions (see Supplementary Materials).
At the individual level, the learning asymmetry was differentially expressed across subjects. We were able to capture this variability with a computational measure (the ΔBIC), which quantifies the extent to which a subject’s fit improves when moving from the unbiased “realistic” learning model to a biased one. Importantly, this computational metric was strongly related to a (model-free) behavioral signature of optimistic reinforcement learning: the preferred response rate in the poorly rewarding condition.
We further tested the validity of our computational and behavioral assessment of inter-individual variability by confronting it to external (i.e. task-independent) neurobiological and psychometric measures. First, quantitative neuroanatomical imaging indicated that grey matter density in both the DMFPC and the DLPFC positively correlates with both computational and behavioral measures of optimistic reinforcement learning. This relation was found to have specific functional consequences in the DMPFC, in the form of differential neural response to outcome value in realistic compared to optimistic subjects. Finally, computational and behavioral measures of optimistic learning also correlated with a standard measure of dispositional optimism (LOT-R), providing a link between computational and personality traits.
Our results support the hypothesis that the good news/bad news stands as a core psychological process generating and maintaining unrealistic optimism (Eil and Rao, 2011). In addition, our study has the originality of showing that this effect is not specific to probabilistic belief updating, and that the good news/bad news effect can parsimoniously be considered as an amplification of a primary instrumental learning asymmetry. In other terms, following nomenclature recently proposed by Sharot and Garrett, we found that asymmetric update applies to both “estimation errors” and “prediction errors” (Sharot and Garrett, 2016).
The asymmetric model (RW±) included two different learning rates following positive and negative prediction errors and we found the “positive” learning rate higher compared to the “negative” one (Doll et al., 2011; Niv et al., 2015). Distinct learning rates should be taken as supplementary evidence that the systems used to learn from positive and negative prediction errors are dissociable across different brain circuits and areas (Frank et al., 2004; Palminteri et al., 2012). Note that since our first experiment did not implicate losses, but only reward omissions, our results cannot be interpreted as a consequence of loss aversion (Kahneman and Tversky, 1979). In other terms the learning asymmetry is not outcome sign-based, but prediction error sign-based.
However, higher learning rates for positive compared to negative prediction errors was not the only computational metric distinguishing optimistic from realistic subjects. In fact, we also found that optimistic subjects had a greater tendency to exploit previously rewarded option, as opposed to realistic subjects who were more prone to explore both options. Importantly the higher stochasticity of realistic subjects was associated neither with lower performance in the asymmetrical conditions, nor with a lower baseline quality of fit, as measured by the maximum likelihood. This overexploitation tendency was particularly striking in the symmetrical 25/25% condition, in which both options are poorly rewarding compared to the average task reward rate
We found an interesting association between optimistic update and lower propensity to explore. Indeed, the tendency to ignore negative feedback about chosen options was linked to considering the preferred option better than it is, and hence to stick to this preference. A natural link between optimism and such “conservatism” is not new; it can be dated back to Voltaire’s work “Candide ou l’Optimisme”, where the belief of “living in the best of the possible worlds” was consistently associated with a strong rejection and condemnation of progress and explorative behavior. In the word’s of the 18th century philosopher:
“Optimism,” said Cacambo, “What is that?” “Alas!” replied Candide, “It is the obstinacy of maintaining that everything is best when it is worst”b.
Accordingly, optimism bias has been recently recognized as an important psychological factor helping maintain inaction regarding pressing social problems, such as climate changes (Gifford, 2011).
Recent studies investigated the neural implementation of the good news/bad news effect when analyzed in the context of probabilistic belief updating. Decreased belief updating after worst-than-expected information has been associated with a diminished correlation between negative prediction errors and BOLD signal in the right inferior prefrontal gyrus (IFG) (Sharot et al., 2011) and with a stronger white matter connectivity in a system including left IFG and left subcortical regions (Moutsiana et al., 2015). Congruent with those studies indicating prefrontal cortex implication in the good news/bad news effect, we found that the tendency to implement optimistic reinforcement learning was associated with a greater grey matter density in dorso-medial prefrontal cortex (DMPFC) and dorso-lateral prefrontal cortex (DLPFC). Our VBM results therefore suggest that inter-individual anatomical differences – either genetically programmed or induced by interaction with environment during development (Moutsiana et al., 2013)- in the aforementioned region could determine the way people learn from good and bad news. Importantly, as opposed to task-elicited fMRI activations, where interindividual activation differences could be simply derived from differences in the behavioral variable, structural measures are not affected by performing the task and could therefore provide insight into stable neural markers of a trait of interest (Kanai and Rees, 2011).
Structural imaging leaves the question of the functional consequence of inter-individual differences unanswered. To fill this gap, we analyzed fMRI activity within the regions that have been shown to discriminate between optimists and realists at the anatomical level. We analyzed both decision (reaction time) and learning (outcome encoding) related activations. The results confirmed the implication of the DMPFC in the decision process (Shenhav et al., 2013). However, decision-related activity was not different between optimists and realists. On the other side, outcome-related activity in optimistic subjects was found significantly different from zero and significantly higher compared to realistic subjects. These results suggest that, firstly, anatomical differences have functional specific effects and, secondly, that optimistic update (i.e. the good news/bad news effect) is associated with increased outcome representation in the DMPFC (Ullsperger et al., 2014).
We defined subjects as optimists or pessimists, based on their computational phenotype that we extracted from an instrumental learning task analyzed in the framework of reinforcement learning models. To liken our model-based definition to the psychological concept of optimism, we confronted the computational and behavioral variables capturing optimistic reinforcement learning with a standard measure of dispositional optimism: the Life Orientation Test – revised. The LOT-R is the most commonly used self-report measure of dispositional optimism (Glaesmer et al., 2012) and has been accordingly found in previous studies to correlate with behavioral and neural signatures of the good news/bad news effect (Sharot et al., 2011). We found a positive and significant correlation between learning optimistically in our task and being optimistic in real life, meaning that answering to question about the expectation of good things to happen in one’s life is connected to the way one incorporates positive and negative prediction errors in simple reinforcement situations.
An important question is unanswered by our study and remains to be addressed. Whereas our results clearly show an asymmetry in the learning process, we cannot decide whether the learning process itself involves the representational space of values or in that of probabilities. This question is related to the broader debate whether the reinforcement or the Bayesian learning framework better captures learning and decision-making: two views that has been hard to disentangle, because of largely overlapping predictions, both at the behavioral and neural level (Hampton et al., 2006; Lebreton et al., 2015; Mathys et al., 2011).
A legitimate question is why such learning bias survived in the course of evolution? An obvious answer to this question is that being (unrealistically) optimistic is and/or has been, at least in certain conditions, adaptive, meaning that it confers an advantage. Consistent with this idea, in everyday life dispositional optimism (Carver et al., 2010) has been linked for instance to better global emotional well-being, interpersonal relationship or physical health. Optimists are less likely to develop coronary heart disease (Tindle et al., 2009), have broader social network (Macleod and Conway, 2005) and are less subject to distress when facing adversity (Scheier et al., 1994). Such advantages of dispositional optimism could explain, at least in part, the pervasiveness of an optimistic bias in human. Concerning the specific context of optimistic reinforcement learning a recent paper Cazé and al. showed that in certain conditions (low rewarding environments), an agent learning asymmetrically in an optimistic manner (i.e. with a higher learning rate for positive than for negative feedback) objectively outperforms another “unbiased” agent in a simple probabilistic learning task. Thus, before any social, well-being or health consideration, it is normatively advantageous (in certain contingencies) to take more into account positive than negative feedback. Thus a possible explanation for an asymmetric learning system is that the conditions identified by Cazé et al. closely resemble to the statistics of the natural environment that shaped the evolution of our learning system.
Finally, when reasoning about the adaptive value of optimism, a crucial point to take into account is the significant inter-individual variability of unrealistic optimism (Garrett et al., 2014; Moutsiana et al., 2015, 2013; Sharot et al., 2011). As a social animal humans face both private and collective decision-making problems (Raafat et al., 2009). An intriguing possibility is that multiple “sub-optimal” reinforcement learning strategies are maintained in the natural population to ensure an “optimal” learning repertoire, flexible enough to solve at the group-level the value learning and exploration-exploitation tradeoff (Hills et al., 2014). This hypothesis needs to be formally addressed using evolutionary simulations.
To conclude, our findings shed new light on the nature of the good news/bad news effect and therefore on the mechanistic origins of unrealistic optimism. We found that the optimistic learning is “not specific” to high-level belief updating but a particular consequence of a more general “low-level” instrumental learning asymmetry, both anatomically and functionally linked to the dorsomedial prefrontal cortex.
Methods
Subjects
The first dataset (N=50) served as a cohort of healthy control subjects in a previous clinical neuroimaging study (Worbe et al., 2011). The second dataset involved the recruitment of new subjects (N=35). The local ethics committees approved both experiments. All subjects gave written informed consent before inclusion in the study and the study was carried out in accordance with the declaration of Helsinki (1964, revised 2013). In both studies the inclusion criteria were being older than 18 years and having no history of neurologic or psychiatric disorders. In experiments 1 and 2, men / women ratios were 27/23 and 20/15 respectively and the age means 27.1 ± 1.3 and 23.5 ± 0.7 respectively (expressed as mean ± S.E.M). In the first experiment subjects believed that they would be playing for real money, but to avoid discrimination between healthy subjects and patients, the final payoff was rounded up to a fixed amount of 80€ for every participant. In the second experiment subjects were paid the exact amount of money earned in the learning task, plus a fixed amount (average payoff 15.7±7.6€). Table 3 reports the demographic features of the two cohorts.
Behavioral task and analyses
Subjects performed a probabilistic instrumental learning task described previously (Palminteri et al., 2009) (Figure 1A). Briefly, the task involved choosing between two cues that were associated with stationary reward probability (25% or 75%). There were 4 pairs of cues, randomly constituted and assigned to the 4 possible combinations of probabilities (25/25%, 25/75%, 75/25%, and 75/75%). Subjects were encouraged to accumulate as much money as possible and were informed that some cues would result in a win more often than others (the instructions have been published in appendix of the original study (Palminteri et al., 2009)). Subjects were given no explicit information regarding reward probabilities, which they had to learn through trial and error. The positive outcome reward was winning money (+0.50€); the negative outcome was getting nothing (0.0€) in the first experiment and losing money (-0.50€) in the second experiment. Subjects made their choice by pressing left or right response buttons with a left or right hand finger. Two given cues were always presented together, thus forming a fixed pair (choice context).
Regarding payoff, learning mattered only for pairs with unequal probabilities (75/25% and 25/75%). As dependent variable we extracted the correct response rate in asymmetric conditions (i.e. the left response rate for the 75/25% pair and right response rate in the 25/75% pair) (Figure 1B). In symmetrical reward probability conditions, we calculated the so-called “preferred response rate”. The preferred response was defined as the most chosen option, i.e. chosen by the subject more than 50% of the trials. This quantity is therefore, by definition greater that 50%. The analyses focused on the preferred choice rate in the low reward condition (25/25%), where standard models predict greater frequency of negative prediction errors. Behavioral variables were compared within-subjects using paired two-tailed t-test and between-subjects using two-sample two-tailed t-test. Interactions were assessed using ANOVA.
Optimistic trait
In the second experiment, subjects also completed the French version of Life Orientation Test – Revised (LOT-R), which is a classical and the most frequently used (Glaesmer et al., 2012) psychological self report measure of dispositional optimism (Scheier et al., 1994; Trottier et al., 2008). Furthermore, recent studies showed that this scale might also explain part of the inter-individual variability associated with the prefrontal neural signature of the good news/bad news effect (Sharot et al., 2011). The scale includes ten sentences to be graded from 0 to 4 (from total disagreement to total agreement) and gives a score from pessimistic to optimistic. Subjects were then ranked according their scoring as previously described and their ranking was used to assess the correlation of individual scores with behavioral and computational measures of optimistic behavior (Sharot et al., 2007). We nonetheless note that the same analysis with raw scores lead to similar results.
Computational models
We fitted the data with reinforcement learning models. The model space included a standard Rescorla-Wager model (or Q-leanring) (Rescorla and Wagner, 1972; Sutton and Barto, 1998) (thereafter referred to as RW) and a modified version of the latter accounting differentially for learning from positive and negative prediction errors (thereafter referred to as RW±) (Frank et al., 2007; Niv et al., 2015). For each pair of cues, the model estimates the expected values of left and right options, QL and QR, on the basis of individual sequences of choices and outcomes. These Q values essentially represent the expected reward obtained by taking a particular option in a given context. In the first experiment, that involved only reward and reward omission, Q values were set at 0.25€ before learning, corresponding to the a priori expectation of 50% chance of winning 0.5€ plus a 50% chance of getting nothing. In the second experiment, which involved reward and punishment, Q values were set at 0.0€ before learning, corresponding to the a priori expectation of 50% chance of winning 0.5€ plus 50% chance of losing 0.5€. After every trial t, the value of the chosen option (e.g., L) was updated according to the following rule:
In the equation, a(t) was the prediction error, calculated as:
and R(t) was the reward obtained as an outcome of choosing L at trial t. In other words, the prediction error δ(t) is the difference between the expected reward QL(t) and the actual reward R(t). The reward magnitude R was +0.5 for winning 0.5€, 0 for getting nothing and -0.5 for losing 0.5€. The learning rate, α, is a scaling parameter that adjusts the amplitude of value changes from one trial to the next. Following this rule, option values are increased if the outcome is better than expected and decreased in the opposite case and the amplitude of the update is similar following positive and negative prediction errors.
The modified version of Q-Learning algorithm (RW±) differs from the original one (RW) by its Q values updating rule as follows:
The learning rate a+ adjusts the amplitude of value changes from one trial to the next when prediction error is positive (when the actual reward R(t) is better than the expected reward QL(t)) and the second learning rate α- does the same when prediction error is negative. Thus the RW± model allows for the amplitude of the update being different, following positive (“good news”) and negative (“bad news”) prediction errors and permits to account for individual differences in the way subjects learn from positive and negative experience. If both learning rates are equivalent, α+ < α−, RW± model equals the RW model. If α+ > α-, subjects learn more from positive than negative events. We refer to this case here as optimistic reinforcement learning. If α+ < α-, subjects learn more from negative than positive events. We refer to this case here as pessimistic reinforcement learning (Figure 2B).
Finally, given the Q values, the associated probability (or likelihood) of selecting each option was estimated by implementing the soft-max rule for choosing L, which is as follows:
This is a standard stochastic decision rule that calculates the probability of selecting one of a set of options according to their associated values. The temperature, β, is another scaling parameter that adjusts the stochasticity of decision-making and by doing so controls the exploration/exploitation trade-off.
Model comparison and subjects categorization
We optimized model parameters by minimizing the negative log-likelihood of the data given different parameters settings using Matlab’s fmincon function, as previously described (Palminteri et al., 2015). Negative log-likelihoods (LLmax) were used to compute at the individual level (random effects) for each model the Bayesian information criterion as follows:
We computed then the inter-individual average BIC in order to compare the quality of the fit of the two models, while accounting for their difference in complexity. The intra-individual difference in BIC (DBIC=BICRW - BICRW±) was also computed in order to categorize subjects in two groups and to quantitatively describe at the individual level the diverge from a realistic model (Figure 2A): RW± subjects, whose DBIC is positive, are better explained by RW± model. RW± subjects, whose DBIC is negative, are better explained by RW± model. We note that lower BIC indicated better fit.
The model parameters, (α+, α- and 1/β) were also compared between the two groups of subjects. Learning rates were compared using a mixed ANOVA with group (RW vs RW±) as a between-subject factor and learning rate type (+ or -) as a within-subject factor. The temperature was compared using a two-sample two-tailed t-test.
Model simulation
We also analyzed the models’ generative performance by the mean of model simulations. For each participant we devised a virtual subject, represented by a set of individual best fitting parameters obtained. Each virtual subject dataset was obtained averaging 100 simulations, to avoid any local effect of the individual history of choice and outcome. The model simulations included all task conditions. The evaluation of generative performances involved the assessment of the “winning model’s” ability to reproduce the key statistical effects of the data, as opposite to the “losing model”. Unlike Bayesian model comparison, model simulation comparison is bounded to a particular behavioral effect of interest (in our case the preferred response rate. The model simulation analysis, which is focused on the evidence “against” a given model, it is complementary to the Bayesian model comparison analysis, which is focused in the evidence in favor of a model (Dienes, 2008; Popper, 1959).
Imaging data Acquisition & Analysis
Subject of the first experiment (N=50) performed the task magnetic resonance imaging (MRI) scanning. T1-weighted structural images and T2*-weighted echo planar images (EPIs) were acquired during the first experiment and analyzed with the Statistical Parametric Mapping software (SPM8; Wellcome Department of Imaging Neuroscience, London, England). Acquisition and preprocessing parameters were previously and extensively described (Palminteri et al., 2009; Worbe et al., 2011). We refer to these publications for details about image acquisition and preprocessing.
Voxel-based morphometry
The aim of the VBM analysis was to link computational phenotypes to neuroanatomical signatures, thus providing external validity to the computational constructs (optimistic as opposite to realistic) identified in the behavioral analysis. As in previous healthy subjects and patients studies, VBM analysis relied on the Diffeomorphic Anatomical Registration Through Exponentiated Lie algebra (DARTEL) toolbox implemented in SPM8 software and followed the standard procedure outlined in the VBM tutorial (Ashburner and Friston, 2000; Lebreton et al., 2013; Palminteri et al., 2012). The individual grey matter images images were entered in a multiple regression design analysis with one variable of interest (ΔBIC, as an integrate computational measure of optimistic reinforcement learning) and three regressors of no interest: gender, age and total intracranial volume used as covariates to control for demographic characteristics and confounding effects of brain size. All significant disclosed on glass brain and coronal slices (Figure 3A and 3B), and reported in the text survived a threshold of p<0.001 (uncorrected) over the whole brain and contained a minimum of 30 contiguous voxels. The areas identified with this model-based VBM analysis (DMPFC and DLPFC) were then used as regions of interest (ROIs) for a supplementary VBM analysis, aimed to confirm the result with a model-free signature of optimistic learning (the preferred choice rate in the 25/25% condition) and for the subsequent fMRI analysis.
Functional magnetic resonance imaging
The aim of the fMRI analysis was to explore the functional consequence of the neuroanatomical differences observed in the prefrontal cortex and associated with optimistic trait. The fMRI analysis was based on a single general linear model. Each trial was modeled as having two time points, stimuli and outcome onsets. Each time point was regressed with a parameter modulator. Stimuli onset was modulated by the reaction time at each trial (continuous variable); outcome onset was modulated by the outcome obtained in the trial (reward: 0.50€ and no reward: 0.0€). The parametric modulators were z-scored to ensure between subject scaling of regression coefficients (Palminteri and Lebreton, 2016). Given that different subjects have been shown to implement different computational strategies, to allow between-group commensurability we opted for taking as parametric modulators model-free quantities: reaction times, which reflect the choice difficulty (decision process) and outcome representation, which is used in the learning (update process). Linear contrasts of regression coefficients were computed at the subject level and compared against zero to assess the presence of variable encoding effect and between-group to assess functional correlates of the two computational phenotypes. The comparisons were made within ROIs (DMPFC and DLPFC) identified with the VBM analysis that served as hypotheses generator for the fMRI analysis.
Acknowledgments
The authors acknowledge Yulia Worbe and Mathias Pessiglione for granting access to the first dataset. We thank Valentin
Wyart and Bahador Bahrami for helpful comments. SP was supported by a Marie Sklodowska-Curie Individual European Fellowship (pIEF-GA-2012 Grant 328822). GL was supported by a PHD fellowship of the Ministère de l’enseignement supérieur et de la recherche. ML was supported by an EU Marie Sklodowska-Curie Individual Fellowship (IF-2015 Grant 657904) and acknowledges the support of the Bettencourt-Schueller Foundation. The second experiment was supported by the ANR-ORA, Nesshi 2010-2015 research project to SBG.
Footnotes
↵a Bacon, F. (1939). Novum organum. In Burtt, E. A. (Ed.), The English philosophers from Bacon to Mill (pp. 24-123). New York: Random House. (Original work published in 1620)
↵b Original French citation: “Qu’est-ce qu’optimisme? disait Cacambo. – Hélas! dit Candide, c’est la rage de soutenir que tout est bien quand on est mal.” Voltaire (2014), Candide ou l’optimisme, Arvensa editions, p56, Ch. XIX. (Original work published in 1759).