Abstract
Humans use emotions to decipher complex cascades of internal events. However, which mechanisms link subjective descriptions of affective states to brain activity is unclear, with evidence supporting either local or distributed processing. A biologically favorable alternative is provided by the notion of gradient, which postulates the isomorphism between functional representations of stimulus features and cortical distance. Here, we used fMRI activity evoked by an emotionally charged movie and continuous ratings of the perceived emotion intensity, to reveal the topographic organization of affective states. We found that three orthogonal and spatially overlapping gradients encode the polarity, complexity and intensity of emotional experiences in right temporo-parietal territories. The spatial arrangement of these gradients allows the brain to map a variety of affective states within a single patch of cortex. As this organization resembles how sensory regions represent psychophysical properties (e.g., retinotopy), we propose emotionotopy as the underlying principle of emotion coding.
Introduction
Emotions promptly translate inner experiences into specific patterns of interpretable behaviors. The ability to infer others’ affective states represents a crucial aspect both when humans directly relate to each other and when they simply observe social interactions. Through years, the relevance of such an ability motivated the quest for models that optimally associate behavioral responses to subjective emotional experiences.
In this regard, seminal works pointed toward the existence of discrete basic emotions, characterized by distinctive and culturally stable facial expressions1, patterns of autonomous nervous system activity2,3 and bodily sensations4. Happiness, surprise, fear, sadness, anger and disgust represent the most frequently identified set of basic emotions5, though alternative models propose that other emotions, such as pride or contempt, should be included for their social and biological relevance6. To prove the neurobiological validity of these models, neuroscientists investigated whether basic emotions would elicit specific patterns of brain responses, consistently across subjects. Findings show that activity in amygdala, medial prefrontal, anterior cingulate, insular, middle/inferior frontal, and posterior superior temporal cortex, is associated to the perceived intensity of emotions and supports their recognition7–10. However, this perspective has been challenged by other studies, which failed to demonstrate significant associations between single emotions and activity within distinct cortical areas or networks11–13.
An alternative theory proposes that behavioral, physiological and subjective characteristics of emotions would be more appropriately described along a number of continuous cardinal dimensions14,15, generally one governing pleasure versus displeasure (i.e., valence) and another one the strength of the experience (i.e., arousal). While these two dimensions have been reliably and consistently described, alternative models propose that additional dimensions, such as dominance or unpredictability, are needed to adequately explain affective states16,17. Neuroimaging studies also demonstrated that stimuli varying in valence and arousal elicit specific and reliable brain responses18,19, which have been recently employed to decode emotional experiences20. Activity recorded in insula, amygdala, ventral striatum, anterior cingulate, ventromedial prefrontal and posterior territories of the superior temporal cortex, is associated to transitions between positive and negative valence and fluctuations in arousal21,22.
Interestingly, the recruitment of the posterior portion of the superior temporal cortex, extending to temporo-parietal territories, is consistently reported using either discrete emotion categories10,12 or emotion dimensions22–25, regardless of the sensory modality9,10. Furthermore, these temporoparietal regions are fundamental for social cognition, as they support empathic processing26,27 and the attribution of intentions, beliefs and emotions to others28,29.
However, despite this large body of evidence, it remains to be determined whether the subjective emotional experience is better explained through basic emotions or emotion dimensions. Moreover, regardless of the adopted model, it is still debated how emotion features are spatially encoded in the brain8,11,13,30–32. As a matter of fact, while findings support the role of distinct regions7, others indicate the recruitment of distributed networks in relation to specific affective states33.
An alternative and biologically favorable perspective may be provided by the notion of gradient. Gradients have been proven a fundamental organizing principle through which the brain efficiently represents and integrates stimuli coming from the external world. For instance, the location of a stimulus in the visual field is easily described through two orthogonal spatially overlapping gradients in primary visual cortex: rostrocaudal for eccentricity and dorsoventral for polar angle34. Thus, using functional magnetic resonance imaging (fMRI) and retinotopic mapping, one can easily predict the location of a stimulus in the visual field considering the spatial arrangement of recruited voxels with respect to these orthogonal gradients. Crucially, recent investigations revealed that gradients support the representation of higher-order information as well35–37, with features as animacy or numerosity being topographically arranged onto the cortical mantle35,38,39.
Following this view, we hypothesize that a gradient-like organization may represent the anatomo-functional grounding principle of the human emotional system. Specifically, different affective states would be mapped onto the cortical mantle through spatially overlapping gradients, either representing specific single emotions (e.g., sadness) or, alternatively, emotion dimensions (e.g., valence). The pattern of brain activity would therefore be interpreted according to emotion gradients to predict the subjective affective state.
Here, we tested this hypothesis using moment-by-moment ratings of the perceived intensity of emotions elicited by an emotionally charged movie. To unveil cortical regions involved in emotion processing, behavioral ratings were used as predictors of fMRI activity in an independent sample of subjects exposed to the same movie. The correspondence between functional characteristics and the relative spatial arrangement of distinct patches of cortex was then tested to reveal the existence of emotion gradients, which we named emotionotopy.
Results
Emotion Ratings
A group of Italian native speakers continuously rated the perceived intensity of six basic emotions5 (i.e., happiness, surprise, fear, sadness, anger and disgust) while watching an edited version of the Forrest Gump movie (R. Zemeckis, Paramount Pictures, 1994). We first assessed how much each basic emotion contributed to the behavioral ratings and found that happiness and sadness explained 28% and 36% of the total variance, respectively. Altogether, fear (18%), surprise (8%), anger (7%), and disgust (3%) explained the remaining one-third of the total variance. We also evaluated the agreement in ratings of the six basic emotions (Figure 1A), and found that happiness (Spearman’s ρ = 0.476 ± 0.102, range 0.202 - 0.717), fear (ρ = 0.522 ± 0.134, range 0.243 - 0.793), sadness (ρ = 0.509 ± 0.084, range 0.253 - 0.670), and anger (ρ = 0.390 ± 0.072, range 0.199 - 0.627) were consistent across all the subjects, whereas surprise (ρ = 0.236 ± 0.099, range 0.010 - 0.436) and disgust (ρ = 0.269 ± 0.115, range 0.010 - 0.549) were not. Nonetheless, ratings for these latter emotions were on average significantly different from a null distribution of randomly assigned emotion ratings (p-value < 0.05).
To reveal emotion dimensions, we averaged across subjects the ratings of the six basic emotions, measured their collinearity (Figure 1B) and performed Principal Component (PC) analysis (Figure 1C). The first component reflected a measure of polarity (PC1: 45% explained variance) as positive and negative emotions demonstrated opposite loadings. The second component was interpreted as a measure of complexity (PC2: 24% explained variance) of the perceived affective state, ranging from a positive pole where happiness and sadness together denoted inner conflict and ambivalence, to a negative pole mainly representing fearful events. The third component was a measure of intensity (PC3: 16% explained variance), since all the six basic emotions showed positive loadings (Figure 1C). Altogether, the first three components explained approximately 85% of the total variance. We further assessed the stability of the PCs and found that only these first three components (polarity: ρ = 0.610 ± 0.089, range 0.384 - 0.757; complexity: ρ = 0.453 ± 0.089, range 0.227 - 0.645; intensity: ρ = 0.431 ± 0.071, range 0.258 - 0.606), hereinafter emotion dimensions, were consistent across all the subjects (Figure 1D). The fourth PC described movie segments during which participants experienced anger and disgust at the same time (PC4: 8% explained variance, ρ = 0.329 ± 0.128, range −0.003 - 0.529), whereas the fifth PC was mainly related to surprise (PC5: 6% explained variance, ρ = 0.214 ± 0.090, range 0.028 - 0.397). Notably, these two PCs were not consistent across all the subjects, even though their scores were on average significantly different from a null distribution (p-value < 0.05). Scores of the sixth PC were not significantly consistent across subjects (PC6: 1% explained variance, p-value > 0.05).
Richness of the Reported Emotional Experience
In our behavioral experiment, participants were allowed to report the perceived intensity of more than one emotion at a time. Thus, the final number of elicited emotional states might be greater than the original six emotion categories. To measure the richness of affective states reported by our participants, we performed dimensionality reduction and clustering analyses on group-averaged behavioral ratings. Results revealed the existence of 15 distinct affective states throughout the movie (Figure 2). Among these, some states were characterized by a single basic emotion, whereas others by a peculiar mixture of them. Combinations of distinct emotions likely expressed secondary affective states, as ambivalence (i.e., cluster j depicting movie scenes in which happiness and sadness are simultaneously experienced) or resentment (i.e., cluster i representing movie segments in which a mixture of sadness, anger and disgust is perceived). Of note, this evidence is supported also by single-subject reports, in which the 38% (SE: ±2.3%) of timepoints were associated to a single emotion, the 29% (SE: ±3.5%) to two basic emotions and the 6% (SE: ±1.4%) to the concurrent experience of three distinct emotions. Altogether, these results show that the Forrest Gump movie evoked complex and multifaceted experiences, which cannot be reduced to the original six categories.
Brain Regions encoding Emotion Ratings
Emotion ratings obtained from the behavioral experiment were used as predictors of brain activity in independent subjects exposed to the same movie (studyforrest project; http://studyforrest.org40). The model significantly explained activity in right inferior frontal gyrus (IFG), rostral middle frontal gyrus (rMFG), medial superior frontal gyrus (mSFG), occipitotemporal sulcus (OTS), precentral sulcus (preCS), posterior part of the superior temporal sulcus/temporoparietal junction (pSTS/TPJ), middle occipital gyrus (MOG) and posterior middle temporal gyrus (pMTG). We also observed significant results in the left supramarginal gyrus (SMG) and pMTG (q < 0.01 FDR corrected and cluster size > 10; Figure 3A and Supplementary Table 1). Notably, the peak of association between emotion ratings and brain activity was located in the right pSTS/TPJ, an important region for social cognition12,22,26,28,29 (R2 = 0.07 ± SE = 0.009; Center of Gravity – CoG: x = 61, y = −40, z = 19; noise ceiling lower bound 0.13, upper bound 0.23; Figure 3B and Supplementary Figure 1). The peak of association was also located in proximity (11 mm displacement) of the reverse inference peak for the term “TPJ” (CoG: x = 58, y = −50, z = 16) as reported in the NeuroSynth database (neurosynth.org; Figure 3B).
Emotion Gradients in Right Temporo-Parietal Territories
We tested the existence of either basic emotion or emotion dimension gradients in a spherical region of interest located at the reverse inference peak for the term “TPJ”. This analysis was conducted on behavioral ratings consistent across all the subjects: happiness, sadness, fear and anger for basic emotions and polarity, complexity and intensity for emotion dimensions.
Using β coefficients obtained from the encoding analysis, we observed that, within right TPJ, voxels appeared to encode happiness in an anterior to posterior arrangement, fear and sadness in an inferior to superior manner, while anger showed a patchier organization (Figure 3C). With respect to emotion dimensions, voxels seemed to encode polarity and intensity in a more inferior to superior fashion, whereas complexity in a more posterior to anterior direction (Figure 3D).
To prove the existence and precisely characterize the orientation of these gradients, we tested the association between physical distance and functional characteristics of right TPJ voxels (Supplementary Figure 2). Results demonstrated that within a 15 mm radius sphere, the relative spatial arrangement and functional features of right TPJ were significantly and maximally correlated, either considering the basic emotion model (ρ = 0.352, p-value = 0.004, 95% Confidence Interval - CI: 0.346 to 0.357) or the emotion dimension one (ρ = 0.399, p-value < 0.001, 95% CI: 0.393 to 0.404; for alternative definitions of the right TPJ region see Supplementary Table 2).
Crucially, when focusing on each emotion dimension, results revealed the existence of three orthogonal and spatially overlapping gradients: polarity (ρ = 0.241, p-value = 0.041, 95% CI: 0.235 to 0.247), complexity (ρ = 0.271, p-value = 0.013, 95% CI: 0.265 to 0.277) and intensity (ρ = 0.229, p-value = 0.049, 95% CI: 0.223 to 0.235; Figure 4 and Supplementary Table 3). On the contrary, happiness (ρ = 0.275, p-value = 0.013, 95% CI: 0.269 to 0.281), but not other basic emotions (fear: ρ = 0.197, p-value = 0.091; sadness: ρ = 0.182, p-value = 0.160; anger: ρ = 0.141, p-value = 0.379; Supplementary Table 3), retained a gradient-like organization. Of note, the peculiar arrangement of group-level emotion dimension gradients (Figure 4) was also identified using single-subject fMRI data (Supplementary Figure 3 and Supplementary Table 4).
As any orthogonal rotation applied to the emotion dimensions would result into different gradients, we measured to what extent rotated solutions explained the topography of right TPJ. Therefore, we tested the correspondence between anatomical distance and the fitting of ~70,000 rotated versions of polarity, complexity and intensity (see Supplementary Materials for a comprehensive description). Results showed that the original unrotated emotion dimensions represented the optimal solution to explain the gradient-like organization of right temporo-parietal territories (Supplementary Figure 4).
Further, we performed a data-driven searchlight analysis to test whether right TPJ was the only region significantly encoding all the three emotion dimension gradients (please refer to Supplementary Materials for details). Results obtained from the meta-analytic definition of right TPJ were confirmed using this alternative approach (q < 0.05 FDR corrected and cluster size > 10; CoG: x = 58, y = −53, z = 21; Supplementary Figure 5), as no other region encoded the combination of polarity, complexity and intensity in a topographic manner.
Moreover, we conducted three separate searchlight analyses to characterize the spatial arrangement of single emotion dimension gradients (see Supplementary Material). Polarity, complexity and intensity maps revealed specific topographies: regions as the right preCS represented the three emotion dimensions in distinct - yet adjoining - subregions, whereas the right OTS encoded overlapping gradients of complexity and intensity (Supplementary Figure 6).
When we explored whether the left hemisphere homologue of TPJ (CoG: x = −59, y = −56, z = 19) showed a similar gradient-like organization, we did not find significant associations between spatial and functional characteristics either for the basic emotion model (ρ = 0.208, p-value = 0.356) or the emotion dimension one (ρ = 0.251, p-value = 0.144; Supplementary Table 2). Specifically, neither any of the emotion dimensions (polarity: ρ = 0.132, p-value = 0.354; complexity: ρ = 0.157, p-value = 0.222; intensity: ρ = 0.149, p-value = 0.257) nor any of the basic emotions showed a gradient-like organization in left TPJ (happiness: ρ = 0.158, p-value = 0.216; fear: ρ = 0.142, p-value = 0.293; sadness: ρ = 0.156, p-value = 0.213; anger: ρ = 0.073, p-value = 0.733; Supplementary Table 3).
Lastly, as spatial smoothness of functional data and cortical folding may affect the estimation of gradients, we performed additional analyses considering the unfiltered version of group-average brain activity and obtaining a measure of the anatomical distance respectful of cortical topology. Results showed that the topographic arrangement of emotion dimensions in right temporo-parietal territories was not affected by smoothing (Supplementary Figure 7) and respected the cortical folding (polarity: ρ = 0.248, p-value = 0.026, CI: 0.238-0.257; complexity: ρ = 0.314, p-value = 0.001, CI: 0.304-0.323; intensity: ρ = 0.249, p-value = 0.013, CI: 0.239-0.258). For details about this procedure and a comprehensive description of the results please refer to Supplementary Materials.
To summarize, polarity, complexity and intensity dimensions were highly consistent across individuals, explained the majority of the variance in behavioral ratings (85%) and were mapped in a gradient-like manner in right (but not left) TPJ. Happiness (28% of the total variance in behavioral ratings) was the only basic emotion to be consistent across subjects and to be represented in right TPJ. Importantly, though, happiness and complexity demonstrated high similarity both in behavioral ratings (ρ = 0.552) and in brain activity patterns (ρ = 0.878). Taken together, these pieces of evidence indicate the existence of emotion dimension gradients in right temporo-parietal territories, rather than the coding of single basic emotions.
Emotion Dimension Gradients represent the Subjective Emotional Experience
In movie watching, actions and dialogues are not generally directed toward the observer. Thus, the affective state of a bystander is influenced by character emotions, intentions and beliefs. Although we specifically asked our subjects to report their own subjective experience, one could argue that the emotion dimension gradients we measured depended on the attribution of mental states to others.
Therefore, we investigated whether the gradient-like organization of right TPJ specifically reflected the subjective experience, rather than emotions portrayed in the movie. To this aim, we took advantage of publicly available tagging data of Forrest Gump41, in which participants indicated the portrayed emotion of each character and whether such an emotion was directed toward the character itself (self-directed; e.g., Forrest feeling sad) or toward another one (other-directed; e.g., Forrest feeling happy for Jenny). As in Labs and colleagues41, these reports constituted two third-person emotion attribution descriptions, which we used as models of the attribution of affective states to others (please refer to Supplementary Materials for details).
On average, subjective ratings shared the 11.4% ± 8.6% of the variance with the self-directed emotion attribution model and the 35.3% ± 16.1% with the other-directed model, indicating that the subjective emotional experience could be inferred from portrayed emotions only in part. Moreover, in line with previous studies highlighting the role of right TPJ in the attribution of mental states to others28,29,42, the other-directed emotion attribution model significantly explained activity of this region (Supplementary Figure 8). Crucially, though, none of the first six components obtained from the other-directed emotion attribution model (i.e., 87% of the explained variance) retained a topographic organization in right TPJ (Supplementary Table 5). We also used canonical correlation analysis to transform the other-directed model into the space defined by subjective emotion ratings and tested whether starting from a third-person complex description of portrayed emotions, one can fully reconstruct the brain topography of emotion dimensions. Noteworthy, only the first aligned component was mapped in a topographic manner within right TPJ (reconstructed polarity: ρ = 0.221, p-value = 0.036; reconstructed complexity: ρ = 0.150, p-value = 0.384; reconstructed intensity: ρ = 0.207, p-value = 0.092). This finding suggests that the information coded in the other-directed emotion attribution model is not sufficient to fully reconstruct the topography of the subjective emotional experience.
Characterization of Emotion Dimension Gradients
To detail how right TPJ gradients encode perceived affective states, we have reconstructed fMRI activity for movie segments connoted by either positive or negative polarity, as well as higher or lower complexity and intensity. The orientation of the three emotion dimension gradients was represented by the symmetry axis of our region of interest. Specifically, for polarity events connoted by positive emotions increased activity in ventrorostral territories, lying close to the superior temporal sulcus, whilst highly negative events augmented hemodynamic activity in dorsocaudal portions of right TPJ, extending to the posterior banks of Jensen sulcus (Figure 5A and 3D).
Events connoted by higher complexity (e.g., concurrent presence of happiness and sadness) were associated to signal increments in rostrolateral territories of right TPJ, whereas those rated as having lower complexity (e.g., fearful events) increased hemodynamic activity in its caudal and medial part, encompassing the ascending ramus of the superior temporal sulcus (Figure 5B and 3D). Higher levels of intensity were related to increased activity in rostrodorsal and ventrocaudal territories, reaching the ascending ramus of the lateral sulcus and posterior portions of the middle temporal gyrus, respectively. On the contrary, low-intensity events augmented hemodynamic activity in a central belt region of right TPJ, located along the superior temporal sulcus (Figure 5C and 3D). Noteworthy, the orthogonal arrangement of polarity and complexity and the fact that intensity was represented both superiorly and inferiorly to the superior temporal sulcus determined that the variety of emotional states elicited by the Forrest Gump movie (see Figure 2) could be mapped within a single patch of cortex.
Moreover, in sensory areas, topographies result from the maximal response of neurons to a graded stimulus feature. To parallel right TPJ emotion dimension gradients with those observed in primary sensory regions, we investigated whether distinct populations of voxels were selective for specific polarity, complexity and intensity scores. Thus, we employed the population receptive field method43 to estimate the tuning curve of right TPJ voxels for each emotion dimension. The map of voxel selectivity was consistent with the topography obtained from the original gradient estimation for the three emotion dimensions (polarity: ρ = 0.547, p-value = 0.001; complexity: ρ = 0.560, p-value < 0.001 and intensity: ρ = 0.596, p-value < 0.001). Specifically, results demonstrated the existence of four populations of voxels tuned to specific polarity values, which encoded highly and mildly positive and negative events, respectively (Figure 6A). Also, two distinct populations of voxels were tuned to maximally respond during cognitively mediated affective states (i.e., highly and mildly positive complexity values), and two other populations were selective for emotions characterized by higher and lower levels of automatic responses (i.e., highly and mildly negative complexity values; Figure 6B). Lastly, for the intensity dimension two specific populations of voxels were engaged depending on the strength of the emotional experience (Figure 6C). This further evidence favored the parallel between emotion and sensory gradients.
Discussion
Previous studies reported that activity of individual brain regions codes distinct emotion features7, whereas others suggested that a distributed network of cortical areas conjointly interacts to represent affective states33. Interestingly, though, the possibility that gradients may encode the emotional experience as function of either basic emotions, or emotion dimensions, has never been explored. The topological isomorphism between feature space and cortical distances has been adopted to successfully relate psychophysical characteristics of stimuli to patterns of activity in sensory regions34. Nonetheless, this biologically advantageous mechanism has been recently proven to lie at the basis of the cortical representation of higher-level (e.g., semantic) features as well35,38,39. Building upon this evidence, we tested whether different affective states could be mapped onto the cortical mantle through spatially overlapping gradients.
We demonstrated that the topography of right TPJ, a crucial cortical hub for social cognition12,22,26,28,29, is described by emotion dimensions, rather than by single basic emotions. Indeed, within this region, we discovered three orthogonal and spatially overlapping gradients encoding the polarity, complexity and intensity of the emotional experience. The peculiar arrangement of these gradients allows a gamut of emotional experiences to be represented in a single patch of cortex, including affective states perceived as pleasant, unpleasant or ambivalent, connoted by calmness or excitement and mediated by primitive reactions or mentalization processes. Therefore, TPJ organization resembles the one that can be observed in primary sensory areas, where stimulus properties are topographically arranged onto the cortical mantle, as in the case of eccentricity and polar angle in the primary visual cortex (V1), pitch in the primary auditory region (A1) and body parts in the primary somatosensory area (S1). In this regard, the evidence that temporo-parietal territories encode emotion dimensions in a gradient-like manner represents an alternative and biologically plausible mechanism of coding of affective states in the human brain. Thus, we propose that as retinotopy represents the organizing principle of sight in V1, cochleotopy of hearing in A1 and somatotopy of somesthesis in S1, emotionotopy constitutes the basis of emotion processing in right TPJ. Indeed, as in vision precise portions of V1 map distinct locations of the visual field, specific regions of temporo-parietal territories code unique emotional experiences. This emerged also from the analysis of response tuning, showing how within each emotional hemifield of polarity and complexity, populations of voxels code specific levels of the emotional experience.
As for the case of polar angle and eccentricity in the primary visual cortex, right TPJ emotion dimension gradients are lower-dimensional descriptions of the underlying neural activity. The retinotopic representation of azimuth and elevation in V1 overlaps with local maps of ocular dominance and orientation tuning. Therefore, multiple neural codes exist at different spatial scales and the ability to capture either global or local representations relates to the resolution of the imaging technique. Our data provide evidence for a lower-dimensional, yet biologically favorable, neural code to represent emotions in temporo-parietal regions. Considering the parallel with the organization of sensory areas, we believe that the topography of right TPJ does not prevent the existence of other neural codes, especially considering the coexistence of global and local representations and the multifaceted nature of this region.
Furthermore, the fact that subjective reports explained the activity of other cortical modules is not necessarily in contrast with the central role of TPJ. In fact, as in vision a rich and complex percept requires both the primary visual cortex to extract fundamental features and other regions to process specific stimulus properties (i.e., V5 for motion), in emotion processing TPJ may represent a crucial hub embedded in a distributed network of regions carrying out distinct computations.
Richness of the Subjective Emotional Experience in Movie Watching
In our study, we employed a naturalistic continuous stimulation paradigm since it fosters emotional contagion and empathic reactions, leading to complex subjective emotional experiences, akin to real life22. Indeed, we found that within a 60 seconds time window, emotion transitions represented in the Forrest Gump movie are similar to those experienced in real life44 and are predicted by a mental model of emotion co-occurrence45 (please refer to Supplementary Materials for details). This supports the ecological validity of our stimulus and emphasizes that movies can be successfully adopted to nurture emotional resonance46–48 also in the fMRI setting.
In movie watching, actions and dialogues generally are not directed toward the observer. Thus, the subjective emotional experience results, on the one hand, from narrative choices aimed at fostering empathic responses and emotional contagion49 and, on the other hand, from perspective-taking and mentalizing processes50,51. The fact that character intentions and beliefs shape the subjective experience in a bystander may also explain the high between-subjects agreement in reports of experienced emotions46–48. This is in line with the consistency of behavioral ratings of happiness, fear, sadness and anger present in our data. Noteworthy, surprise and disgust were not consistent across all participants and, even though this may appear as a contradiction with respect to the supposed52 universalism of basic emotions, it should be noted that our stimulus was not specifically built to reflect the well-established definition of these six emotions. For instance, some of our subjects reported that movie scenes rated as disgusting were mainly associated to situations for which interpreting the context was necessary (e.g., the principal of the school using his power to obtain sexual favors), rather than to repulsive images. This cognitive interpretation of the basic emotion disgust was apparently not present in all the subjects, with some of them relying more on its well-established definition for their ratings. Also, the use of six distinct emotion categories offered the opportunity to compare the basic emotion model with the emotion dimension one starting from the same data. In fact, as we (and others15,17) have demonstrated, emotion dimensions can be easily derived from reports based on single emotions, whereas the opposite may not be feasible. For instance, happiness is associated to positive valence, yet the intensity of such an experience may be different depending on the context (e.g., win the lottery versus meet an old friend). Moreover, while the definition of basic emotions is common across individuals, ratings based on emotion dimensions require participants to be acquainted with the meaning of psychological constructs (e.g., dominance16).
Nonetheless, single basic emotions provide a coarse description of subjective experiences, since humans may perceive a complex blend of apparently conflicting emotions53, and affective states could emerge from psychological processes not directly reducible to single emotions54. Our rating model, though, does account for this possibility, since subjects were allowed to report throughout the movie the occurrence of more than one emotion at the same time, as when simultaneously experiencing happiness and sadness. Besides the divergences in literature on the precise temporal overlap in perceiving conflicting emotions53,55, when subjects are free to detail their personal experience, this peculiar emotional state seems to arise.
Therefore, even though our rating model was based on six emotion categories, the possibility to report the perceived intensity of more than one emotion at a time provided a rich online description of subjects’ emotional experiences. Indeed, using this procedure we identified fifteen distinct affective states elicited by the Forrest Gump movie (Figure 2), a number compatible with the one reported by previous studies aimed at investigating a wide range of emotion categories (e.g., craving, terror)56,29.
Polarity, Complexity and Intensity of the Emotional Experience
With respect to emotion dimensions, the components we identified were deliberately interpreted not following any known model. Nonetheless, the first dimension, polarity, mainly relates to positive against negative emotions as in valence14, whereas the third one, intensity, is unipolar and mimics arousal14. We considered the second component as a measure of complexity of the emotional state. Indeed, this dimension contrasts events in the movie rated as fearful, an emotion with a fast and automatic response57, against scenes characterized by ambivalence, where cognitive processes play a significant role in generating “mixed emotions”58. Even though this component does not pertain to classical emotion dimension theories, an interesting interpretation may relate complexity to the involvement of Theory of Mind28 in emotion perception59. In this regard, the two extremes of this bipolar continuum characterize automatic responses on one side and cognitive processing of context and others’ mental and affective states on the other one.
Right Temporo-Parietal Cortex encodes Subjective Experiences in a Topographic manner
In our analyses, we used the collected behavioral ratings to explain brain activity in independent subjects. In line with previous studies,10,12 results highlighted a set of regions located mainly in the right hemisphere (Figure 3 and Supplementary Table 1). Interestingly, the peak of association between emotion ratings and brain activity was located in right pSTS/TPJ. This cortical area has been consistently identified as having a central role in the attribution of mental states to others, as demonstrated by functional neuroimaging28,29, noninvasive transcranial stimulation60 and lesion studies61. In addition, this region spans across the posterior portion of the superior temporal sulcus, which is implicated in emotion perception10,12,22,62,63. In line with this, we demonstrated that activity of right TPJ is significantly explained by the process of emotion attribution to others and by the subjective emotional experience as well. Crucially, though, the topography of emotion dimensions is explained by the subjective experience, and not by emotions portrayed in the movie. We reason that activity of this region is modulated by emotion attribution to movie characters, as it requires mentalization. At the same time, portrayed emotions influence the affective state of the observer through empathy and emotional contagion, and the final subjective experience is mapped in the same patch of cortex following the three cardinal axes represented by emotion dimensions. This view would reconcile previous studies demonstrating the involvement of right TPJ in the representation of subjective emotional experience10,22,63, in empathic processes26,27 and in the attribution of beliefs and emotions to others.28,29,42
In addition, in the current study, ratings of the emotional experience elicited by an American movie in Italian participants explained brain activity of German subjects. This suggests that the topographic representation of emotions in right temporo-parietal territories exists regardless of linguistic or micro-cultural differences. What may depend on the cultural background of each individual is instead the mapping of distinct emotional states within these gradients. For instance, ruminative thinking, sadness and apathy characterize melancholy in the Western culture and we speculate that such an emotion would be mapped in the brain as a negative state having high complexity. However, if different levels of polarity, complexity and intensity characterize melancholy in other cultures (e.g., Eastern), this emotion would be mapped differently with respect to the three emotion dimension topographies.
Of note, the optimal description of emotion dimension gradients is obtained considering a 15mm radius sphere centered at the meta-analytic peak for the term “TPJ”. This region of interest is ~42% larger in volume as compared to the reverse inference TPJ map (i.e., the likelihood that the term “TPJ” is used in a study given the presence of reported activation), but also ~32% smaller than the forward inference definition (i.e., the likelihood that a region will activate if a study uses the “TPJ” term; please refer to Supplementary Materials for further details). Therefore, emotion dimension gradients are best represented in a patch of cortex that approximates the definition of right TPJ based on brain activation studies. Nevertheless, the fact that emotion topography generalizes across a range of spatial scales (i.e., up to 27mm radius sphere) motivated the use of “temporo-parietal territories” in describing the location and size of emotion dimension gradients.
Limitations
First, the effect size we report for the relationship between emotion ratings and brain activity appears to be relatively small (i.e., 7% of explained variance in right TPJ). However, we would like to emphasize four aspects: (1) first, brain regions significantly encoding emotions are selected after rigorous correction for multiple comparisons; (2) second, the magnitude of the effect is in line with recent fMRI literature on the coding of emotions in the brain29 and the evaluation of the noise ceiling suggests that our emotion dimension model explains between 30% (i.e., upper bound) and 54% (i.e., lower bound) of right TPJ activity; (3) third, the encoding procedure for psychophysical features of the stimulus yielded similar effect size, with volume energy explaining approximately 6% of the total variance in primary auditory cortex and 3% in inferior colliculus (see Supplementary Figure 10); (4) lastly, we used a parsimonious encoding model, in which only six predictors explained 3,595 samplings of brain activity.
Second, the collected emotion ratings are specific for the present stimulus. Thus, we cannot exclude that the use of alternative movies depicting horror, or sexual desire56 may produce different emotion dimensions. However, we demonstrated that when subjects could choose among a larger set of emotion categories, including social and secondary, the same polarity, complexity and intensity dimensions still emerged (see Supplementary Materials for details).
Conclusions
In summary, our results demonstrate that moment-by-moment subjective ratings of perceived emotions explain brain activity recorded in independent subjects. Most importantly, we proved the existence of emotionotopy in the right temporo-parietal cortex, where orthogonal and spatially overlapping gradients of emotion dimensions may represent the underlying principle of emotion coding in the human brain.
Methods
In the present study, we took advantage of a high quality publicly available dataset, part of the studyforrest project40 (http://studyforrest.org), to demonstrate the existence of a gradient-like organization in brain regions coding the subjective emotional experience. Particularly, we used moment-by-moment ratings of the perceived intensity of six basic emotions elicited by an emotionally charged movie (Forrest Gump; R. Zemeckis, Paramount Pictures, 1994), as predictors of fMRI activity in an independent sample. We then tested the correspondence between the fitting of the emotion rating model in TPJ voxels and their relative spatial arrangement to reveal the existence of orthogonal spatially overlapping gradients.
Behavioral Experiment
Participants
To obtain moment-by-moment emotion ratings during the Forrest Gump movie, we enrolled 12 healthy Italian native speakers (5F; mean age 26.6 years, range 24-34). None of them reported to have watched the movie in one year period prior to the experiment. Subjects signed an informed consent to participate in the study, had the right to withdraw at any time and received a small monetary compensation for their participation. The study was conducted in accordance with the Declaration of Helsinki and was approved by the local IRB (Protocol N°1485/2017).
Experimental setup
We started from the Italian dubbed version of the movie, edited following the exact same description reported in the studyforrest project (eight movie segments ranging from a duration of 11 to 18 minutes). The movie was presented in a setting free from distractions using a 24” monitor with a resolution of 1920×1080 pixels connected to a MacBook™ Pro running Psychtoolbox64 v3.0.14. Participants wore headphones in a noiseless environment (Sennheiser™ HD201; 21-18,000 Hz; Maximum SPL 108dB) and were instructed to continuously rate the subjective perceived intensity (on a scale ranging from 0 to 100) of six basic emotions5 throughout the entire movie: happiness, surprise, fear, sadness, anger and disgust. Specific buttons mapped the increase and decrease in intensity of each emotion and subjects were instructed to represent their inner experience by freely adjusting or maintaining the level of intensity. Participants were allowed to report more than one emotion at the same time and ratings were continuously recorded with a 10Hz-sampling rate. Subjects were presented with the same eight movie segments employed in the fMRI study one after the other, for an overall duration of 120 minutes. Further, before starting the actual emotion rating experiment, all participants performed a 20 minutes training session to familiarize with the experimental procedure. Specifically, they had to reproduce various levels of intensity for random combinations of emotions that appeared on the screen every ten seconds.
Behavioral data pre-processing
For each subject, we recorded six timeseries representing the moment-by-moment perceived intensity of basic emotions. First, we downsampled timeseries to match the fMRI temporal resolution (2s) and, afterwards, we introduced a lag of 2s to account for the delay in hemodynamic activity. The resulting timeseries were then temporally smoothed using a moving average procedure (10s window). This method allowed us to further account for the uncertainty of the temporal relationship between the actual onset of emotions and the time required to report the emotional state.
Agreement across subjects of the six basic emotions
To verify the consistency in the occurrence of affective states while watching the Forrest Gump movie, we computed the Spearman’s ρ correlation coefficient across subjects for each of the six ratings (Figure 1B). Statistical significance of the agreement was assessed by generating a null distribution of random ratings using the IAAFT procedure (Iterative Amplitude Adjusted Fourier Transform65; Chaotic System Toolbox), which provided surrogate data with the same spectral density and temporal autocorrelation of the averaged ratings across subjects (1,000 surrogates).
Basic emotion and emotion dimension models
Preprocessed and temporally smoothed singlesubject emotion ratings were averaged to obtain six group-level timeseries representing the basic emotion model. After measuring the Spearman’s ρ between pairings of basic emotions (Figure 1B), we performed principal component (PC) analysis and identified six orthogonal components, which constituted the emotion dimension model (Figure 1C).
Agreement across subjects of the emotion dimensions
To verify the consistency across subjects of the PCs, we computed the agreement of the six components by means of a leave-one-subject-out cross validation procedure (Figure 1D). Specifically, for each iteration, we performed PC analysis on the left-out subject behavioral ratings and on the averaged ratings of all the other participants. The six components obtained from each left-out subject were rotated (Procrustes analysis, reflection and orthogonal rotation only) to match those derived from all the other participants. This procedure generated for each iteration (i.e., for each of the left-out subjects) six components, which were then compared across individuals using Spearman’s ρ similarly to what has been done for the six basic emotions. To assess the statistical significance, we created a null distribution of PCs from the generated surrogate data of the behavioral ratings, as described above (1,000 surrogates).
Richness of the reported emotional experience
Although subjects were asked to report their inner experience using six emotion categories, their ratings were not limited to binary choices. Indeed, at each timepoint raters could specify the perceived intensity of more than one emotion, leading to the definition of more complex affective states as compared to the basic ones. To further highlight this aspect, we performed dimensionality reduction and clustering analyses on emotion timeseries. Starting from emotion ratings averaged across participants, we selected timepoints characterized by the highest intensity (i.e., by summing the six basic emotion and setting the threshold to the 50th percentile) and applied Barnes-Hut t-distributed stochastic neighbor embedding56,66 (t-SNE; perplexity = 30; theta = 0.05). The algorithm measures the distances between timepoints in the six-dimensional space defined by the basic emotions as joint probabilities according to a Gaussian distribution. These distances are projected onto a two-dimensional embedding space using a Student’s t probability distribution and by minimizing the Kullback–Leibler divergence. To further describe the variety of affective states elicited by the movie, we then applied k-means clustering analysis to the projection of timepoints in the t-SNE manifold and determined the number of clusters using the silhouette criterion67.
fMRI Experiment
We selected data from the phase II of the studyforrest project, in which 15 German mother tongue subjects watched an edited version of the Forrest Gump movie during the fMRI acquisition. Participants underwent two 1-hour sessions of fMRI scanning (3T, TR 2s, TE 30ms, FA 90°, 3mm ISO, FoV 240mm, 3599 tps), with an overall duration of the experiment of 2h across eight runs. Subjects were instructed to inhibit any movement and simply enjoy the movie (for further details40). We included in our study all participants that underwent the fMRI acquisition and had the complete recordings of the physiological parameters (i.e., cardiac trace) throughout the scanning time (14 subjects; 6F; mean age 29.4 years, range 20-40 years). For the fMRI pre-processing pipeline please refer to Supplementary Materials.
Encoding Analysis
Voxel-wise encoding68,69 was performed using a multiple linear regression approach to measure the association between brain activity and the emotion ratings, constituted by the six principal components. Of note, performing a least square linear regression using either the six principal components or the six basic emotion ratings yields the same overall fitting (i.e., full model R2), even though the coefficient of each column could vary among the two predictor sets.
To reduce the computational effort, we limited the regression procedure to gray matter voxels only (~44k with an isotropic voxel resolution of 3mm). We assessed the statistical significance of the R2 fitting of the model for each voxel using a permutation approach, by generating 10,000 null encoding models. Null models were obtained by measuring the association between brain activity and surrogate data having the same spectral density and temporal autocorrelation of the original six PCs. This procedure provided a null distribution of R2 coefficients, against which the actual association was tested. The resulting p-values were corrected for multiple comparisons through the False Discovery Rate70 method (q < 0.01; Figure 3A, Supplementary Figure 1 and Supplementary Table 1). R2 standard error was calculated through a bootstrapping procedure (1,000 iterations). Moreover, we conducted a noise-ceiling analysis for right TPJ data, similarly to what has been done by Ejaz and colleagues71 (please see Supplementary Materials).
Emotion Gradients in right TPJ
Estimation of right TPJ topography
We tested the existence of emotion gradients by measuring the topographic arrangement of the multiple regression coefficients72 in regions lying close to the peak of fitting for the encoding procedure (i.e., right pSTS/TPJ). To avoid any circularity in the analysis73, we first delineated a region of interest (ROI) in the right pSTS/TPJ territories using an unbiased procedure based on the NeuroSynth74 database v0.6 (i.e., reverse inference probability associated to the term “TPJ”). Specifically, we started from the peak of the “TPJ” NeuroSynth reverse inference meta-analytic map to draw a series of cortical ROIs, with a radius ranging from 9 to 27 mm. Afterwards, to identify the radius showing the highest significant association, for each spherical ROI we tested the relationship between anatomical and functional distance75 (Supplementary Table 2). This procedure was performed using either multiple regression coefficients obtained from the three emotion dimensions or from the four basic emotions stable across all subjects. As depicted in Supplementary Figure 2, we built for each radius two dissimilarity matrices: one using the Euclidean distance of voxel coordinates, and the other one using the Euclidean distance of the fitting coefficients (i.e., β values) of either the three emotion dimensions or the four basic emotions. The rationale behind the existence of a gradient-like organization is that voxels with similar functional behavior (i.e., lower functional distance) would also be spatially arranged close to each other on the cortex75 (i.e., lower physical distance). The functional and anatomical dissimilarity matrices were compared using the Spearman’s ρ coefficient. To properly address the significance of the anatomo-functional association, we built an ad hoc procedure that maintained the same spatial autocorrelation structure of TPJ in the null distribution. Specifically, we generated 1,000 IAAFT-based null models for the emotion dimension and the basic emotion data, respectively. These null models represented the predictors in a multiple regression analysis and generated a set of null β regression coefficients. Starting from these coefficients we built a set of functional dissimilarity matrices that have been correlated to the anatomical distance and provided 1,000 null Spearman’s ρ coefficients, against which the actual anatomo-functional relationship was tested. Confidence intervals (CI, 2.5 and 97.5 percentile) for the obtained correlation values were calculated employing a bootstrap procedure (1,000 iterations). We also tested the existence of gradients in other brain regions encoding emotion ratings using a data-driven searchlight analysis. Results and details of this procedure are reported in Supplementary Materials.
Impact of spatial smoothing on emotion topography
To estimate the significance of right TPJ gradients we used null models built on emotion ratings, leaving untouched the spatial and temporal structure of brain activity. However, as spatial smoothness may still affect the estimation of gradients, we tested right TPJ topography using the group-average unfiltered data. In brief, all the steps described in the fMRI data pre-processing section (see Supplementary Materials) were applied, with the only exception of spatial filtering. Following this procedure, the estimated smoothness of the right TPJ region was 4.5×4.2×3.6mm (3dFWHMx tool). Using these data and the same procedure described in the Estimation of right TPJ topography paragraph, we measured the significance of emotion gradients. Results are detailed in Supplementary Table 6 and Supplementary Figure 7.
Impact of cortical folding on emotion topography
The Euclidean metric does not take into account cortical folding. Indeed, because of the morphological characteristics of TPJ, which include a substantial portion of STS sulcal walls, the estimation of emotion gradients would benefit from the use of a metric respectful of cortical topology.
For this reason, we ran the Freesurfer recon-all analysis pipeline76 on the standard space template77 used as reference for the nonlinear alignment of single-subject data. We then transformed the obtained files in AFNI-compatible format (@SUMA_Make_Spec_FS). This procedure provided a reconstruction of the cortical ribbon (i.e., the space between pial surface and gray-to-white matter boundary), which has been used to measure the anatomical distance. In this regard, we particularly employed the Dijkstra algorithm as it represents a computationally efficient method to estimate cortical distance based on folding78,79. The single-subject unsmoothed timeseries were then transformed into the standard space, averaged across individuals and projected onto the cortical surface (AFNI 3dVol2Surf, map function: average, 15 steps). Afterwards, we performed a multiple linear regression analysis using principal components derived from emotion ratings as predictors of the unsmoothed functional data. This analysis was carried out within a cortical patch that well approximated the size of the 3D-sphere used in the original volumetric pipeline and centered at the closest cortical point with respect to the Neurosynth “TPJ” peak. Thus, for each regressor of interest, we obtained unsmoothed β values projected onto the cortical mantle. We then tested the existence of a gradient-like organization for each predictor, using the Dijkstra algorithm and the same procedure described above (i.e., Estimation of right TPJ topography paragraph). Results are detailed in Supplementary Table 6 and Figure 4.
Subjective Emotional Experience versus Third-Person Emotion Attribution
We tested whether the gradient-like organization of right TPJ specifically reflects the subjective emotional experience, rather than portrayed emotions. Thus, we took advantage of publicly available emotion tagging data of the same movie, provided by an independent group41. Differently from our behavioral task, raters were asked to indicate the portrayed emotion of each character (e.g., Forrest Gump, Jenny) in 205 movie segments presented in random order and labeled over the course of approximately three weeks. As also suggested by the authors41, this particular procedure minimizes carry-over effects and help observers to exclusively focus on indicators of portrayed emotions. In addition, the possibility to tag emotions independently in each movie segment and to watch each scene more than once, allowed subjects to choose among a larger number of emotion categories80 (N = 22), as compared to our set of emotions. Moreover, each observer was instructed to report with a binary label whether the portrayed emotion was directed toward the character itself (self-directed; e.g., Forrest feeling sad) or toward another character (other-directed; e.g., Forrest feeling happy for Jenny). These two descriptions served as third-person emotion attribution models and underwent the exact same processing steps (i.e., 2s lagging and temporal smoothing), which have been applied to our subjective emotion rating model. As the two third-person emotion attribution models included the four basic emotions found to be consistent across observers in our experiment (i.e., happiness, fear, sadness and anger), we have been able to directly assess the correlation for these ratings using Spearman’s ρ.
We then measured the extent to which the two third-person emotion attribution models explained brain activity in right TPJ following the method described in the Encoding analysis section. As these two descriptions are higher in dimensionality as compared to our subjective emotion rating model, we assessed the significance of fitting using three different procedures: (A) matching the dimensionality across models by selecting the first six principal components only; (B) matching the emotion categories in ratings, by performing PCA on the four basic emotions shared across models (i.e., happiness, fear, sadness and anger); (C) using the full model regardless of the dimensionality (i.e., six components for our subjective emotion rating model and 22 for each of the two emotion attribution models). In addition, to allow a direct and unbiased comparison between R2 values obtained from different models, we performed cross-validation using a half-run split method (for further details please refer to Supplementary Materials).
Lastly, we tested whether right TPJ gradients encode emotion attribution models. Specifically, we evaluated two different scenarios: (1) the existence of right TPJ gradients encoding the 22 components of each emotion attribution model; (2) the possibility to identify emotion gradients following the multidimensional alignment81 (i.e., canonical correlation analysis) of the 22-dimensional emotion attribution space to the 6-dimensional space defined by subjective ratings. These alternative procedures relate to two different questions: (1) whether the process of emotion attribution is associated to emotion gradients in right TPJ and (2) whether starting from a third-person complex description of portrayed emotions, one can reconstruct the subjective report of our raters. Results for these two procedures are detailed in Supplementary Table 5.
Characterization of Emotion Gradients in right TPJ
Principal orientation of right TPJ gradients
Once the optimal ROI radius was identified, we tested the gradient-like organization of right TPJ for each individual emotion dimension and basic emotion (Supplementary Table 3), using the same procedure described above (i.e., Estimation of right TPJ topography section). We calculated the numerical gradient of each voxel using β values. This numerical gradient estimates the partial derivatives in each spatial dimension (x, y, z) and voxel, and can be interpreted as a vector field pointing in the physical direction of increasing β values. Afterwards, to characterize the main direction of each gradient, rather than calculating its divergence (i.e., Laplacian of the original data82,83), we computed the sum of all vectors in the field (Figure 4 and Supplementary Figure 2). This procedure is particularly useful to reveal the principal direction of linear gradients and provides the opportunity to represent this direction as the orientation of the symmetry axis of the selected ROI. The above-mentioned procedure was also adopted to assess the reliability of the emotion gradients in each subject. Results and details of this procedure are reported in Supplementary Materials. Furthermore, since gradients built on β coefficients could reflect positive or negative changes in hemodynamic signal depending on the sign of the predictor, we represented the average TPJ activity during movie scenes characterized by specific affective states (Figure 5).
Population receptive field estimates in right TPJ
We investigated whether distinct populations of voxels are selective for specific affective states. To this aim, we employed the population receptive field method43 (pRF) and estimated the tuning curve of right TPJ voxels for each predictor found to be topographically encoded within this region. We modeled the tuning curve of each voxel as a Gaussian distribution, in which μ represented the preferred score of the predictor and σ the width of the response. The optimal combination of tuning parameters was selected among ~5k plausible values of μ (5th-95th percentile of the scores of each predictor - 0.5 step) and σ (ranging from 1 to 12 - 0.25 step), sampled on a regular grid. Each emotion timeseries was then filtered using these ~5k Gaussian distributions and fitted in brain activity using a linear regression approach. This produced t-values (i.e., β/SE β) expressing the goodness of fit of μ and σ combinations, for each right TPJ voxel. The principal tuning of voxels was then obtained by selecting the combination characterized by the highest t-value across the ~5k samples (Figure 6).
To estimate the similarity between tunings (i.e., μ parameters) obtained from the pRF approach and our original results (i.e., β coefficients of the gradient estimation), we computed Spearman’s ρ across right TPJ voxels. The significance of such an association was tested against a null distribution of β coefficients obtained through the IAAFT procedure (N = 1,000).
Lastly, we further characterized the prototypical responses of populations of voxels as function of affective states. To do so, we used the non-negative matrix factorization84 and decomposed the multivariate pRF data (i.e., voxels t-values for each μ and σ) into an approximated matrix of lower rank (i.e., 10, retaining at least 90% of the total variance). This method allows parts-based representations, as the tuning of right TPJ voxels is computed as a linear summation of nonnegative basis responses. The results of this procedure are summarized in Supplementary Figure 9. All the analyses were performed using MATLAB R2016b (MathWorks Inc., Natick, MA, USA).
Author Contributions
G.L., G.H. and L.C., conceived the study, designed the behavioral experiment, developed the code, performed behavioral and fMRI data analysis. G.L., G.H., L.C. and P.P. interpreted the obtained results and drafted the manuscript. A.L., Pa.P. and M.B. contributed to the fMRI data analysis. L.C., E.R. and P.P. critically revised the manuscript. All the authors approved the final version of the manuscript.
Data Availability
The data that support the findings of this study are available from OpenScience foundation, https://osf.io/tzpdf. Raw fMRI data are available at http://studyforrest.org. Real life experience-sampling dataset is available at https://osf.io/zrdpa. Portrayed emotions dataset is available at https://f1000research.com/articles/4-92/v1.
Code Availability
The code and the preprocessed data are publicly available at OpenScience foundation (https://osf.io/tzpdf, doi:10.17605/OSF.IO/TZPDF).
Competing Interests
The authors declare no competing interests.
Acknowledgements
We would like to thank all the people behind the studyforrest project, especially Michael Hanke, Annika Labs and colleagues for sharing the Forrest Gump portrayed emotions data and Mark A. Thornton and Diana I. Tamir for making available the real life emotion transitions data. M.B. was in part supported by the PRIN (Research projects of national interest) 2015 – Italian Ministry of Education, University and Research (MIUR) Prot. 2015AR52F9 granted to P.P.
References
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.↵
- 6.↵
- 7.↵
- 8.↵
- 9.↵
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.
- 24.
- 25.↵
- 26.↵
- 27.↵
- 28.↵
- 29.↵
- 30.↵
- 31.
- 32.↵
- 33.↵
- 34.↵
- 35.↵
- 36.
- 37.↵
- 38.↵
- 39.↵
- 40.↵
- 41.↵
- 42.↵
- 43.↵
- 44.↵
- 45.↵
- 46.↵
- 47.
- 48.↵
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.↵
- 54.↵
- 55.↵
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.↵
- 67.↵
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.↵
- 78.↵
- 79.↵
- 80.↵
- 81.↵
- 82.↵
- 83.↵
- 84.↵