ABSTRACT
The ability to process time on the scale of milliseconds and seconds is essential for behaviour. A growing number of studies have started to focus on brain dynamics as a mechanism for temporal encoding. Although there is growing evidence in favour of this view from computational and in vitro studies, there is still a lack of results from experiments in humans. We show that high-dimensional brain states revealed by multivariate pattern analysis of human EEG are correlated to temporal judgements. First, we show that, as participants estimate temporal intervals, the spatiotemporal dynamics of their brain activity are consistent across trials. Second, we present evidence that these dynamics exhibit properties of temporal perception, such as the scalar property. Lastly, we show that it is possible to predict temporal judgements based on brain states. These results show how scalp recordings can reveal the spatiotemporal dynamics of human brain activity related to temporal processing.
Introduction
Humans and non-human animals are able to estimate temporal intervals across a wide range of scales1, 2. Intervals ranging from hundreds of milliseconds to seconds are specially critical for sensory and motor processing, learning, and cognition. Although cognitive models that propose the existence of some version of an internal clock have been effective in providing a framework for much of the existent behavioural data3–5, there is still a lack of electrophysiological and anatomical evidence to support their theoretical assumptions. For this reason, a number of alternate models of timing, which take into account neural data, have been put forward as biologically-plausible explanatory candidates, such as state-dependent networks models6. For this class of models, neural circuits would be inherently capable of temporal processing as a result of the natural complexity of cortical networks coupled with the presence of time-dependent neuronal properties. However, the vast majority of evidence in favour of these models rely on computational modelling or in vitro studies7–10 and evidence from human and non-human animal recordings are still sparse11,12.
In human electroencephalography (EEG), the majority of studies that investigate temporal perception have focused on the contingent negative variation (CNV), a slow cortical potential of developing negative polarity that peaks at the reference interval13. However, the relation between the CNV and temporal processing has been recently criticised as it is not clear whether its ramping activity is coding time or using temporal information to anticipate or react to events14. Importantly, no studies have investigated whether state-space trajectories, as proposed by state-dependent networks models, can be characterised using non-invasive human electrophysiological recordings (EEG) and track relevant temporal information.
On the other hand, recent EEG studies have suggested that it is possible to differentiate spatially overlapping brain states by analysing subject-specific patterns15–19. This methodology has been successfully applied to magneto-encephalographic recordings and was able to dissociate between standard and deviant tones, frequent versus rare melodies, visual stimulus location and stimulus orientation15. In the present study, we analysed the time-resolved EEG signals using multivariate pattern analysis (MVPA), to investigate whether the evolution of brain states over time can carry information about the temporal interval tracked by participants and also their future behavioural responses.
Results
Human participants (n = 14) performed a temporal categorisation task, in which they had to judge if an interval delimited by two brief tones was longer, shorter or equal to 1.50 seconds (Figure 1a). In each trial, the interval was chosen randomly from the following set: 0.80,0.98,1.22,1.50,1.85,2.27 and 2.80 seconds (log scale around 1.50 seconds). As expected, participants made few errors when categorising intervals that were clearly shorter, equal or longer than 1.50 seconds (Figure 1b). Participants demonstrated highest uncertainty (point of subjective equality, PSE) between shorter and equal responses for intervals around 1.20 ± 0.03s (mean ± s.e.m) and between equal and longer responses for intervals around 2.10 ± 0.05s (paired t-test between estimated PSEs, t13 = 15.05, p <0.001).
A hallmark of interval timing is its scalar property (a generalisation of the Weber Law), which states that the variability in temporal estimations increases linearly with the magnitude of the interval estimated. That is, errors in temporal estimations scale with the durations of the intervals3, which implies that shorter intervals are easier to discriminate than longer intervals. Accordingly, in our experiment, participants were more sensitive (i.e responded differentially) to small temporal differences across shorter intervals compared to longer intervals. This is illustrated by the smaller Just Noticeable Difference (JND) for the shorter compared to the longer intervals (JNDshort = 0.287 ± 0.031s, JNDlong = 0.438 ± 0.032s, paired t-test, t13 = 4.77, p <0.001). As predicted by the scalar property, when sensitivity was normalised by interval length no difference between shorter and longer intervals was observed (Webershort = 0.118 ± 0.013, Weberlong = 0.103 ± 0.006, paired t-test t13 = −1.39, p = 0.187). For the EEG recordings, we focused our analysis on trials in which the interval was at least 2.27s long, collapsing the data from the two longest intervals. The event-related potential for central electrodes can be seen in Figure 1c. Consistent with previous results, there was a clear Contingent Negative Variation potential (CNV, a slow cortical potential of developing negative polarity) which peaked at the reference interval13, 14.
For a neural system to be able to read time by its trajectory through state space, the trajectory of the activity elicited by a target must be consistent across activations. Thus, we checked if the recorded dynamics were consistent across trials. Indeed, the pattern of the EEG signals across the scalp followed a structured sequence in time during the different trials (Figure2a). Next, we used the Mahalanobis distance19, 20 to perform pair-wise comparisons across time points to determine whether the pattern of the EEG signal contained information about the interval between events. As shown in Figure 2b, multivariate distances between time points followed a diagonal-shaped pattern (i.e a stronger similarity across points closer in time), suggesting a sequential activation of overlapping states15.
The evolution of the spatiotemporal dynamics captured by the EEG sensors can be visualised using multidimensional scaling and plotted against the first three dimensions. This trajectory represents the path linking the sequence of activation states, while the multidimensional distance between time points in state space reflects the difference in the overall response captured by all sensors. In line with the pairwise distances showing similarity in activations for points closer in time, the recovered trajectory preserves the temporal information of the activated states (Figure 2c). Although there is a quick transition of states at the beginning of the interval, as time gets closer to the possible moments in which the interval ends, there is roughly a one-to-one correspondence between temporal distance and state distance.
The velocity with which states change and travel through state space can also be calculated using Mahalanobis distances21. After a quick transition through states following the first temporal marker, the velocity of such transitions continuously decreased over time, suggesting that as time passed, changes in state occurred at a smaller rate (t-test on the estimated slopes, t13 = −9.59, p <0.001, Figure 2d and Figure 2e). This is once again consistent with the scalar property of time: as interval length increases, change rate in state space decreases, which in turn decreases the resolution of longer intervals, making it harder for participants to discriminate between longer intervals. Importantly, when the distance between two states was corrected by the estimated Weber-fraction, there was no significant difference in distance travelled as a function of interval (one-way repeated measures ANOVA with Interval as the main factor, F5,65 = 1.663, p = 0.185, Figure 2f).
To quantify the relation between state space and behaviour, we focused our analysis at time points when the interval could have ended. We performed multivariate pairwise comparisons (using Mahalanobis distances) on data for the six first intervals (0.80,0.98,1.22,1.50,1.85,2.27 seconds) and used multidimensional scaling to represent them in a two dimensional plot (Figure 3a).
To quantify the relationship between distance in time and in state space, we fitted linear functions to the log transformed temporal distance with the state space distance, comparing the first and last interval to all others. As shown in (Figure 3b), there was a strong association between both distances, suggesting that states further apart in time are also further apart in state space (fitted slope for shortest interval = 1.58±0.08, t-test to zero: t13 = 19.48,p = 0.001, fitted slope for longest interval = −1.29±0.09, t-test to zero: t13 = −13.50, p = 0.001). Importantly, the slope for the shortest interval is steeper than for the longest interval (paired t-test on the absolute estimated slopes, t13 = 3.348, p = 0.005). This suggests that, for the shortest interval, the rate of change in state space as a function of time is higher than for the last interval, once again in accordance with the scalar property of time.
To compare this difference in a more direct way, we tested the distance in state space between two conditions that had identical distances in time. Specifically, we compared whether the distance in state space between the first and third interval (0.8s and 1.22s, being 420ms apart) was similar to the distance in state space between the sixth and the fifth interval (2.27s and 1.85s, also 420ms apart). Consistent with the scalar property of time, we found that the difference in state space was larger for the first comparison than for the second (t13 = 2.43 , p = 0.030).
Thus far, we have shown that the properties of brain states revealed by MVPA follow key characteristics of temporal processing. However, if they are related to temporal perception, then it should be possible to decode participants’ temporal judgements based on the recovered states. To test this possibility, we focused our analysis on single trial data from the two intervals closest to when participants had highest uncertainty about their temporal judgement: (1) 1.22s, for which participants had a high uncertainty on whether it was shorter or equal to 1.50s; and (2) 1.85s, for which participants had a high uncertainty on whether it was equal or longer than 1.50s. When participants had a high uncertainty between shorter/equal responses, positions within the path of state space that were closer to the equal state increased the probability of equal responses (t-test on the estimated slopes, t12 = 2.28, p = 0.041). When participants had a high uncertainty between equal/longer responses, positions within the path of state space that were further to the equal state decreased the probability of equal responses (t-test on the estimated slopes, t12 − = −2.81, p = 0.016). Within each case, the physical interval tested was identical in all trials, showing how trial-by-trial fluctuations in the distance travelled in state space can partially account for the variability in temporal judgements.
Discussion
In this study, we investigated whether time-resolved EEG signals can carry information about time. We have shown how dynamic pattern analyses can characterise states correlated with coding of temporal intervals. These states show the properties classically related to temporal perception, such as scale invariance, and were predictive of temporal judgements.
A growing number of studies have investigated if the neural mechanisms of temporal perception in the range of hundreds of milliseconds to seconds are related to population dynamics instead of a single central clock1, 11, 22. According to this view, temporal information emerges from neural properties that are naturally time-varying, such as short-term synaptic plasticity23. Our results show, for the first time to our knowledge, how these dynamics can be captured by non-invasive electrophysiological recordings, addressing an important lack of evidence in favour of this population dynamics view in humans.
The majority of timing studies in humans have focused on the contingent negative variation (CNV) as a potential signature of the subjective experience of time13. However, the exact role of the CNV on temporal perception has been recently criticised both on experimental and theoretical terms14, 24. For example, a recent study has argued that post interval evoked potentials (e.g activity evoked by a stimulus that marks the end of an interval) can reflect the subjective experience of time better than the CNV24.
Our results, on the other hand, suggest that temporal experience can be tracked during the interval to be estimated. Although the averaged amplitude of the CNV in a group of electrodes might not be a good predictor of the elapsed interval, other subtle differences present in individual electrodes, and possibly hindered by the CNV, can actually carry information about time. When a network is activated it will follow a complex trajectory dependent not only on the synapses directly activated by the input, but also on the ongoing activity of the network6. This process leads to distinct spatiotemporal patterns of activity, which, according to our results, produce different patterns of activation across the EEG sensors. This finding adds to the increasing evidence of how multivariate methods represent a powerful approach to decode task-relevant dimensions.
As these trajectories evolve through time in a consistent way, stimuli presented in different time points will find the network in a different state leading to different activity patterns6. Therefore, although small differences in the state of a network might not be detectable using classical EEG methods, they could still lead to marked differences of evoked activity by new inputs in different time points, as recently reported24.
One important question that remains to be answered is whether these recovered brain states are part of a coding scheme used to track time or a by-product of other processes that could generate a time-decodable signal. Although we have found that the decoded brain states in single trials are correlated with participants’ temporal judgements, one could argue that the majority of neural processing has a temporal structure. If this structure is consistent enough, even if not directly used to track time, it could lead to similar results reported by our study. Future studies should address this issue by observing how those brain states behave in tasks that have similar temporal structures, but different temporal demands. There is a myriad of temporal tasks and illusions that can be used to study time in humans. Future studies can combine theses tasks and use similar methods herein described to elucidate how temporal information can be encoded at the population level to support time-dependent functions.
Methods
Participants
Sixteen volunteers (age range, 22 − 27 years; 9 female) gave informed consent to participate in this study. All of them had normal or corrected-to-normal vision and were free from psychological or neurological diseases. The experimental protocol was approved by The Research Ethics Committee of the Federal University of ABC. Two participants were excluded from the analyses because more than 3 channels during EEG recordings presented an increased number of artifacts.
Stimuli and Procedures
The experiment consisted of a computerized “shoot the targetx” task25. The stimuli were presented using Psychtoolbox v.3.0 package for MATLAB on a 17−inch CRT monitor with a vertical refresh rate of 60Hz, placed 50cm in front of the participant. Responses were collected via a response box with 9 buttons (DirectIN High Speed Button; Empirisoft). Each trial started with the presentation of a target (1.5 visual degrees radius, red and black) at the left hemifield of the screen (background RGB-colour 150; 150; 150) and an “aiming sight” (an empty circle with 0.5 visual degree radius) at the centre of the screen. After a random interval (500ms-1000ms) a beep (1000Hz, 70dB, 100ms duration) was presented simultaneously with the start of movement of the target from left to right. The target moved at a constant speed (9 degrees/sec) taking 1.50 seconds to reach the centre of the screen, thus passing through the aiming sight. In Regular trials (350 trials) a button press (with the right index finger) produced a “shot” (a green disc, presented inside the aiming sight) and a second beep (500Hz, 70dB, 100ms duration), presented simultaneously.
Participants were instructed to hit the target by pressing the button at the appropriate moment. Test trials followed the same schema, but the trajectory of the target was masked by a gray rectangle (3 visual degrees of height, RGB-colour 130; 130; 130) and automatic shots were given at seven different intervals (0.80,0.98,1.22,1.50,1.85,2.27or2.80 s). In each Test trial, participants had to judge whether the shot occurred at an interval “shorter”, “equal”, or “longer” than the time the target normally took to reach the screen centre. Participants responded using three different buttons. Responses in Test trials were unspeeded and could be given starting 800ms after the automatic shot. Trials were presented in a random order, with the restriction of having a maximum of three Test trials in sequence. Participants performed a total of 10 blocks, each with 35 Regular and 35 Test trials. The first 10 trials in each block were always Regular trials. The experimental session lasted 60 minutes on average.
EEG recordings and pre-processing
EEG was recorded continuously from 64 ActiCap Electrodes (Brain Products) at 1000Hz by a QuickAmp amplifier (Brain Products). All sites were referenced to FCz and grounded to AFz. The electrodes were positioned according to the International 10 −10 system. Additional bipolar electrodes registered the electrooculogram (EOG). EEG pre-processing was carried out using BrainVision Analyzer (Brain Products). All data were down-sampled to 250Hz, re-referenced to the average of all electrodes, filtered (0.05Hz to 30Hz) and epoched from 500ms before the first beep to 1000ms after the second beep. An independent component analysis (ICA) was performed to reject eye movement artifacts. Eye related components were identified by comparing individual ICA components with EOG channels and by visual inspection. The number of trials rejected for each participant was small (13% on average). ERP analysis were performed on data using the SPM8 and Fieldtrip toolboxes for MATLAB. The CNV for Figure 1c was estimated at central-parietal electrodes (C3,C1,Cz,C2,C4).
Behavioural Analysis
Behavioural analysis was based on the proportions of each type of response as a function of interval. For each participant, two independent sigmoidal functions were fitted to the data. The first function was fitted on the proportion of “shorter” responses and the second on the proportion of “longer” responses. The psychometric data from each participant and condition were fitted with Cumulative Normal functions, each defined by four parameters: threshold, slope, lapse-rate and guess rate26. Guess rates were fixed at 0 across all participants and conditions, and lapse rates were restricted to a maximum of 0.05. The three free parameters were fitted separately for each participant. The points of highest uncertainty were estimated as the predicted interval corresponding to 50% of “shorter” responses and the predicted interval corresponding to 50% of “longer” responses. Quality of fit for each participant was assessed by correlating predicted values to observed responses (r-square 0.96±0.01; lowest individual r-square = 0.88). The analysis of the psychometric function was performed using the Palamedes toolbox for MATLAB. To measure temporal sensitivity for shorter and longer intervals, we calculated Just Noticeable Difference (JND) scores, which represents the absolute difference in seconds between the intervals at which 25% and 75% of shorter or longer responses were given. Weber fractions were estimated as:
Multivariate Pattern Analysis
Similarity between different trials
To calculate the similarity of the sequence of patterns activated in each trial, we used a leave-one-trial-out approach. Specifically, for each participant and trial, data from each time point (a row vector with the amplitude values of all 62 electrodes) was compared to the averaged data of all remaining trials (also a row vector with the mean amplitude values of all 62 electrodes at the same point), using a Pearson correlation (Rho’). After calculating these similarities for each trial, the Rhos’ were Fisher transformed and averaged across trials for each participant. At the group level, Rhos’ were compared using a paired t-test and p-values were corrected using a false discovery rate (FDR) procedure27.
Dissimilarity Matrix
To determine whether the pattern of the EEG signal across channels contained information about the elapsed time, we used the Mahalanobis distance to perform pair-wise comparisons between different time points. Several studies have shown that the Mahalanobis distance is superior to Euclidean distance because it accounts for the covariance structure of the noise between features19, 28. For each participant, data from the two longest intervals (2.27 and 2.88) were collapsed and averaged. Data were smoothed with a Gaussian kernel (SD = 20ms). The pairwise multivariate dissimilarity (Mahalanobis distance) of each time point to all others were calculated as follows: where EEGti and EEGtj are row vectors containing the average signals of the two time points being compared. The pinvC is the pseudo inverse of the error covariance matrix, estimated by pooling over the covariances of both time points being compared, estimated from all trials, using a shrinkage estimator that is more robust than the sample covariance for data sets with many variables and/or few observations19, 28. This procedure was performed for all time points, from 0 to 2.27 seconds, in 4ms bins. Pairwise multivariate dissimilarity matrices were estimated separately for each participant, and the grand mean dissimilarity matrix was calculated by averaging individual matrices across participants.
To estimate the 95% quantile of the estimated Mahalanobis distances (black lines in Figure 2b), a permutation procedure was used. For each participant and iteration: (1) two random time points were chosen; (2) trials between them were randomly mixed; (3) the Mahalanobis distance was calculated between the intermixed conditions. After performing 1000 iterations for each participant, the distances were averaged across participants and the 95% quantile was estimated.
Multidimensional Scaling
Trajectory in state space was visualised using metric multidimensional scaling (MDS) as implemented in MATLAB. In Figure 2c, MDS was performed on the grand mean pairwise dissimilarity for all time points matrix and data was plotted against the first three dimensions (stress = 0.056). For figure 3a, MDS was performed on the grand mean pairwise dissimilarity for possible intervals and data was plotted against the first two dimensions (stress < 0.001)
Velocity
Instantaneous velocity at time t was estimated by calculating the difference in activity state as a function of time:
Here, we used n = 40ms. Velocity was calculated separately for each participant. To calculate if velocity changed at the moments of possible target presentation, velocity was calculated for the six possible moments of target presentation. For each participant, a linear regression was performed and, at the groups level, the estimated linear coefficients were compared to zero using a paired t-test.
To test whether distances in state space were similar when the scalar property was taken into account, we first estimated the mean Weber-fraction as the average of all Weber-fractions estimated previously as described above (mean Weber-fraction= 0.1106±0.008). Distances for each possible moment of target presentation were calculated in a similar manner as previously:
The values of n were calculated as a function of their respective intervals (0.80,0.98,1.22,1.50,1.85,2.27s) multiplied by the estimated Weber-fraction, resulting in the following values of n for each interval: [0.0442,0.0547,0.0669,0.0829,0.1018,0.1255]s. To calculate if these distances varied at the moments of possible target presentation, distances were submitted to a one-way repeated measures ANOVA with interval as a main factor.
Correlation between dissimilarity and performance
To calculate how single trial trajectories correlated with performance, we focused on the two intervals closest to the points of highest uncertainty (1.22s and 1.85s). To measure where in the state-space trajectory participants were in each trial, we compared single-trial states to two other landmarks states, allowing us to estimate the location in the state space where participants were when the interval ended. The landmarks consisted of the averaged data from pre-stimuli period (–100 to 0, relative to target onset) for the 0.8s interval (short landmark), 1.5s interval (equal landmark) and 2.27s (longer landmark). Importantly, these landmarks were estimated on the averaged data of the two longest intervals (collapsing 2.27s e 2.8s, as done previously). Thus, the single-trial data and the data used to estimate the landmarks were completely independent.
For the 1.22s, data from the last 100 ms before target presentation (1.12 –1.22s) was averaged, resulting in a row vector for each trial. This was then compared to the short landmark and the equal landmark, using the Mahalanobis distance as previously described. The subtraction D(1.22,short) – D(1.22,equal) resulted in an index of how similar that state in that particular trial was to each landmark. Trials where the state is more similar to the equal landmark yields higher values. If these distances are correlated to performance, then the proportion of equal responses should increase as this index increases. To test this hypothesis, the estimated distance values were used as predictors in a generalised linear model regression, with a probit link for the binomial distribution. This procedure was performed separately for each participant and at the group level the estimated slope coefficients were tested against zero using a paired t-test. Similarly, we compared single-trial data for 1.85s to equal and longer landmarks and used the difference D(1.85,equal) − D(1.85,longer). In this case, higher values indicate a stronger similarity to the longer state and should be correlated with a decrease in the proportion of equal responses. In both cases, data from one participant was removed due to a small proportion of one of the response types (< 10%).
Acknowledgements
This work was supported by the Fundacção de Amparo aà Pesquisa do Estado de São Paulo (FAPESP) research grants 15/00794-2, 15/04554-6 and 13/24889-7 and by Conselho Nacional de Pesquisa (CNPq) research grant 122942/2014-0. The authors would like to thank Dean Buonomano, Nicholas E. Myers and the members of the Timing and Cognition Laboratory at UFABC (http://neuro.ufabc.edu.br/timing/) for useful discussions and suggestions on earlier versions of this manuscript
Author contributions statement
A.M.C., M.S.C., F.D.B and V.C.M. conceived the experiment. F.D.B. and V.C.M. performed the experiments. All authors analysed the data, wrote and reviewed the manuscript.
Additional information
Competing financial interests: The authors declare no competing financial interests