Abstract
Visual attention dramatically improves subjects’ ability to see and also modulates the responses of neurons in every known visual and oculomotor area, but whether those modulations can account for perceptual improvements remains unclear. We measured the relationship between populations of visual neurons, oculomotor neurons, and behavior during detection and discrimination tasks. We found that neither of the two prominent hypothesized neuronal mechanisms underlying attention (which concern changes in information coding and the way sensory information is read out) provide a satisfying account of the observed behavioral improvements. Instead, our results are more consistent with the novel hypothesis that attention reshapes the representation of attended stimuli to more effectively influence behavior. Our results suggest a path toward understanding the neural underpinnings of perception and cognition in health and disease by analyzing neuronal responses in ways that are constrained by behavior and interactions between brain areas.
Introduction
Each of the huge number of psychophysical and physiological studies of visual attention show that attention profoundly affects subjects’ perceptual abilities and also modulates the responses of populations of neurons at every stage of visual and oculomotor processing1-4, Despite these oft replicated observations, whether any of the observed neuronal modulations can account for the improvements in psychophysical performance remains unknown. Two, non-mutually exclusive, hypotheses have dominated the literature (Figure 1A): that attention 1) improves visual information coding5-7, or 2) improves the efficiency with which visual information is read out by the premotor neurons involved in decision-making 8-11. The studies used to support these hypotheses were limited by available data and analysis methods, which primarily involved the responses of single neurons or pairs of simultaneously recorded neurons in the same brain area. We evaluated these hypotheses using the responses of groups of simultaneously recorded neurons in multiple stages of visuomotor processing, psychophysics, and data analysis methods that leverage that unique combination. We recorded simultaneously from groups of neurons in area MT, which encodes motion information 12,13 and the superior colliculus (SC), where neuronal responses are either visual, oculomotor, or intermediate, contribute to gaze control 14-16 and are involved in computing perceptual decisions 17-19. When we analyzed the responses of single neurons or pairs of neurons, we replicated previous observations, including the results from two of our previous studies, which focus on visual area V4 in two different tasks with spatial attention components: an orientation change detection task 5 and a contrast discrimination task 6. However, constraining our analyses of our MT data set or of both V4 data sets by the animals’ behavior and the simultaneous recordings from both areas made it clear that neither prior hypothesis constitutes a satisfying account of the observed attention-related improvements in performance.
Our results suggest that on the timescale of perceptual decisions, across two visual areas and during both detection and discrimination tasks, spatial attention does not act primarily by improving information coding or by changing the way visual information is read out. Instead, the long-observed attention-related changes in the responses of visual cortical neurons account for perceptual improvements, but they do so by reshaping the representation of attended stimuli such that they more effectively drive downstream neurons and guide behavior (Figure 1B). Our study provides a framework for leveraging multi-neuron, multi-area recordings and controlled psychophysics to study how neuronal networks mediate flexible behavior in many systems, timescales, and tasks.
Results
We compared evidence for and against two hypothesized attention mechanisms using neuronal responses collected while two rhesus monkeys performed the widely studied motion direction change-detection task in Figure 1C5,9,20-22, and then compared the results to recordings while monkeys performed a similar orientation change detection task 5 and a contrast discrimination task 6. As in the two previously published data sets, the animals’ performance in our new experiment was greatly affected (Figure 1D) by a cue instructing them to shift spatial attention between a stimulus within the same or opposite hemifield as the joint receptive fields of several dozen neurons that were recorded on multielectrode probes in MT (Figure 1E, red points) and the SC (blue points). MT and the SC represent different stages of perceptual decision-making and therefore provide the opportunity to evaluate each hypothesized attention mechanism. MT contributes to motion perception12,13. The SC is thought to play many roles in visually guided tasks including gaze control 14-16, decision-making 17-19 and attention 4.
Population recordings replicate previously observed effects of attention
The two predominant attention hypotheses make different predictions about how attention should affect MT and the SC in our task. The first (information coding) hypothesis predicts that attention improves the motion direction information encoded in MT. The second (readout) hypothesis posits that attention changes the way that stimulus information is read out of MT to influence downstream responses and ultimately behavior. Our strategy was to show that our data are consistent with those in past studies by replicating the results that have been used as evidence to support each hypothesis and then to evaluate each hypothesis using analyses that leverage our simultaneous measurements from the subjects’ behavior and multi-neuron, multi-area recordings.
Past studies have evaluated these hypotheses by analyzing the responses of individual neurons or pairs of neurons, which typically lack the statistical power to reveal a strong link to behavior. Using our data set, we replicated the observations that have been used as evidence in favor of each hypothesis. Consistent with previous studies evaluating the information coding hypothesis 2,3,23, we found that attention increased the trial-averaged responses of neurons in both MT and the SC (Supplementary Figure 1A and B) and that attention decreased the extent to which the trial to trial fluctuations in neuronal responses to repeated presentations of the same stimulus are shared between pairs of MT neurons5,7,21 (quantified as the average spike count or noise correlation, or rsc24; Supplementary Figure 1C). Consistent with studies evaluating the readout hypothesis, attention increases correlated variability between the two areas9,10,25 (Supplementary Figure 1C). This increase between areas suggests that attention-related effects are not simply due to global reductions in slow fluctuations, which has recently been hypothesized to explain attention-related correlation decreases within a single brain area26 (Supplementary Figure 2 and Supplementary Text). Further, this attention-related increase was weakly dependent on the visual responsivity of SC neurons (Supplementary Figure 3).
Neuronal population decoding methods provide incomplete support the information coding or readout hypotheses
We reasoned that analyzing the relationship between populations of simultaneously recorded neurons in multiple brain areas with the animals’ behavior would provide insight into the relative importance of each hypothesized mechanism. To this end, we determined whether attention affects the amount of stimulus information that can be decoded from the population of MT neurons using cross-validated linear decoders that are optimized to a) dissociate between the original and changed stimuli (Stimulus decoder in Figure 2), b) predict the animals’ choices (whether or not they made an eye movement in response to change stimuli; Choice decoder), or c) predict the activity of the population of SC neurons we recorded (using responses to the original stimulus; SC decoder). These decoders were always constructed using data from trials with the intermediate change amount (see Figure 1D).
The information coding hypothesis posits that attention improves the stimulus information that could be gleaned by an optimal Stimulus decoder, but our data provided only weak support for this idea. Attention did not significantly affect the performance of an optimal decoder in our data set, even when we used a decoder optimized separately for each attention condition (Figure 3A, left bars). Recent theoretical work has demonstrated that high-dimensional decoders can ignore pairwise correlations that are orthogonal to the decoding axis and that correlations are increasingly likely to be orthogonal to this axis in larger populations 27-29. This suggests that the effects of attention on the stimulus information that can be decoded from small neuronal populations like the ones we recorded are likely to be even more minimal for larger populations, making it seem unlikely that attention-related improvements in information coding account for the robust improvements in behavioral performance that we observed.
The readout hypothesis posits that attention changes the importance of the attended stimulus in guiding behavior by changing the way its representation is read out by the neurons involved in computing decisions. Therefore, this hypothesis posits that attention should change the weights relating MT responses to either behavior or SC responses. We found that attention had larger effects on the stimulus information that is related to the animals’ choices on individual trials (Figure 3A, middle bars) or that is shared with the SC (Figure 3A, right bars) than it did on the Stimulus decoder (p<.05). However, this difference could arise from either a weight change (Figure 1A) or a change within MT that results in more stimulus-related visual information being projected onto a static readout dimension (Figure 1B).
A new hypothesis: attention reshapes sensory activity so that it more effectively guides decisions
Our data do not support the hypothesis that attention changes weights relating MT responses to SC responses or behavior. Because the responses of MT neurons are correlated and because the behavioral readout is binary, the weights obtained by each decoder are non-unique, making it impossible to identify weight changes by analyzing the weights themselves 23,30. However, we can infer their stability by measuring the stimulus information gleaned by each decoder using weights from the opposite attention condition from which they were calculated (see Methods). Both the Choice and SC decoders gleaned more stimulus information from MT responses in the attended than unattended condition when we used the weights computed in the opposite attention condition (Figures 3B and 3C). Together, these neuronal population analyses that use the animals’ behavior and the activity of downstream neurons to assess the hypothesized attention mechanisms reveal that neither the information coding nor readout hypothesis provide a satisfactory account of the large observed attention-related behavioral improvement.
Our observations suggest that in MT neurons recorded while monkeys performing a change detection task, attention acts primarily by changing the visual information that is used to guide behavior using relatively fixed readout weights. To investigate the generality of these observations to different visual areas and different tasks, we tested these hypotheses using two additional datasets. In the first dataset, monkeys performed the same direction change detection described here while we recorded from populations of V4 neurons5. Similar to our results in MT, we found that attention had larger effects on the stimulus information that is related to the animals’ choices (Choice decoder; Figure 4A) than it did on the stimulus information that could be gleaned using an optimal (Stimulus) decoder (p<0.05; Figure 4B). As in our MT data set (Figure 3B), the results from this data set suggest that attention typically reshapes V4 responses to align with relatively fixed readout mechanisms: decoding performance was typically better using the V4 responses from the cued condition and the Choice decoder weights from the uncued condition (y-axis) than using the V4 responses from the uncued condition and the Choice decoder weights from the cued condition (p<0.05).
In the second new data set, we searched for attention-related changes in information coding in V4 neurons while monkeys performed a discrimination task6. These data provide a particularly important test of the information coding hypothesis because unlike in the change detection task in which attention has fairly uniform effects on V4 and MT neurons (increasing rates and decreasing noise correlations), we showed that in our discrimination task, attention can flexibly increase or decrease noise correlations in a way that is broadly consistent with improving information coding. Despite these findings, the results of our decoding analyses were similar for the detection and discrimination tasks, meaning that we did not find strong evidence that attention improves the amount of stimulus information that can be optimally extracted from a population of visual neurons in either task (Figure 4C, p=.09). Together, these results provide evidence that in multiple visual areas and visually-guided tasks, attention acts primarily to reshape population activity so that more stimulus information is used to guide behavior using relatively fixed decision mechanisms.
Our data support the hypothesis that attention reshapes the representation of attended stimuli to more effectively guide behavior (Figure 1B). In this scenario, the critical changes are in visual cortex. However, this reshaping does not result in a large improvement in the stimulus information that can be gleaned by an optimal Stimulus decoder. Instead, the modulated neuronal activity in MT better aligns with the readout dimensions using relatively static weights.
How could a reshaping of the representation of an attended stimulus be implemented? The simplest mechanism would make use of the oft observed signatures of attention such as changes in firing rate gain2,3,23 or pairwise noise correlations5-7,9,20-22,31-37. We investigated the possibility that these simple response changes can account for the attention-related improvement in the stimulus information decoded using both the Choice and SC decoders in two stages. First, to verify the prediction of the weight-swapping analyses (Figures 3B and 3C), we constructed a single Choice decoder for both attention conditions (Figure 5A) and determined that it captured the attention-related improvement in decoded stimulus information (compare the blue and yellow bars in Figure 5B; p< .05). Second, we used those same weights to decode stimulus information from population responses constructed using the mean rates from the uncued condition and the residuals from the cued condition (green bar). We found that simply using residuals (which incorporate both response variability that is private to each neuron and that which is shared between neurons) from the cued condition was enough to completely account for the attention-related improvement in decoded stimulus information in both the Choice (Figure 5B) and SC decoders (Figure 5C). These common decoders captured the attention-related improvement in decoded stimulus information (p< .05) and using residuals from the cued condition completely accounted for the attention-related improvement in decoded stimulus information (p= .84 for the Choice decoder and p=.48 for the SC decoder).
Discussion
We used multi-neuron, multi-area recordings and psychophysics in detection and discrimination tasks to test two previous hypotheses and one novel hypothesis about the relationship between attention-related changes in perception and in neuronal responses on the timescale of perceptual decisions. In contrast with the hypotheses motivating most of the extensive literature concerning the neuronal basis of attention, our data are most consistent with the novel hypothesis that attention reshapes population activity so that information about the attended stimulus is read out to guide behavior. Our conclusions are based on comparing the visual information that can be gleaned from decoders optimized for the stimulus, the animals’ choices, and the activity of groups of visuomotor neurons. These results support the idea that behavioral flexibility is mediated by reshaping the representation of visual stimuli rather than improvements in information coding (which may be impossible given the immense amount of sensory information encoded in the brains of even anesthetized animals29 or in the responses of single neurons13) or by changing read out, which may be comparatively difficult to flexibly alter on the approximately 150 ms timescale on which subjects can behaviorally shift attention38-40.
The idea of reshaping sensory information to better align with static read out mechanisms seems like it would require much more exotic mechanisms than the other hypothesized attentional mechanisms. However, we showed that commonly observed effects of attention on neuronal response variability were sufficient to reshape the representation of attended stimuli so that they more effectively influence the activity of downstream neurons and behavior (Figure 5B and 5C). Changing covariability may require a simpler mechanism than changing information coding or synaptic weights: we showed recently in a model that the covariability of a population of neurons can be readily changed by altering the balance of inhibition to excitation 41,42.
The idea that changing correlated variability better aligns sensory responses to a fixed readout is also consistent with our recent observation that in the change detection task, monkeys’ choices are well-aligned with the axis in population space that explains the most correlated noise 21. One exciting possibility is that the correlated variability axis represents the fixed readout dimension, perhaps because it is well-positioned to decode the motion direction of the broad set of stimuli that animals encounter outside the limited environment of most laboratory tasks 23. If so, reducing noise correlations and increasing firing rate gains would improve the stimulus information projected along that readout axis (following the intuitions in 43).
While our results were broadly consistent across two tasks and two visual cortical areas, it remains possible that attention uses different mechanisms in different tasks, brain areas, or sensory modalities. In particular, it is possible that the mechanisms underlying change detection, which is an important component of natural vision, are different than other tasks or that the mechanisms differ by brain areas. Therefore, the observation that attention also does not change the amount of stimulus information that can be decoded from visual cortex during a contrast discrimination task provides strong independent support for the generality of our findings. However, even if we happened upon a special, albeit common, scenario using these two tasks, it is remarkable to observe a situation in which the large attention-related change in behavioral performance can be accomplished without changing information coding or weights between areas. In contrast, theoretical models and machine learning techniques accomplish flexibility in computation almost solely by changing weights 44-47. Our results constitute an existence proof: an example of a situation in which flexibility can be mediated by simple changes within sensory cortex.
In the future, it will be interesting to use the same approach to determine whether similar mechanisms can account for behavioral changes associated with other cognitive processes (e.g. task switching) that might seem more likely to change the weights relating stimulus information to downstream neurons or behavior. Further, many neuropsychiatric disorders (including disorders of attention, Autism, and schizophrenia) are thought to involve changes in the same computations thought to underlie attention48. An exciting possibility is that these changes might be identified and potential therapies evaluated in animal models using the combination of behavioral evaluation and multi-neuron, multi-area recordings that we described here.
Online Methods
Methods Summary
The subjects of the simultaneously recorded MT and SC experiments were two adult male rhesus monkeys (Macaca mulatta, 8 and 9 kg). All animal procedures were approved by the Institutional Animal Care and Use Committees of the University of Pittsburgh and Carnegie Mellon University. Using linear 24 channel moveable probes (Plexon), we simultaneously recorded extracellular activity from direction-selective neurons in area MT and neurons in the superior colliculus that responded either visually, prior to a saccade, or both. Before beginning the experiment, we searched for neurons in both areas that had overlapping spatial receptive fields (Figure 1E) as determined by mapping with both drifting gratings and a delayed saccade task. The monkeys performed a direction change-detection task that commenced upon fixation of a central spot (Figure 1C). Two drifting Gabor stimuli, whose direction was selected to drive the recorded population of MT neurons well, flashed on and off until the direction of one stimulus changed at a random, unsignaled time. The monkeys signaled detection of the change by making a saccade to the changed stimulus within 450 ms of its onset. The location of the stimulus change was cued using instruction trials prior to each block of trials and the cue was valid 80% of the time. On the other 20% of trials, the change happened at the uncued location. In order to encourage fixation on longer trials, catch trials, where no stimulus changed direction and monkeys were rewarded for maintaining fixation, were intermixed.
Materials and Methods
The subjects of the simultaneously recorded MT and SC experiments were two adult male rhesus monkeys (Macaca mulatta, 8 and 9 kg). All animal procedures were approved by the Institutional Animal Care and Use Committees of the University of Pittsburgh and Carnegie Mellon University.
We presented visual stimuli using custom software (written in MATLAB using the Psychophysics Toolbox 49,50 on a CRT monitor (calibrated to linearize intensity; 1024×768 pixels; 120 Hz refresh rate) placed 54 cm from the animal. We monitored eye position using an infrared eye tracker (Eyelink 1000; SR Research) and recorded eye position and pupil diameter (1000 samples/s), neuronal responses (30,000 samples/s), and the signal from a photodiode to align neuronal responses to stimulus presentation times (30,000 samples/s) using hardware from Ripple.
Behavioral Task
As previously described5, a trial began when the monkey fixated a small, central spot within a 1.25° per side square fixation window in the center of a video display while two peripheral full contrast, drifting Gabor stimuli (one overlapping the receptive fields of the recorded neurons, the other in the opposite visual hemifield) synchronously flashed on (for 200 ms) and off (for a randomized period between 200-400 ms) until, at a random, unsignaled time, the direction of one of the stimuli changed from that of the preceding stimuli (Figure 1C). The monkey received a liquid reward for making a saccade to the stimulus that changed. Attention was cued in blocks of 50-100 trials, and alternated between blocks where attention was cued to either the left or the right stimulus. In each block, the direction change occurred at the cued stimulus on 80% of trials, and at the uncued stimulus in 20% of trials (all uncued changes used either the middle or largest direction change, Figure 1D). Catch trials, where no stimulus changed direction and the monkey was rewarded for maintaining fixation, were randomly intermixed throughout each block and made up approximately 12% of total trials. Psychometric data were fit with Weibull functions. Before recording commenced, the monkeys were extensively trained to have stable thresholds across a range of spatial locations (3-6 months). Because we recorded from several dozen neurons simultaneously, we could not optimize the stimuli for all neurons. We made sure to position one Gabor stimulus in the joint receptive field of the recorded neurons in both areas and we made an effort to set the properties of the size (approximately 3-6 degrees of visual angle), speed (approximately 3-12 degrees of visual angle per second) and direction of the stimuli so that they drove as many MT units as possible. The direction of all of the stimuli prior to the direction change (termed original stimulus) was constant throughout a recording session and this direction was typically either the median or mode of the distribution of MT preferred directions from that session. The range of direction changes differed from session to session, was selected based on the animals’ training history and depended on stimulus properties such as eccentricity and size. A typical range of change amounts for both animals was 1-35 degrees in log-spaced steps.
Electrophysiological Recordings
We simultaneously recorded extracellularly from single units and sorted multiunit clusters (the term “unit” refers to either). The dataset consisted of a total of 306 responsive MT units and 345 responsive SC units total (36-58 units per session, mean 20 in MT, 24 in the SC for Monkey HO; 36-53 units per session, mean 21 in MT, 22 in SC for Monkey ST) in both MT and the SC in the right hemisphere using moveable, linear 24-channel V-probes (Plexon; inter-electrode spacing in MT = 50μm, SC = 100μm). We presented visual stimuli and tracked eye position as previously described9. The data presented are from 6 days of recording for Monkey HO and 9 days of recording for Monkey ST. Each day consisted of multiple blocks of the attention task (Figure 1C; mean 1015 of trials for Monkey HO, 745 for Monkey ST) preceded by receptive field mapping using a delayed saccade task and direction tuning during passive fixation.
Data Analysis
All spike sorting was done offline manually using Offline Sorter (version 3.3.5; Plexon). We based our analyses on both single units and multiunit clusters and use the term “unit” to refer to either. Neuronal analyses in Supplemental Figure 1 and 2 used spike count responses between 50-250 ms after stimulus onset to account for visual latencies in the two areas. To remove response contamination from eye movements during change stimuli, data presented in the decoding analyses in Figure 3 and 4 used shorter response windows. Responses to both original and changed stimuli were measured from 50-185 ms after stimulus onset for monkey HO and 50-220 ms for monkey ST. These times were selected based on the distribution of each animal’s reaction times with the goal of maximizing the number of trials that could be included in the analyses. Trials with reaction times that began during those windows were excluded. Using these shorter response windows did not qualitatively affect the measures of attention described in Supplemental Figure 1. Attention still increased the firing rates of MT (mean attention index = 0.034, median attention index = 0.034), Wilcoxon signed rank test, p< 10−18) and SC neurons (mean attention index = 0.071, median attention index = 0.05), Wilcoxon signed rank test, p< 10−44) and decreased noise correlations within MT (Wilcoxon signed rank test, p< 10−72). To minimize the impact of adaptation on our results, we did not analyze the first stimulus presentation in each trial. We only analyzed a recorded MT unit if its stimulus-driven firing rate was 10% higher than its firing rate as measured in the 100 ms prior to the onset of the first stimulus. We only analyzed a recorded SC unit if its stimulus-driven firing rate was 10% higher than its firing rate as measured in the 100 ms prior to the onset of the first stimulus or if its response during a 100 ms epoch prior to a saccade on hit (correct) trials to the contralateral side was 10% larger than that same baseline. Stimulus presentations during which a microsaccade was detected were excluded from analyses9,51).
For firing rate analyses in Supplementary Figure 1A and B, attention indices were calculated using average spike counts on the (original) stimulus presentation prior to correct detections of the intermediate change amount depending on whether attention was directed into or out of the receptive fields of the recorded neurons using the formula (attendin – attendout)/(attendin + attendout). Significance of individual units was determined by a paired t-test (p<0.05).
Noise correlations
We defined the correlated variability of each pair of simultaneously recorded units (quantified as spike count correlation or rSC 24) as the Pearson correlation coefficient between the responses of the two units to repeated presentations of the same stimulus. This measure of rSC represents noise correlations rather than signal correlations because the responses used in this analysis were always to an identical visual stimulus. For Supplementary Figure 1C, we included responses from stimulus presentations 2 though 10 from trials that ended with either a hit, miss or correct catch trial and that were immediately followed by the maintenance of fixation and continuation of the trial (i.e., stimulus presentations where the behavioral response on the subsequent stimulus presentation was not a saccade). We z-scored responses as a function of the stimulus presentation number in each trial and then pooled data across stimulus presentations before calculating noise correlations. Results did not qualitatively change if we did not perform this z-score procedure. For Supplementary Figure 1D, we included data from all stimulus presentations prior to the change stimulus (except the first) and sorted them depending on what the behavioral outcome was on the subsequent stimulus presentation. Pairs of units that were recorded on the same electrode were not included in correlation analyses. The data presented in Supplementary Figures 1C consisted of 3,285 MT pairs, 3,948 SC pairs and 6,934 between area pairs.
Decoding
We focused our decoding analyses (Figures 2, 3 and 5) on trials in which the third largest (middle) direction change occurred, because changes of that magnitude occurred in both attention conditions. This approach also serves to linearize the problem by attempting to classify between one of two directions of motion. Therefore, we have restricted our decoding approach to using linear methods. We performed the decoding analyses using responses from trials that were either hits (correct detection) or misses (maintained fixation after change stimulus) only from data sets for which there were at least 10 trials in each attention condition and at least three hits and three misses in each condition. These criteria led us to exclude one data set. We did not include false alarms in the analyses because there were too few (and they were too inconsistent across recording sessions) to handle appropriately.
We used the decoding strategy schematized in Figure 2. We began by constructing a matrix of MT responses for each attention condition: ‘MT responses’ (a # MT neurons by 2*# trials matrix of MT responses to the stimuli before the direction change and the changed stimulus on the relevant trials). The stimulus decoder was performed using two matrices for each attention condition: all of ‘MT responses’ (a # MT neurons by 2*# trials matrix of MT responses to the stimuli before the direction change and the changed stimulus on the relevant trials) and ‘motion direction’ (a 1 by 2*# trials vector of zeros for the stimulus before the change, referred to as ‘original’, and ones for the changed stimulus, referred to as ‘change’). The Choice decoder was performed using two matrices for each attention condition: the responses during change stimulus presentations from ‘MT responses’ (a # MT neurons by 1*# trials matrix of MT responses to the change stimulus on the relevant trials) and ‘choice’ (a 1 by 1*# trials vector of zeros for change stimulus presentations on which the animal did not make an eye movement, referred to as ‘no saccade’, and ones when the animal made an eye movement, referred to as ‘saccade’). The SC decoder was performed using two matrices for each attention condition: the responses during original stimulus presentations ‘MT responses’ (a # MT neurons by 1*# trials matrix of MT responses to the original stimulus on the relevant trials) and ‘SC responses’ (a # SC neurons by 1*# trials matrix of SC responses to the original stimuli on the relevant trials). We refer to this final decoder as an ‘SC decoder’ but the weights are defined with no directionality: we have simply identified the weights that best relate the activity between the two areas.
We cross validated by holding out the two stimulus presentations from ‘MT responses’ (for the original and changed stimuli) from one trial at a time to perform the classification of motion direction. To reduce the number of weights we needed to fit and therefore improve our confidence in the weights we did fit, we performed PCA on the MT and SC responses to find the first 10 PCs in each area. The choice of number of vectors did not qualitatively affect the results in the range of 4-15 vectors. We then performed linear regression to find the weight vectors (for the Stimulus and Choice decoders) or weight matrices (for the SC decoder) that related projections along the first ten MT PCs plus a vector of ones to ‘motion direction’, ‘choice’, or projections along the first 10 SC PCs in each attention condition.
We assessed the stimulus information in each decoder (Figure 3) by multiplying projections of MT responses to the original and changed stimuli from the held-out trial by the fitted weights and either determining whether those weighted sums correctly classified the stimuli as original or changed (Stimulus and Choice decoders) or whether a linear classifier correctly classified those stimulus presentations on the basis of the predicted SC responses (SC decoder). The performance of the decoder is defined as the area under the receiver operating characteristic curve comparing the distributions of weighted average responses to each stimulus using the weights constructed for each decoder.
The critical aspect of the decoding analysis is that we ask how much stimulus information is contained in each different subset of MT activity. The Stimulus (or optimal) decoder will perform best, because it was designed specifically to ask this question. The Choice and SC decoders identify different subspaces of MT activity and then ask how much stimulus information is contained in those subspaces. These decoders, by definition, will perform worse than the Stimulus decoder, but they are asking the same question.
To assess the stability of the weights for each decoder in the two attention conditions, we assessed the stimulus information gleaned by each decoder using the sensory responses from one attention conditions and the weights calculated from the other (Figures 3 and 4). Because the responses of visual neurons are non-unique and because our behavioral response is binary, the weights found with our linear decoding methods are non-unique23,30. It is therefore not informative to make direct comparisons of the weights across conditions. Instead, we borrowed the spirit of the analyses in a recent study30 and compared the stimulus information that could be gleaned using each set of weights in each attention condition. In general, the choice and SC decoders performed better with weights computed from the same attention condition, even though we cross-validated these analyses (this effect could be attributed to non-stationarities in the recordings or the monkey’s behavior). The critical comparison is the performance of the decoders using sensory responses from one attention condition and weights from the other (Figures 3 and 4).
For the decoding analysis in Figure 5, we took a similar approach to the previously described Choice and SC decoders, except that we combined data from both the cued and uncued conditions to calculate decoding weights. We then decomposed the responses of the population responses to each stimulus in each attention conditions into mean responses and residuals (R=M+S, where R is the number of neurons by number of trials matrix of spike count responses to one stimulus in one attention condition, M is a matrix of mean responses for each neuron, and S is the matrix of residuals). We tested the hypothesis that attention-related changes in the residuals account for the improvement in stimulus information used to guide behavior by decoding stimulus information from responses created by using the mean responses from the uncued condition and residuals from the cued condition.
The analyses of the V4 data from the change detection task (Figure 4A and 4B) were identical manner to the MT data described above. This dataset consisted of multineuron recordings using Utah arrays placed in both hemispheres of V4 during 37 experimental sessions in two animals, the details of which are described in5. Data from each hemisphere was treated separately in the decoding analyses, so each session contributes two data points for each analysis (gray lines in Figure 4A). The details of the contrast discrimination task used in Figure 4C required a different form of the Stimulus decoder. This dataset consisted of multineuron recordings using Utah arrays placed in both hemispheres of V4 during 17 experimental sessions in two animals. The details of this experiment have been previously described6. Briefly, two monkeys judged which of two stimuli in a pair was higher contrast by making a saccade to a target representing its choice. Attention toward one pair of stimuli or the other was changed in blocks. The Stimulus decoder (Figure 4C) compares performance using V4 responses to distinguish between a given stimulus configuration and its opposite configuration in the attended and unattended conditions. As in the other V4 data set, data from each hemisphere was treated separately.
Code availability
Custom Matlab code is available upon reasonable request to the authors.
Author contributions
Both authors conceived and designed the experiments, analyzed the data and wrote the manuscript. D.A.R. collected the data.
Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
Competing interests
The authors declare no competing interests
Acknowledgements
We are grateful to John Maunsell for granting permission for the use of the V4 data, to Karen McKracken for providing technical assistance, and to Faisal Baqai, Matt Getz, Jay Hennig, Chengcheng Huang, Amy Ni and John Maunsell for comments on an earlier version of this manuscript.