Reviewer #1 (Public Review):
Summary:
Li and colleagues describe an experiment whereby sequences of dots in different locations were presented to participants while electroencephalography (EEG) was recorded. By presenting fixed sequences of dots in different locations repeatedly to participants, the authors assumed that participants had learned the sequences during the experiment. The authors also trained classifiers using event-related potential (ERP) data recorded from separate experimental blocks of dots presented in a random (i.e., unpredictable) order. Using these trained classifiers, the authors then assessed whether patterns of brain activity could be detected that resembled the neural response to a dot location that was expected, but not presented. They did this by presenting an additional set of sequences whereby only one of the dots in the learned sequence appeared, but not the other dots. They report that, in these sequences with omitted stimuli, patterns of EEG data resembled the visual response evoked by a dot location for stimuli that could be expected, but were not presented. Importantly, this only occurred for an omitted dot stimulus that would be expected to appear immediately after the dot that was presented in these partial sequences.
This exciting finding complements previous demonstrations of the ability to decode expected (but not presented) stimuli in Blom et al. (2020) and Robinson et al. (2020) that are cited in this manuscript. It suggests that the visual system is able to generate patterns of activity that resemble expected sensory events, approximately at times at which an observer would expect them.
Strengths:
The experiment was carefully designed and care was taken to rule out some confounding factors. For example, gaze location was tracked over time, and deviations from fixation were marked, in order to minimise the contributions of saccades to above-chance decoding of dot position. The use of a separate block of dots (with unpredictable locations) to train the classifiers was also useful in isolating visual responses evoked by each dot location independently of any expectations that might be formed during the experiment. A large amount of data was also collected from each participant, which is important when using classifiers to decode stimulus features from EEG data. This careful approach is commendable and draws on best practices from existing work.
Weaknesses:
While there was clear evidence of careful experiment design, there are some aspects of the data analysis and results that significantly limit the inferences that can be drawn from the data. Both issues raised here relate to the use of pre-stimulus baselines and associated problems. As these issues are somewhat technical and may not be familiar to many readers, I will try to unpack each line of reasoning below. Here, it should be noted that these problems are complex, and similar issues often go undetected even by highly experienced EEG researchers.
Relevant to both issues, the authors derived segments of EEG data relative to the time at which each dot was presented in the sequences (or would have appeared when the stimuli were omitted in the partial sequences). Segments were derived that spanned -100ms to 300ms relative to the actual or expected onset of the dot stimulus. The 300ms post-stimulus time period corresponds to the duration of each dot in the sequence (100ms) plus the inter-stimulus interval (ISI) that was 200ms in duration before the next dot appeared (or would be expected to appear in the partial sequences). Importantly, a pre-stimulus baseline was applied to each of these segments of data, meaning that the average amplitude at each electrode between -100ms and 0ms relative to (actual or expected) stimulus onset was subtracted from each segment of data (i.e., each epoch in common EEG terminology). While this type of baseline subtraction procedure is commonplace in EEG studies, in this study design it is likely to cause problematic effects that could plausibly lead to the patterns of results reported in this manuscript.
First of all, the authors compare event-related potentials (ERPs) evoked by dots in the full as compared to partial sequences, to test a hypothesis relating to attentional tuning. They reported ERP amplitude differences across these conditions, for epochs corresponding to when a dot was presented to a participant (i.e., excluding epochs time-locked to omitted dots). However, these ERP comparisons are complicated by the fact that, in the full sequences, dot presentations are preceded by the presentation of other dots in the sequence. This means that ERPs evoked by the preceding dots in the full sequences will overlap in time with the ERPs corresponding to the dots presented at the zero point in the derived epochs. Importantly, this overlap would not occur in the partial sequence conditions, where only one dot was presented in the sequence. This essentially makes any ERP comparisons between full and partial sequences very difficult to interpret, because it is unclear if ERP differences are simply a product of overlapping ERPs from previously presented dots in the full sequence conditions. For example, there are statistically significant differences observed even in the pre-stimulus baseline period for this ERP analysis, which likely reflects the contributions ERPs evoked by the preceding dots in the full sequences, which are absent in the partial sequences.
The problems with interpreting this data are also compounded by the use of pre-stimulus baselines as described above. Importantly, the use of pre-stimulus baselines relies on the assumption that the ERPs in the baseline period (here, the pre-stimulus period) do not systematically differ across the conditions that are compared (here, the full vs. partial sequences). This assumption is violated due to the overlapping ERPs issue described just above. Accordingly, the use of the pre-stimulus baseline subtraction can produce spurious effects in the time period after stimulus onset (for examples see Feuerriegel & Bode, 2022, Neuroimage). This also makes it very difficult to meaningfully compare the ERPs following dot stimulus onset in these analyses.
The second issue relates to the use of pre-stimulus baselines and concerns the key finding reported in the paper: that EEG patterns corresponding to expected but omitted events can be decoded in the partial sequences. In the partial sequences, there are two critical epochs that were derived: One time-locked to the presentation of the dot, and another that was time-locked to 300ms after the dot was presented (i.e. when the next dot would be expected to appear). The latter epoch was used to test for representations of expected, but omitted, stimulus locations.
For the epochs in which the dots were presented, above-chance decoding can be observed spanning a training time range from around 100-300ms and a testing time range of a similar duration (see the plot in Figure 4b). This plot indicates that, during the time window of around 200-300ms following dot stimulus onset, the position of the dot can be decoded not only from trained classifiers using the same time windows spanning 200-300ms, but also using classifiers trained using earlier time windows of around 100-200ms.
This is important because the 200-300ms time period after dot onset in the partial sequences is the window used for pre-stimulus baseline subtraction when deriving epochs corresponding to the first successor representation (i.e., the first stimulus that might be expected to follow from the presented dot, but did not actually appear). In other words, the 200-300ms time window from dot onset corresponds to the -100 to 0 ms time window in the first successor epochs. Accordingly, the pattern that is indicative of the preceding, actually presented dot position would be subtracted from the EEG data used to test for the successor representation. Notably, the first successor condition would always be in another visual field quadrant (90-degree rotated or the opposite quadrant) as stated in the methods. In other words, the omitted stimulus would be expected to appear in the opposite vertical and/or horizontal visual hemifield as compared to the previously presented dot in these partial sequences.
This is relevant because ERPs tend to show reversed polarity across hemifields. For example, a stimulus presented in the right hemifield will have reversed polarity patterns at the same electrode as compared to an equivalent stimulus presented in the left hemifield (e.g., Supplementary Figure 3 in the comparable study of Blom et al., 2020). By subtracting the ERP patterns evoked by the presented dot in the partial sequences during the time period of 200-300ms (corresponding to the -100 to 0ms baseline window), this would be expected to bias patterns of EEG data in the first successor epochs to resemble stimulus positions in opposite hemifields. This could plausibly produce above-chance decoding accuracy in the time windows identified in Figure 5a, where the training time windows broadly correspond to the periods of above-chance decoding during 200-300ms from dot stimulus onset in Figure 4b.
In other words, the above-chance decoding of the first successor representation may plausibly be an artefact of the pre-stimulus baseline subtraction procedure used when deriving the epochs. This casts some doubt as to whether genuine successor representations were actually detected in the study. Additional tests for successor representations using ERP baselines prior to the presented dot in the partial sequences may be able to get around this, but such analyses were not presented, and the code and data were not accessible at the time of this review.
Although the study is designed well and a great amount of care was taken during the analysis stage, these issues with ERP overlap and baseline subtraction raise some doubts regarding the interpretability of the findings in relation to the analyses currently presented.