On 2017 Sep 13, Michael Cole commented:

As first author of this paper, I am grateful to have the chance to evaluate the possibility of a logical error in one of the reported analyses, in order to correct the scientific record and possibly strengthen the conclusions drawn in that paper. Indeed, it appears that the reported negative correlation between resting-state functional connectivity (FC) strength and task-to-rest FC differences could (in theory) be driven by regression to the mean. Critically, having this as a logical possibility does not mean that regression to the mean actually drove the reported result. I therefore conducted a new analyses of publicly-available data to determine whether these results were due to regression to the mean.

I used a slightly different (but overlapping) set of subjects from what was used in the 2014 paper, due to easy availability of these data in my lab and also given that this set of subjects is more standard in the field than the one used in the 2014 paper. Specifically, I used the “100 unrelated” healthy young adult subjects from the WashU-Minn Human Connectome Project dataset (https://www.humanconnectome.org/) . I also switched to using a different set of functionally-defined brain regions – the Glasser et al. 2016 (Nature) region set (360 cortical regions) – rather than the original Power et al. 2011 (Neuron) region set. Additionally, unlike the original study, global signal regression was not used, given concerns about its effect on fMRI signals when comparing resting-state FC to task-state FC (based on simulations conducted in my lab). Finally, a finite impulse response (FIR) general linear model was used to fit the task-evoked activations instead of a canonical hemodynamic response general linear model, given that FIR modeling removes more cross-trial mean variance than canonical hemodynamic response modeling. (As before, the residuals from such a general linear model were used for computing task-state FC estimates). Other than these changes the data are, to my knowledge, identical to those reported in the 2014 paper. As expected, the identified resting-state FC to task-rest FC difference correlation was similar to the original r-value (which was r=-0.49, averaged across the 7 tasks) when using this slightly different dataset: r=-0.63 (averaged across the 7 tasks).

My approach to testing whether regression to the mean could explain the negative correlation between resting-state FC and task-rest FC was to use a split-half analysis, as suggested by Joern in his comment. Specifically, I split the 100 subjects into two sets of 50 each. Then the resting-state FC data from the first half were compared to the task-state FC data in the second half. Thus, correlations were computed for X1 with Y2-X2, such that the two to-be-compared FC matrices were completely independent (since each subject’s data were collected and processed independently).

The resulting split-half cross-validated r-value (averaged across the 7 tasks) was r=-0.60. This is highly similar to the original non-split-half r-value (r=-0.63), despite using only half of the data. Similar r-values were found for each task independently. Specifically (by task): -0.50, -0.69, -0.60, -0.56, -0.62, -0.62, -0.62. This demonstrates that the originally reported effect is likely not driven by regression to the mean.

As an aside, it might be surprising that such a similar result would arise despite using half of the data used in the original analysis. A quick test of repeat reliability, however, indicates that the two halves are highly similar to each other, explaining the strong replication. Specifically, the task-rest FC differences across all 360 regions correlated across the split halves at r=0.86 (mean across 7 tasks). The task-state FC values correlated across the split halves at r=0.97 (mean across 7 tasks). Similarly, the resting-state FC values correlated across the split halves at r=0.98.

Finally, I thought it would be interesting to test whether there was anything special about the resting-state FC values with respect to how anti-correlated they are with task-rest FC differences. I therefore repeated the identical split-half cross-validation analysis, but using each task (one at a time) in place of the resting-state FC data. This indicated that there was a much weaker anti-correlation when using task-state FC in place of resting-state FC: r=-0.10 (mean across the 7 tasks, each separately “standing in” for resting-state FC in the analysis). This was consistent across all 7 tasks, as they each had correlations much less negative (and even positive in the case of the Gambling task) than the resting-state FC data.

Even though these analyses cast strong doubt on the possibility that regression to the mean drove the previously-reported negative correlation, it is still important to take this potential confound into account going forward when conducting analyses similar to this (and more generally).

On 2017 Sep 11, Jörn Diedrichsen commented:

In this otherwise very interesting paper, there is a potential problem with the analysis presented on page 244 and Figure 8b. Here, the authors correlate a connectivity weight estimated during rest (X) with the difference between connectivity weights estimated from task-based data and rest (Y-X). They found a negative correlation. This result could of course be trivial, as both X and Y are measured with noise, and a negative correlation can emerge simply by regression to the mean (https://en.wikipedia.org/wiki/Regression_toward_the_mean). Note that this phenomenon can be substantial – even with a halfway decent re-test reliability of r=0.5 for both X and Y and a large positive correlation between the true X and Y of r=0.9, the expected correlation between X and Y-X will be r=-0.52. One way to test the idea of a relationship would be to split the resting-state data in half and correlate X1 with Y – X2. Assuming that X1 and X2 are now independent (which will depend on the preprocessing and details of the analysis), the resulting correlation coefficient should be unbiased.

On 2017 Sep 11, Jörn Diedrichsen commented:

In this otherwise very interesting paper, there is a potential problem with the analysis presented on page 244 and Figure 8b. Here, the authors correlate a connectivity weight estimated during rest (X) with the difference between connectivity weights estimated from task-based data and rest (Y-X). They found a negative correlation. This result could of course be trivial, as both X and Y are measured with noise, and a negative correlation can emerge simply by regression to the mean (https://en.wikipedia.org/wiki/Regression_toward_the_mean). Note that this phenomenon can be substantial – even with a halfway decent re-test reliability of r=0.5 for both X and Y and a large positive correlation between the true X and Y of r=0.9, the expected correlation between X and Y-X will be r=-0.52. One way to test the idea of a relationship would be to split the resting-state data in half and correlate X1 with Y – X2. Assuming that X1 and X2 are now independent (which will depend on the preprocessing and details of the analysis), the resulting correlation coefficient should be unbiased.