On 2017 Dec 24, Dorothy V M Bishop commented:

It is great to hear that the authors are conducting a large scale follow-up study: I would urge them to do this as a Registered Report, or at least to pre-register their hypotheses. Yes, small samples are not inherently bad, but, as demonstrated by Ramus in his 2017 paper,it is a sad reality that most findings from small studies in this areas fail to replicate in larger samples - perhaps not surprisingly given heterogeneity of dyslexia. There is a danger that, aware of this history, we might throw out the baby with the bathwater and dismiss genuine, important findings. But it is hard to know which findings are solid because most researchers in this area currently fail to distinguish hypothesis-testing and exploratory studies. A proper, large-scale prospective study with hypotheses identified a priori would be extremely valuable.

On 2017 Dec 21, Christa Müller-Axt commented:

We here respond to the questions and concerns raised by Franck Ramus (23th of Nov 2017). We organize our response around four topics: (i) sample size and statistical power, (ii) interpretation of effect sizes, (iii) specific predictions and correlation analyses, and (iv) current status and future directions.

(i) Sample size and statistical power: Franck Ramus argues that our study lacks statistical power due to the modest sample size (i.e., n=12 dyslexics and n=12 matched controls). First, as recommended by Button et al. (2013), we have explicitly acknowledged and discussed the modest sample size in our manuscript (see pp. 3694 & 3696). Second, given the well-known statistical problems associated with modest sample sizes, we took measures to reproduce our results in additional analyses, as laid out in the paper (i.e., surface-based analyses, see pp. 3694 & e3). While we are aware that these analyses do not constitute an independent replication, they further provide confidence in our main results, as they make it unlikely that the findings are merely due to methodological specificities. Third, the Button et al. (2013) paper elicited a more detailed discussion on sample size issues and neither Button et al. (2013) themselves nor any of the comments and papers in response to Button et al. (2013) suggested that small sample size studies should be dismissed altogether (see e.g., Bacchetti P., 2013; Quinlan P.T., 2013 for a more detailed discussion of these arguments).

(ii) Interpretation of effect sizes: Franck Ramus questions the plausibility of our results, specifically because the observed effect size for the group difference in left V5/MT-LGN connectivity is larger (d=1.28) than the observed effect size for the associated behavioral difference in rapid automatized naming (RAN) for letters and numbers (composite score: d=1.16). We were surprised by this remark. A larger effect size for left V5/MT-LGN connectivity differences than for behavioral differences on RAN could be explained by a number of reasons, e.g.: 1) altered V5/MT-LGN connectivity might also affect other cognitive processes that are not captured by RAN measures, 2) RAN is supported by additional brain networks, other than V5/MT-LGN connectivity alone, that contribute to task performance, 3) recruitment of compensatory mechanisms that partially substitute altered V5/MT-LGN connectivity to enable RAN performance. In our view, all of these mechanisms (as well as their combinations) seem plausible given the complex and nonlinear nature of the human brain and given the multi-component nature of the RAN. Therefore, we do not see how the apparent difference in reported effect sizes between the behavioral and neural measures would undermine the plausibility of our results.

(iii) Specific predictions and correlation analyses: Franck Ramus asks whether we had specific predictions about the correlations reported in our paper. We had indeed specific predictions about the RAN and reading comprehension scores. As described in the paper (pp. 3693 & 3695), these predictions were based on a previous finding from our research group that only these two scores correlated with sensory thalamus function in dyslexics in an fMRI study (Diaz et al., 2012). The correlation in that study was only found for RAN and reading comprehension in dyslexics, but not in controls (Diaz et al., 2012). Hence, in the present paper we tested the specific hypothesis that RAN and reading comprehension scores are related to V5/MT-LGN connectivity in dyslexics. Franck Ramus states that there are other core symptoms of dyslexia. He asks whether we tested correlations of V5/MT-LGN connectivity with more behavioral measures than reported, without appropriately correcting for multiple tests. We agree that dyslexia also encompasses other core symptoms (e.g., spelling and reading speed deficits). We did, however, not test for correlations with these or other behavioral measures, as we did not have specific predictions about them. It should go without saying that we do not do selective reporting of test results. Our predictions about the correlations with RAN and reading comprehension related to the dyslexia group, while we did not expect a correlation in the control group. We therefore consider it valid to apply a Bonferroni correction for the two tested correlations within each group separately. We did not compute correlations across the whole sample of dyslexics and controls together, as this would not have addressed our hypothesis (i.e., that RAN and reading comprehension scores are related to V5/MT-LGN connectivity in individuals with developmental dyslexia).

(iv) Current status and future directions: We appreciate that Franck Ramus highlighted the issue of the replication crisis. Indeed, while we think it is important not to dismiss results merely based on sample size, we agree that novel research findings necessitate replication. For this reason, our research lab is currently conducting a large scale follow-up study for replication as well as testing of further hypotheses generated by the present findings. We hope that the present study will inspire further research on potential deficits at the level of the sensory thalami and cortico-thalamic connections in the dyslexia field – a field that is currently dominated by an almost exclusive cerebral-cortex-centric view.

Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365‑376. https://doi.org/10.1038/nrn3475

Bacchetti, P. (2013). Small sample size is not the real problem. Nature Reviews Neuroscience, 14(8), 585-585. https://doi.org/10.1038/nrn3475-c3

Quinlan, P. T. (2013). Misuse of power: in defence of small-scale science. Nature Reviews Neuroscience, 14(8), 585-585. https://doi.org/10.1038/nrn3475-c1

Díaz, B., Hintz, F., Kiebel, S. J., & von Kriegstein, K. (2012). Dysfunction of the auditory thalamus in developmental dyslexia. Proceedings of the National Academy of Sciences, 109(34), 13841-13846. https://doi.org/10.1073/pnas.1119828109

On 2017 Nov 24, Dorothy V M Bishop commented:

I agree with Franck's comment. This seems yet another example of a high impact journal favouring newsworthiness of a result over methodological quality. See my earlier blogpost on this topic:

https://figshare.com/articles/High-impact_journals_where_newsworthiness_trumps_methodology/5631748

On 2017 Dec 12, Jose M. Moran commented:

Great analysis! completely agree!

On 2017 Nov 23, Franck Ramus commented:

This study seems severely underpowered, which is surprising for such a frequent disorder as dyslexia, and given that we are in the midst of a replication crisis (e.g., Button et al. 2013; Ramus et al. 2017).

With 12 participants per group, this study had 29% chance to observe a moderate group difference (d=0.6). Here, the significant result is due to a huge group difference (d=1.28) in the left V5/MT-LGN connectivity. Even larger than the corresponding behavioural difference in RAN letters (d=1.27) and numbers (d=0.95) (from Table S1). How plausible is it that there should be such a large brain difference between two groups of dyslexic and control individuals, even larger than the cognitive symptoms that this brain difference is presumed to underlie?

Similarly, with 12 dyslexic individuals, only huge correlations greater than 0.576 could be significant. Luckily this study observed a correlation of 0.588 between left V5/MT-LGN connectivity and RAN (using a one-tailed test and correcting for two tests), but not with reading comprehension. But what about the other behavioural variables, spelling and reading speed? Are they not core symptoms of dyslexia, even more so than RAN? Do they not rely on visual abilities? Were the a priori predictions so specific to RAN and reading comprehension, that correlations with spelling and reading speed were not even tested? If those predictions had been preregistered, this might be credible. Alternatively, were those correlations tested, but not taken into account in the correction for multiple tests? (not even mentioning correlations within the control group, or across the two groups)

Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365‑376. https://doi.org/10.1038/nrn3475 Ramus, F., Altarelli, I., Jednoróg, K., Zhao, J., & Scotto di Covella, L. (2017). Neuroanatomy of developmental dyslexia : pitfalls and promise. Neuroscience & Biobehavioral Reviews. https://doi.org/10.1016/j.neubiorev.2017.08.001

On 2017 Nov 23, Franck Ramus commented:

This study seems severely underpowered, which is surprising for such a frequent disorder as dyslexia, and given that we are in the midst of a replication crisis (e.g., Button et al. 2013; Ramus et al. 2017).

With 12 participants per group, this study had 29% chance to observe a moderate group difference (d=0.6). Here, the significant result is due to a huge group difference (d=1.28) in the left V5/MT-LGN connectivity. Even larger than the corresponding behavioural difference in RAN letters (d=1.27) and numbers (d=0.95) (from Table S1). How plausible is it that there should be such a large brain difference between two groups of dyslexic and control individuals, even larger than the cognitive symptoms that this brain difference is presumed to underlie?

Similarly, with 12 dyslexic individuals, only huge correlations greater than 0.576 could be significant. Luckily this study observed a correlation of 0.588 between left V5/MT-LGN connectivity and RAN (using a one-tailed test and correcting for two tests), but not with reading comprehension. But what about the other behavioural variables, spelling and reading speed? Are they not core symptoms of dyslexia, even more so than RAN? Do they not rely on visual abilities? Were the a priori predictions so specific to RAN and reading comprehension, that correlations with spelling and reading speed were not even tested? If those predictions had been preregistered, this might be credible. Alternatively, were those correlations tested, but not taken into account in the correction for multiple tests? (not even mentioning correlations within the control group, or across the two groups)

Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365‑376. https://doi.org/10.1038/nrn3475 Ramus, F., Altarelli, I., Jednoróg, K., Zhao, J., & Scotto di Covella, L. (2017). Neuroanatomy of developmental dyslexia : pitfalls and promise. Neuroscience & Biobehavioral Reviews. https://doi.org/10.1016/j.neubiorev.2017.08.001

On 2017 Nov 24, Dorothy V M Bishop commented:

I agree with Franck's comment. This seems yet another example of a high impact journal favouring newsworthiness of a result over methodological quality. See my earlier blogpost on this topic:

https://figshare.com/articles/High-impact_journals_where_newsworthiness_trumps_methodology/5631748

On 2017 Dec 21, Christa Müller-Axt commented:

We here respond to the questions and concerns raised by Franck Ramus (23th of Nov 2017). We organize our response around four topics: (i) sample size and statistical power, (ii) interpretation of effect sizes, (iii) specific predictions and correlation analyses, and (iv) current status and future directions.

(i) Sample size and statistical power: Franck Ramus argues that our study lacks statistical power due to the modest sample size (i.e., n=12 dyslexics and n=12 matched controls). First, as recommended by Button et al. (2013), we have explicitly acknowledged and discussed the modest sample size in our manuscript (see pp. 3694 & 3696). Second, given the well-known statistical problems associated with modest sample sizes, we took measures to reproduce our results in additional analyses, as laid out in the paper (i.e., surface-based analyses, see pp. 3694 & e3). While we are aware that these analyses do not constitute an independent replication, they further provide confidence in our main results, as they make it unlikely that the findings are merely due to methodological specificities. Third, the Button et al. (2013) paper elicited a more detailed discussion on sample size issues and neither Button et al. (2013) themselves nor any of the comments and papers in response to Button et al. (2013) suggested that small sample size studies should be dismissed altogether (see e.g., Bacchetti P., 2013; Quinlan P.T., 2013 for a more detailed discussion of these arguments).

(ii) Interpretation of effect sizes: Franck Ramus questions the plausibility of our results, specifically because the observed effect size for the group difference in left V5/MT-LGN connectivity is larger (d=1.28) than the observed effect size for the associated behavioral difference in rapid automatized naming (RAN) for letters and numbers (composite score: d=1.16). We were surprised by this remark. A larger effect size for left V5/MT-LGN connectivity differences than for behavioral differences on RAN could be explained by a number of reasons, e.g.: 1) altered V5/MT-LGN connectivity might also affect other cognitive processes that are not captured by RAN measures, 2) RAN is supported by additional brain networks, other than V5/MT-LGN connectivity alone, that contribute to task performance, 3) recruitment of compensatory mechanisms that partially substitute altered V5/MT-LGN connectivity to enable RAN performance. In our view, all of these mechanisms (as well as their combinations) seem plausible given the complex and nonlinear nature of the human brain and given the multi-component nature of the RAN. Therefore, we do not see how the apparent difference in reported effect sizes between the behavioral and neural measures would undermine the plausibility of our results.

(iii) Specific predictions and correlation analyses: Franck Ramus asks whether we had specific predictions about the correlations reported in our paper. We had indeed specific predictions about the RAN and reading comprehension scores. As described in the paper (pp. 3693 & 3695), these predictions were based on a previous finding from our research group that only these two scores correlated with sensory thalamus function in dyslexics in an fMRI study (Diaz et al., 2012). The correlation in that study was only found for RAN and reading comprehension in dyslexics, but not in controls (Diaz et al., 2012). Hence, in the present paper we tested the specific hypothesis that RAN and reading comprehension scores are related to V5/MT-LGN connectivity in dyslexics. Franck Ramus states that there are other core symptoms of dyslexia. He asks whether we tested correlations of V5/MT-LGN connectivity with more behavioral measures than reported, without appropriately correcting for multiple tests. We agree that dyslexia also encompasses other core symptoms (e.g., spelling and reading speed deficits). We did, however, not test for correlations with these or other behavioral measures, as we did not have specific predictions about them. It should go without saying that we do not do selective reporting of test results. Our predictions about the correlations with RAN and reading comprehension related to the dyslexia group, while we did not expect a correlation in the control group. We therefore consider it valid to apply a Bonferroni correction for the two tested correlations within each group separately. We did not compute correlations across the whole sample of dyslexics and controls together, as this would not have addressed our hypothesis (i.e., that RAN and reading comprehension scores are related to V5/MT-LGN connectivity in individuals with developmental dyslexia).

(iv) Current status and future directions: We appreciate that Franck Ramus highlighted the issue of the replication crisis. Indeed, while we think it is important not to dismiss results merely based on sample size, we agree that novel research findings necessitate replication. For this reason, our research lab is currently conducting a large scale follow-up study for replication as well as testing of further hypotheses generated by the present findings. We hope that the present study will inspire further research on potential deficits at the level of the sensory thalami and cortico-thalamic connections in the dyslexia field – a field that is currently dominated by an almost exclusive cerebral-cortex-centric view.

Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365‑376. https://doi.org/10.1038/nrn3475

Bacchetti, P. (2013). Small sample size is not the real problem. Nature Reviews Neuroscience, 14(8), 585-585. https://doi.org/10.1038/nrn3475-c3

Quinlan, P. T. (2013). Misuse of power: in defence of small-scale science. Nature Reviews Neuroscience, 14(8), 585-585. https://doi.org/10.1038/nrn3475-c1

Díaz, B., Hintz, F., Kiebel, S. J., & von Kriegstein, K. (2012). Dysfunction of the auditory thalamus in developmental dyslexia. Proceedings of the National Academy of Sciences, 109(34), 13841-13846. https://doi.org/10.1073/pnas.1119828109

On 2017 Dec 24, Dorothy V M Bishop commented:

It is great to hear that the authors are conducting a large scale follow-up study: I would urge them to do this as a Registered Report, or at least to pre-register their hypotheses. Yes, small samples are not inherently bad, but, as demonstrated by Ramus in his 2017 paper,it is a sad reality that most findings from small studies in this areas fail to replicate in larger samples - perhaps not surprisingly given heterogeneity of dyslexia. There is a danger that, aware of this history, we might throw out the baby with the bathwater and dismiss genuine, important findings. But it is hard to know which findings are solid because most researchers in this area currently fail to distinguish hypothesis-testing and exploratory studies. A proper, large-scale prospective study with hypotheses identified a priori would be extremely valuable.