On 2016 Jun 24, Ole Jakob Storebø commented:

We think that the clinicians prescribing methylphenidate for ADHD and others have been insufficiently critical of the literature for decades, trusting that the quality of methylphenidate research was reasonable. In accordance, Shaw in an editorial accompanying our JAMA article Storebø OJ, 2016 stated that the Epstein et al. review on methylphenidate for adults with ADHD was an example of good assessments of quality Shaw P, 2016. It seems Shaw erred, as the Epstein et al. review has now been withdrawn from The Cochrane Library due methodological flaws Epstein T, 2016.

Banaschewski et al. suggest that we included five trials in our analyses that should have been excluded. We think they are wrong. They highlight four trials which they cite as having used “active controls” whereas these are actually co-interventions, used in both the methylphenidate and the control group. Such trials are includable in accordance with our protocol Storebø OJ, 2015. Moreover, excluding these trials from our review would only have produced negligible changes in our results. Furthermore, the trial including children aged 3 to 6 years ought also to have be included in accordance with our protocol. Excluding all five trials would not have changed our conclusions at all. We concluded that methylphenidate might improve teacher reported symptoms of ADHD. However, the very low quality of the evidence, the magnitude of that effect size is uncertain. A change in the effect size of 0.12 points on the standardised mean difference of this outcome would not change anything.

In a subgroup analysis comparing parallel trials and crossover trials, we did not find a significant difference either. However, we noted considerable heterogeneity between the two groups of trials. It is not recommended to pool cross-over trials which only have “end-of-trial data” with parallel group trials (http://handbook.cochrane.org/) and had we done so we would have would risked introducing a “unit-of-analysis error” as we only had “end-of-trial data” from these cross-over trials.

We agree that the variability of the minimal relevant difference is important which is why we reported the 95% confidence interval of the transformed mean value in our review Storebø OJ, 2015. Banaschewski et al. also suggest that we have overlooked information in the Coghill 2007 trial and thereby wrongly assessed this as a trial with “high risk of bias”. We stated in our protocol that we would consider trials with one or more unclear or high risk of bias domain as trials with high risk of bias Storebø OJ, 2015.

We did not overlook information from the Coghill 2007 trial but twice emailed the authors for additional information. They did not respond. The information required is not available in the published study. We presented the risk of bias assessments for the various domains of all 185 included trials. It is correct that we assessed seven cross-over trials as low risk of bias and not six as reported. Thank you for spotting this error. The seventh trial is reported, however, in our table in which the risk of bias assessments for all the domains is shown. All trials, irrespective of vested interest bias, were regarded as having a high risk of bias due to broken outcome assessor blinding given the easily recognisable, well-known adverse effects of methylphenidate. When adding this seventh cross-over trial to the subgroup analysis on the outcome “teacher-rated ADHD symptoms – cross-over trials”, we now find significant differences between the trials with “high” compared to “low” risk of bias (standardised mean difference (SMD) -0.96 [95% confidence interval -1.09 to -0.82] compared to -0.64 [-0.91 to -0.38]. Test for subgroup difference: Chi² = 4.27, df = 1 (P = 0.04), I² = 76.6%).

Banaschewski et al. focus only on our assessment of risk of bias and do not mention the core instrument for assessing quality of meta-analyses namely the Grades of Recommendation Assessment Development and Evaluation (GRADE) approach Andrews J, 2013. Our assessment of the evidence as “very low quality” is not only based on the assessment of risk of bias, but also on other factors such as heterogeneity, imprecision, and indirectness of the evidence. This is clearly reported in our review.

We downgraded the quality of the included trials in the meta-analysis for imprecision and for moderate heterogeneity. The durations of included trials were short, with an average of 75 days. Most patients receive methylphenidate treatment for substantially longer periods and the beneficial effects may diminish over time Jensen PS, 2007 Molina BS, 2009. The short trial duration could suggest the need for further downgrading for “indirectness” according to GRADE Andrews J, 2013. We did not downgrade for this, but we could have. This further underlines that the evidence for the benefits and harms for the use of methylphenidate for children and adolescents with ADHD is of very low quality.

We have assessed 71 trials as having high risk of bias in the “vested interest” domain as they were funded by the industry and/or the authors were affiliated with the industry.

It is not incorrect for us to state that none of the trials funded by the pharmaceutical industry showed a low risk of bias in all other areas as we considered all the trials as high risk of bias on the domain of blinding. This is clearly reported in our review.

We have now conducted the requested subgroup analysis comparing those trials with high compared to low risk of vested interest bias on the teacher-rated ADHD symptoms outcome. The effect of methylphenidate in the 14 trials with high risk of vested interest bias was SMD -0.86 [-0.99 to -0.72] compared to SMD -0.50 [-0.69 to -0.31] in the 5 trials with low risk of vested interest bias. Test for subgroup differences is Chi² = 8.67, df = 1, P = 0.003. So even in this small sample we find a significant difference.

We recommend Banaschewski et al. to read the essay by John P Ioannidis about vested interests Ioannidis JP, 2016.

It is important to stress that the results of our review would have been the same had we disregarded the issue of vested interest.

Had there been inconsistencies regarding one domain of bias in a few trials they would not change the fact that these trials are to be considered as trials at high risk of bias. For example, in two trials, Konrad 2004 and Konrad 2005, there is inconsistency in how our author teams assessed the randomisation process. However, both trials have several other domains at “unclear risk of bias” or “high risk of bias”. In the Ullman 2006 trial, three domains are assessed as “unclear risk of bias”. In Wallace 1994, five domains are assessed as being of “unclear risk of bias” and one as “high risk of bias”. In Wallander 1987, five domains are assessed as “unclear risk of bias”. Even if there was inconsistency between one or two items, these trials are high risk of bias trials. There may well be small differences in our judgements, but that does not change the fact that the trials included are, in general, trials at high risks of bias Storebø OJ, 2015. It is important to understand that we followed the Cochrane guidelines in every aspect of our review.

Conclusions

We have demonstrated that the trial selection in our review was not flawed and was undertaken with sufficient scientific justification The effect sizes are not too small. We have followed a sound methodology for assessing risk of bias and our conclusion is not misleading. We are concerned about the state of the academic literature and at the financial and academic waste that has occurred, given that more than 250 reviews and 3000 single works have been published on psychostimulants for ADHD treatment. Despite this, there is still no sound evidence regarding the benefits and harms of methylphenidate.

Ole Jakob Storebø, Morris Zwi, and Christian Gluud.

On 2016 Jun 24, Ole Jakob Storebø commented:

We think that the clinicians prescribing methylphenidate for ADHD and others have been insufficiently critical of the literature for decades, trusting that the quality of methylphenidate research was reasonable. In accordance, Shaw in an editorial accompanying our JAMA article Storebø OJ, 2016 stated that the Epstein et al. review on methylphenidate for adults with ADHD was an example of good assessments of quality Shaw P, 2016. It seems Shaw erred, as the Epstein et al. review has now been withdrawn from The Cochrane Library due methodological flaws Epstein T, 2016.

Banaschewski et al. suggest that we included five trials in our analyses that should have been excluded. We think they are wrong. They highlight four trials which they cite as having used “active controls” whereas these are actually co-interventions, used in both the methylphenidate and the control group. Such trials are includable in accordance with our protocol Storebø OJ, 2015. Moreover, excluding these trials from our review would only have produced negligible changes in our results. Furthermore, the trial including children aged 3 to 6 years ought also to have be included in accordance with our protocol. Excluding all five trials would not have changed our conclusions at all. We concluded that methylphenidate might improve teacher reported symptoms of ADHD. However, the very low quality of the evidence, the magnitude of that effect size is uncertain. A change in the effect size of 0.12 points on the standardised mean difference of this outcome would not change anything.

In a subgroup analysis comparing parallel trials and crossover trials, we did not find a significant difference either. However, we noted considerable heterogeneity between the two groups of trials. It is not recommended to pool cross-over trials which only have “end-of-trial data” with parallel group trials (http://handbook.cochrane.org/) and had we done so we would have would risked introducing a “unit-of-analysis error” as we only had “end-of-trial data” from these cross-over trials.

We agree that the variability of the minimal relevant difference is important which is why we reported the 95% confidence interval of the transformed mean value in our review Storebø OJ, 2015. Banaschewski et al. also suggest that we have overlooked information in the Coghill 2007 trial and thereby wrongly assessed this as a trial with “high risk of bias”. We stated in our protocol that we would consider trials with one or more unclear or high risk of bias domain as trials with high risk of bias Storebø OJ, 2015.

We did not overlook information from the Coghill 2007 trial but twice emailed the authors for additional information. They did not respond. The information required is not available in the published study. We presented the risk of bias assessments for the various domains of all 185 included trials. It is correct that we assessed seven cross-over trials as low risk of bias and not six as reported. Thank you for spotting this error. The seventh trial is reported, however, in our table in which the risk of bias assessments for all the domains is shown. All trials, irrespective of vested interest bias, were regarded as having a high risk of bias due to broken outcome assessor blinding given the easily recognisable, well-known adverse effects of methylphenidate. When adding this seventh cross-over trial to the subgroup analysis on the outcome “teacher-rated ADHD symptoms – cross-over trials”, we now find significant differences between the trials with “high” compared to “low” risk of bias (standardised mean difference (SMD) -0.96 [95% confidence interval -1.09 to -0.82] compared to -0.64 [-0.91 to -0.38]. Test for subgroup difference: Chi² = 4.27, df = 1 (P = 0.04), I² = 76.6%).

Banaschewski et al. focus only on our assessment of risk of bias and do not mention the core instrument for assessing quality of meta-analyses namely the Grades of Recommendation Assessment Development and Evaluation (GRADE) approach Andrews J, 2013. Our assessment of the evidence as “very low quality” is not only based on the assessment of risk of bias, but also on other factors such as heterogeneity, imprecision, and indirectness of the evidence. This is clearly reported in our review.

We downgraded the quality of the included trials in the meta-analysis for imprecision and for moderate heterogeneity. The durations of included trials were short, with an average of 75 days. Most patients receive methylphenidate treatment for substantially longer periods and the beneficial effects may diminish over time Jensen PS, 2007 Molina BS, 2009. The short trial duration could suggest the need for further downgrading for “indirectness” according to GRADE Andrews J, 2013. We did not downgrade for this, but we could have. This further underlines that the evidence for the benefits and harms for the use of methylphenidate for children and adolescents with ADHD is of very low quality.

We have assessed 71 trials as having high risk of bias in the “vested interest” domain as they were funded by the industry and/or the authors were affiliated with the industry.

It is not incorrect for us to state that none of the trials funded by the pharmaceutical industry showed a low risk of bias in all other areas as we considered all the trials as high risk of bias on the domain of blinding. This is clearly reported in our review.

We have now conducted the requested subgroup analysis comparing those trials with high compared to low risk of vested interest bias on the teacher-rated ADHD symptoms outcome. The effect of methylphenidate in the 14 trials with high risk of vested interest bias was SMD -0.86 [-0.99 to -0.72] compared to SMD -0.50 [-0.69 to -0.31] in the 5 trials with low risk of vested interest bias. Test for subgroup differences is Chi² = 8.67, df = 1, P = 0.003. So even in this small sample we find a significant difference.

We recommend Banaschewski et al. to read the essay by John P Ioannidis about vested interests Ioannidis JP, 2016.

It is important to stress that the results of our review would have been the same had we disregarded the issue of vested interest.

Had there been inconsistencies regarding one domain of bias in a few trials they would not change the fact that these trials are to be considered as trials at high risk of bias. For example, in two trials, Konrad 2004 and Konrad 2005, there is inconsistency in how our author teams assessed the randomisation process. However, both trials have several other domains at “unclear risk of bias” or “high risk of bias”. In the Ullman 2006 trial, three domains are assessed as “unclear risk of bias”. In Wallace 1994, five domains are assessed as being of “unclear risk of bias” and one as “high risk of bias”. In Wallander 1987, five domains are assessed as “unclear risk of bias”. Even if there was inconsistency between one or two items, these trials are high risk of bias trials. There may well be small differences in our judgements, but that does not change the fact that the trials included are, in general, trials at high risks of bias Storebø OJ, 2015. It is important to understand that we followed the Cochrane guidelines in every aspect of our review.

Conclusions

We have demonstrated that the trial selection in our review was not flawed and was undertaken with sufficient scientific justification The effect sizes are not too small. We have followed a sound methodology for assessing risk of bias and our conclusion is not misleading. We are concerned about the state of the academic literature and at the financial and academic waste that has occurred, given that more than 250 reviews and 3000 single works have been published on psychostimulants for ADHD treatment. Despite this, there is still no sound evidence regarding the benefits and harms of methylphenidate.

Ole Jakob Storebø, Morris Zwi, and Christian Gluud.