On 2015 Apr 02, Harri Hemila commented:

Errors in the Cochrane review (2009) on vitamin C for asthma by Kaur B, 2009.

In 2009, I found that the Cochrane review on vitamin C for asthma by Kaur B, 2009 had severe errors in the extraction and analysis of data. I wrote a Feedback to the Cochrane review and it is part of the 2009 Cochrane review, DOI, but the Feedback is also available as a separate document. Then I found out that the errors originated in the first Cochrane review version by Kaur B, 2001, see Pubmed Commons. Those errors were repeated in the updates by Ram FS, 2004 and by Kaur B, 2009, until the review was withdrawn by Kaur B, 2013. Because of the severe errors, the Cochrane review “vitamin C for asthma” misled readers for over a decade. This below is the Feedback (2009).

The Feedback (2009) on the Cochrane review by Kaur B, 2009

Schachter EN, 1982 carried out a trial with participants who had exercise-induced bronchoconstriction (EIB) so that each of the 12 participants was administered placebo and vitamin C at different times. Thus, each participant served as his or her own control (crossover). In Table III Schachter reported pre-post-exercise change of FEV1, so that the later FEV1 was measured 5 minutes after the exercise. Because two observations are measured from the same participant, the placebo period and vitamin C period difference in FEV1 change should be analysed using the paired t-test. The FEV1 data in Schachter’s Table III gives the mean difference between the vitamin C and placebo periods as 0.20 (SD 0.33) litres/s. Schachter EN, 1982 calculated t=2.13 in their paper, corresponding to P[1-tail]=0.028.

The Cochrane review by Kaur B, 2009 presents Schachter’s FEV1 changes in Analysis 1.2. However, data in Analysis 1.2 were extracted from Schachter’s Table II, which presents post-exercise FEV1 value measured immediately after the exercise. In EIB the fall in FEV1 occurs 5 to 20 minutes after the end of exercise, Rundell KW, 2008. Even Schachter EN, 1982 reported that, on the screening day, there was no fall in FEV1 immediately after exercise, but a significant fall 5 minutes after the exercise (Schachter Fig. 2). Therefore, extracting the FEV1 changes from Schachter’s Table II (FEV1 immediately after the exercise) is not reasonable if the purpose is to examine the effect of vitamin C on EIB.

Cohen HA, 1997 carried out an EIB trial with 20 participants who were administered placebo and vitamin C at different times (crossover). Post-exercise FEV1 was measured 8 minutes after the end of the exercise. The observations are paired also in this case and the results should be analysed using a paired test. 9 participants had FEV1 decrease >15% on both vitamin C and placebo treatments. 11 participants had >15% FEV1 decrease on placebo but <15% FEV1 decrease on vitamin C (Cohen 1997 Fig. 2). None of the participants had the opposite effect: <15% FEV1 decrease on placebo and >15% FEV1 decrease on vitamin C. In the paired 2x2 table analysis, the question is whether the difference between the corners (here 11 and 0) is statistically significant. This difference gives z=(11-0)/sqrt(11+0)=3.31, corresponding to P[1-tail]=0.0005.

A basic principle in controlled trial analysis requires that all randomized participants should be included in the analysis, the ITT principle. However, the Cochrane review by Kaur B, 2009 does not give the results for all 20 participants of (Cohen Fig. 2); Analysis 1.2 gives the results for only the 11 participants who had benefit of vitamin C (Cohen Table 2).

Furthermore, the Cochrane review by Kaur B, 2009 presents the average of post-exercise FEV1 values and not the pre-post-exercise difference in FEV1 in analysis 1.2. The post-exercise averages for Cohen’s Table 2 are 1.66 (SD 0.80) litres/s in the placebo period and 1.93 (SD 0.78) litres/s in the vitamin C period, P=0.42. However, given that the EIB is defined by the pre-post change in FEV1, the measurement of the effect on EIB should be based on the pre-post-exercise difference in FEV1, Rundell KW, 2008.

Furthermore, the relative effect calculated by Cohen HA, 1997 (Table 2; in %-units) is a better measure than the absolute value (in litres/s) because the relative effect adjusts for the great variation in baseline FEV1; the relative decrease in FEV1 is also used in guidelines, Rundell KW, 2008. Cohen HA, 1997 reports that the average relative fall in FEV1 is 25% in the placebo period and 5% in the vitamin C period (Table 2). Because the observations are paired, the paired t-test should be used. The average of the differences is 20% (SD 12%, SE 3.7%), which gives t=5.57, corresponding to P[1-tail]=0.00012. Thus, although the Cochrane review by Kaur B, 2009 presented only the 11 participants in which vitamin C was beneficial, their calculation suggests that even in this subgroup vitamin C was without effect, P=0.42, whereas a correct calculation gives a much smaller P-value.

In their EIB trial, Tecklenburg SL, 2007 studied 8 participants who were administered vitamin C and placebo at different times. They measured post-exercise FEV1 at 1, 5, 10, 15, 20, and 30 min after the exercise. Tecklenburg 2007 reported that the decrease in FEV1 in the vitamin C period was 6.4% (SE 2.4%) and decrease in the placebo period was 12.9% (SE 2.4%). Tecklenburg did not publish the paired comparison, nor original data so that the paired t-test could be calculated. Nevertheless, these averages give unpaired t=1.91, corresponding to P[1-tail]=0.038, which is conservative, the paired test P-value would be smaller.

Thus, three trials included in the Cochrane review by Kaur B, 2009 found benefit of vitamin C supplementation against EIB at 5 and 8 minutes after the exercise, Cohen HA, 1997, Schachter EN, 1982, or at the time of maximum fall in FEV1, Tecklenburg SL, 2007. The three P-values calculated above (0.028, 0.0005, 0.038) can be combined by using the Fisher's method. The combined P = 0.00007 provides evidence that the effects of vitamin C on EIB in these three trials are not explained by random fluctuations.

Analyses 1.1, 1.3 and 1.5 by Kaur B, 2009 present baseline data of two EIB trials discussed above (Cohen 1997; Schachter 1982). However, when a trial specifically examines the effect of vitamin C on EIB, the relevant outcome is the difference between the baseline and the 5-10 minutes post-exercise FEV1 values (the pre-post change), and not the baseline FEV1 value alone.

Finally, diagnosis of EIB by the change in FEV1 is well established Rundell KW, 2008. Therefore Kaur B, 2009 should have considered whether there is any benefit for readers from making additional analyses of the FVC and PEF values of the oldest trial by Schachter 1982. The more recent trials by Cohen 1997 and Tecklenburg 2007 did not report changes in FVC and PEF.

Postscript in 2015

Three years after my Feedback (above), in 2012, the Cochrane review authors replied to my criticism. Since their response had further errors, I wrote a second Feedback in 2013. The second Feedback was rejected by the Cochrane Airways group. Their rejection was inconsistent with the Principles of the Cochrane Collaboration (accessed April 2015). Eight principle is about “Ensuring quality: by applying advances in methodology, developing systems for quality improvement, and being open and responsive to criticism” [bold added]. The second Feedback (2013) is available as a separate document.

The above Feedback (2009) was extended to full length papers by Hemilä H, 2013 and Hemilä H, 2014.