On 2015 Apr 02, Harri Hemila commented:

Cochrane review (2001) on vitamin C and asthma by Kaur B, 2001 misled readers for a decade due to the errors in the extraction and analysis of data.

The Cochrane review (2001) on vitamin C and asthma by Kaur B, 2001 has substantial errors. I found errors in the 2009 update of that review by Kaur B, 2009. My comments of the 2009 update are available as a separate document. Then I found that the errors originated in the first version by Kaur B, 2001, and they were repeated in the updates by Ram FS, 2004 and by Kaur B, 2009, until the review was withdrawn by Kaur B, 2013. Thus the Cochrane review (2001) on vitamin C and asthma misled readers for over a decade due to the errors in the extraction and analysis of data. Here I briefly describe the errors in the first version of the Cochrane review (2001) on vitamin C and asthma by Kaur B, 2001.

As relevant background knowledge, it was known long before Kaur B, 2001 that exercise can cause exercise-induced bronchoconstriction (EIB or EIA, exercise-induced asthma), eg Mahler DA, 1993, and that respiratory virus infections can cause exacerbations of asthma, eg Nicholson KG, 1993. Thus, if vitamin C influences bronchoconstriction, the effects might be most pronounced during exercise and/or viral infections.

Errors in the extraction of data

Kaur B, 2001 wrote in the Results section (p.4): “there were two studies where the protective effects of vitamin C were investigated using exercise challenge. Both these studies reported pulmonary function outcomes, but one (Cohen 1997) reported absolute mean value post-exercise, while the other (Schachter 1982) reported absolute change post-exercise .”

In Comparison 01: Outcomes 01 and 02 (p. 10), Kaur B, 2001 presented the results of those two studies. However, Cohen HA, 1997 had 20 participants, yet Kaur B, 2001 showed data for only 11 participants, without revealing to the reader that data for 9 of Cohen's participants are missing from their review. Cohen showed the data of 11 participants on the basis that vitamin C was beneficial for them. However, it is inappropriate and violates the ITT principle to show such a biased subgroup in the Cochrane review by Kaur B, 2001, without explaining that another 9 participants were excluded. Cohen HA, 1997 also reported that on the placebo day, 20/20 of the participants had EIB, whereas on the vitamin C day, 10/20 had EIB. This comparison does not need any imputations, but these data were not extracted for the Cochrane review by Kaur B, 2001.

A decade before the Kaur B, 2001 review, papers reported that “EIA symptoms start after exercise, peak 8 to 15 minutes after exercise” so that there is a delay between the end of exercise and the peak of FEV1 decline, and “a fall of 10% or more in the FEV1 after exercise is diagnostic” so that the relative change in FEV1 (in %) is the most reasonable outcome when assessing EIB, eg Mahler DA, 1993. Such facts are relevant when analyzing EIB trials.

Kaur B, 2001 considered that “exercise challenge” was a particular issue in the Cohen HA, 1997 and Schachter EN, 1982 studies, see the above extract. However, in Outcome 01 (p.10) the Cochrane review showed data on the pre-exercise FEV1 levels, which is not an outcome influenced by exercise. In Outcome 02 (p.10) the Cochrane review extracted data on the absolute post-exercise FEV1 level for Cohen HA, 1997 and absolute FEV1 change for Schachter EN, 1982. These outcomes are not relevant, since the standard outcome for EIB is the relative change in FEV1 (in %), see above.

Furthermore, the greatest decline in FEV1 occurs about 8 to 15 min after the end of exercise, above. However, in Outcome 02 (p.10), Kaur B, 2001 shows Schachter's FEV1 changes immediately after exercise in Schachter's Table II, instead of FEV1 changes 5 min after exercise which Schachter reported in Table III. The latter is much more relevant in a study on EIB, since “EIA symptoms ... peak 8 to 15 minutes after exercise”, Mahler DA, 1993.

Kaur B, 2001 (p.4) write that “Anah did not report data in a manner that permitted further analysis”. However, in the following paragraph, Kaur B, 2001 wrote that, in the Anah study, “there were 9 exacerbations in the intervention group which had 22 patients and 35 exacerbations in the placebo group which had 19 patients ”. Thus, Anah CO, 1980 did report data that can be statistically analyzed, see below.

Errors in the analysis of data

Schachter EN, 1982 and Cohen HA, 1997 were cross-over studies so that the same participants were randomly administered vitamin C or the placebo. Such data should be analyzed by the paired t-test.

In Outcomes 01 and 02 (p.10), Kaur B, 2001 analyzed Schachter and Cohen data by the unpaired t-test, which is unsound for paired data. Kaur B, 2001 reported no difference between the vitamin C and placebo days for 11 of Cohen's participants with P=0.4 (unpaired t-test), whereas the correct paired t-test yields P=0.003.

Furthermore, given that the goal was to examine the effect of vitamin C on EIB, Kaur B, 2001 should have calculated the relative changes in FEV1 (in %), see above.

Schachter EN, 1982 published the data for all their 12 participants, and Kaur B, 2001 could have used the paired t-test to analyze the relative changes.

Similarly, Kaur B, 2001 could have calculated the relative changes for the 11 participants published by Cohen HA, 1997, and could have imputed eg “no effect” for the 9 participants who had missing data.

Nevertheless, using the paired t-test is a simplistic analysis of FEV1 changes, since it is possible that the effect of vitamin C depends on the severity of EIB. Hemilä H, 2013 analyzed the two studies using linear regression. Schachter EN, 1982 showed a 55% reduction in the postexercise FEV1 decline (95%CI 32% to 78%) with vitamin C. Imputation of “no effect” to the 9 participants with missing data of Cohen HA, 1997 yielded an estimated 42% reduction in the postexercise FEV1 decline (19% to 64%) with vitamin C. Finally, Cohen HA, 1997 also published the EIB status, which has no missing data. On the placebo day, 100% (20/20) of participants suffered from EIB, whereas on the vitamin C day, only 50% (10/20) suffered from EIB. Thus vitamin C caused a 50 percentage point decrease (23 to 68) in the occurrence of EIB, see Hemilä H, 2013.

Kaur B, 2001 (p. 4) claimed that “Anah did not report data in a manner that permitted further analysis,” which is incorrect. Anah CO, 1980 reported that in the placebo group (n=19), there were 35 asthma attacks, but in the vitamin C group (n=22), only 9 attacks. By using the standard Poisson approach, these data give RR=0.22 (0.09 to 0.47). Anah did not publish the individual level data and it might be over-dispersed, see Glynn RJ, 1996. Nevertheless, Anah published partial descriptions of the asthma attack distributions which can be used to impute more realistic distributions for the treatment groups, yielding a 95%CI of 0.06 to 0.81, which is still far from the null effect, see Hemilä H, 2013. Furthermore, the exact distribution of severe and moderate asthma attacks in the vitamin C group can be inferred from Anah CO, 1980 and this outcome needs few imputations. Vitamin C reduced the incidence of severe and moderate asthma attacks by RR=0.11 (0.02 to 0.48).

In conclusion, had Kaur B, 2001 extracted data correctly, and had they properly carried out the statistical analysis, they could have concluded in 2001 that there was strong evidence that vitamin C was beneficial against EIB in Cohen HA, 1997 and in Schachter EN, 1982, and against common cold-induced asthma exacerbations in Anah CO, 1980.

Secondary analysis of the Cohen HA, 1997, Schachter EN, 1982 and Anah CO, 1980 studies has been carried out by Hemilä H, 2013, Hemilä H, 2013, and Hemilä H, 2014.

On 2015 Apr 02, Harri Hemila commented:

Cochrane review (2001) on vitamin C and asthma by Kaur B, 2001 misled readers for a decade due to the errors in the extraction and analysis of data.

The Cochrane review (2001) on vitamin C and asthma by Kaur B, 2001 has substantial errors. I found errors in the 2009 update of that review by Kaur B, 2009. My comments of the 2009 update are available as a separate document. Then I found that the errors originated in the first version by Kaur B, 2001, and they were repeated in the updates by Ram FS, 2004 and by Kaur B, 2009, until the review was withdrawn by Kaur B, 2013. Thus the Cochrane review (2001) on vitamin C and asthma misled readers for over a decade due to the errors in the extraction and analysis of data. Here I briefly describe the errors in the first version of the Cochrane review (2001) on vitamin C and asthma by Kaur B, 2001.

As relevant background knowledge, it was known long before Kaur B, 2001 that exercise can cause exercise-induced bronchoconstriction (EIB or EIA, exercise-induced asthma), eg Mahler DA, 1993, and that respiratory virus infections can cause exacerbations of asthma, eg Nicholson KG, 1993. Thus, if vitamin C influences bronchoconstriction, the effects might be most pronounced during exercise and/or viral infections.

Errors in the extraction of data

Kaur B, 2001 wrote in the Results section (p.4): “there were two studies where the protective effects of vitamin C were investigated using exercise challenge. Both these studies reported pulmonary function outcomes, but one (Cohen 1997) reported absolute mean value post-exercise, while the other (Schachter 1982) reported absolute change post-exercise .”

In Comparison 01: Outcomes 01 and 02 (p. 10), Kaur B, 2001 presented the results of those two studies. However, Cohen HA, 1997 had 20 participants, yet Kaur B, 2001 showed data for only 11 participants, without revealing to the reader that data for 9 of Cohen's participants are missing from their review. Cohen showed the data of 11 participants on the basis that vitamin C was beneficial for them. However, it is inappropriate and violates the ITT principle to show such a biased subgroup in the Cochrane review by Kaur B, 2001, without explaining that another 9 participants were excluded. Cohen HA, 1997 also reported that on the placebo day, 20/20 of the participants had EIB, whereas on the vitamin C day, 10/20 had EIB. This comparison does not need any imputations, but these data were not extracted for the Cochrane review by Kaur B, 2001.

A decade before the Kaur B, 2001 review, papers reported that “EIA symptoms start after exercise, peak 8 to 15 minutes after exercise” so that there is a delay between the end of exercise and the peak of FEV1 decline, and “a fall of 10% or more in the FEV1 after exercise is diagnostic” so that the relative change in FEV1 (in %) is the most reasonable outcome when assessing EIB, eg Mahler DA, 1993. Such facts are relevant when analyzing EIB trials.

Kaur B, 2001 considered that “exercise challenge” was a particular issue in the Cohen HA, 1997 and Schachter EN, 1982 studies, see the above extract. However, in Outcome 01 (p.10) the Cochrane review showed data on the pre-exercise FEV1 levels, which is not an outcome influenced by exercise. In Outcome 02 (p.10) the Cochrane review extracted data on the absolute post-exercise FEV1 level for Cohen HA, 1997 and absolute FEV1 change for Schachter EN, 1982. These outcomes are not relevant, since the standard outcome for EIB is the relative change in FEV1 (in %), see above.

Furthermore, the greatest decline in FEV1 occurs about 8 to 15 min after the end of exercise, above. However, in Outcome 02 (p.10), Kaur B, 2001 shows Schachter's FEV1 changes immediately after exercise in Schachter's Table II, instead of FEV1 changes 5 min after exercise which Schachter reported in Table III. The latter is much more relevant in a study on EIB, since “EIA symptoms ... peak 8 to 15 minutes after exercise”, Mahler DA, 1993.

Kaur B, 2001 (p.4) write that “Anah did not report data in a manner that permitted further analysis”. However, in the following paragraph, Kaur B, 2001 wrote that, in the Anah study, “there were 9 exacerbations in the intervention group which had 22 patients and 35 exacerbations in the placebo group which had 19 patients ”. Thus, Anah CO, 1980 did report data that can be statistically analyzed, see below.

Errors in the analysis of data

Schachter EN, 1982 and Cohen HA, 1997 were cross-over studies so that the same participants were randomly administered vitamin C or the placebo. Such data should be analyzed by the paired t-test.

In Outcomes 01 and 02 (p.10), Kaur B, 2001 analyzed Schachter and Cohen data by the unpaired t-test, which is unsound for paired data. Kaur B, 2001 reported no difference between the vitamin C and placebo days for 11 of Cohen's participants with P=0.4 (unpaired t-test), whereas the correct paired t-test yields P=0.003.

Furthermore, given that the goal was to examine the effect of vitamin C on EIB, Kaur B, 2001 should have calculated the relative changes in FEV1 (in %), see above.

Schachter EN, 1982 published the data for all their 12 participants, and Kaur B, 2001 could have used the paired t-test to analyze the relative changes.

Similarly, Kaur B, 2001 could have calculated the relative changes for the 11 participants published by Cohen HA, 1997, and could have imputed eg “no effect” for the 9 participants who had missing data.

Nevertheless, using the paired t-test is a simplistic analysis of FEV1 changes, since it is possible that the effect of vitamin C depends on the severity of EIB. Hemilä H, 2013 analyzed the two studies using linear regression. Schachter EN, 1982 showed a 55% reduction in the postexercise FEV1 decline (95%CI 32% to 78%) with vitamin C. Imputation of “no effect” to the 9 participants with missing data of Cohen HA, 1997 yielded an estimated 42% reduction in the postexercise FEV1 decline (19% to 64%) with vitamin C. Finally, Cohen HA, 1997 also published the EIB status, which has no missing data. On the placebo day, 100% (20/20) of participants suffered from EIB, whereas on the vitamin C day, only 50% (10/20) suffered from EIB. Thus vitamin C caused a 50 percentage point decrease (23 to 68) in the occurrence of EIB, see Hemilä H, 2013.

Kaur B, 2001 (p. 4) claimed that “Anah did not report data in a manner that permitted further analysis,” which is incorrect. Anah CO, 1980 reported that in the placebo group (n=19), there were 35 asthma attacks, but in the vitamin C group (n=22), only 9 attacks. By using the standard Poisson approach, these data give RR=0.22 (0.09 to 0.47). Anah did not publish the individual level data and it might be over-dispersed, see Glynn RJ, 1996. Nevertheless, Anah published partial descriptions of the asthma attack distributions which can be used to impute more realistic distributions for the treatment groups, yielding a 95%CI of 0.06 to 0.81, which is still far from the null effect, see Hemilä H, 2013. Furthermore, the exact distribution of severe and moderate asthma attacks in the vitamin C group can be inferred from Anah CO, 1980 and this outcome needs few imputations. Vitamin C reduced the incidence of severe and moderate asthma attacks by RR=0.11 (0.02 to 0.48).

In conclusion, had Kaur B, 2001 extracted data correctly, and had they properly carried out the statistical analysis, they could have concluded in 2001 that there was strong evidence that vitamin C was beneficial against EIB in Cohen HA, 1997 and in Schachter EN, 1982, and against common cold-induced asthma exacerbations in Anah CO, 1980.

Secondary analysis of the Cohen HA, 1997, Schachter EN, 1982 and Anah CO, 1980 studies has been carried out by Hemilä H, 2013, Hemilä H, 2013, and Hemilä H, 2014.