On 2015 Aug 14, Oliver T Mytton commented:

We note the comments for Kaiser et al. We are in the process of compiling a full response. When this is ready we will post it on the BMC Public Health website, linked directly to the article.

Oliver Mytton Honorary Specialty Registrar, Centre for Diet and Activity Research (CEDAR), MRC Epidemiology Unit, Cambridge.

On 2015 Jul 08, Kathryn Kaiser commented:

Having recently published a paper on the same topic [1], we read this paper (published August, 2014) by Mytton and colleagues (the authors) with great interest. We identified several important issues that vitiate confidence in the paper by the authors [2].

Errors of calculation, stated protocol violations, and documentation:

Our first concern pertains to differences in calculations in a study we both included, as well as the irreproducibility of the standard errors (SEs) reported in the Figure 2a forest plot of effects on body weight in [2]. First, we believe there is a calculation error in how the authors derived the effect size and variance in [2] for Whybrow, 2006 [3], which we both included. For clarity, we note the correct citation for the Whybrow paper is 2006, though it is sometimes cited as 2007 in [2]. Per [4], in order to reduce the potential for the over influence of studies where there are two experimental groups (300 grams per day and 600 grams per day in the case of Whybrow, 2006 [3]) compared to only a single control group, the control number (in this case, n=14) should be divided in half and rounded if necessary to the nearest integer for meta-analysis. We both calculated the same mean differences (ours reported as standardized mean differences), but substantially different confidence intervals. In the original report of outcomes [3], the effects on body weight for both treatment groups were not significant, whereas in Figure 2a [2], the confidence intervals are represented as showing a statistically significant effect for both treatment groups.

After further examination of standard errors for the other studies in included by the authors, and being unable to recreate several of the values reported in the paper [2], we recomputed the meta-analytic statistics from the source papers after making corrections to these values (see list below for all corrected estimates). For the overall analysis, this resulted in a smaller effect [-0.54 (-1.05,-0.03)] than reported in [2]. In a more appropriate analysis and using corrected standard errors, we removed an outlier [5] that the authors had identified in the “Comparison with other studies” section on page 9 of [2]. This study had a drop-out rate of over 52 percent, did not conduct an intent-to-treat analysis, and included only 9 subjects in the final analysis. When removing this paper as suggested in [2] (but not done in Figure 2), the overall effect is -0.31 (-0.70, 0.07), p=0.11, with a drop in heterogeneity from 72 to 53%.

Recalculated forest plot of included studies, excluding [5] are as follows (Mean Difference in Kg, 95% Confidence Interval, Standard Error): Smith-Warner 2000: (-0.09, -0.40 to 0.22, 0.16); Whybrow 2006 (600g group): (-0.62, -1.52 to 0.28, 0.46); Whybrow 2006 (300g group): (-0.77, -1.73 to 0.19, 0.49); Basu 2010: (-0.90, -1.88 to 0.08, 0.50); Peterson 2011: (0.40, -0.05 to 0.85, 0.23); Dow 2012: (-0.50, -1.30 to 0.30, 0.41); Christensen 2013: (-0.90, -2.17 to 0.37, 0.65).

Concerning study inclusion based on the stated protocol, the authors were not consistent in the application of their criteria. The authors opted to include one study of individuals with newly diagnosed diabetes [6], despite their exclusion criterion of studies with “subjects with a medical illness that was liable to lead to weight loss or weight gain.”[2] Excluding only this study also results in non-significant findings [excluding [6]: -0.51 (-1.06, 0.03); excluding both the outlier [5] and [6] results in: -0.27 (-0.67, 0.13)]. In other words, correcting the apparent errors of inclusion and effect size calculations changes the conclusions as stated in the paper [2].

Other points of note:

Although some differences between our papers are the result of simple methodological differences, such as our limitation to longer than eight week studies versus the authors limitation to four weeks, other differences are more substantial. We note that the authors [2] did not use the term “obesity” in their list of keywords, which is an important omission in a review looking at weight outcomes. In addition, the authors included three studies that investigated changes in only a single fruit or vegetable, while we opted to include studies where the intervention was a variety of fruits and vegetables, in order to evaluate the more general public health message that is common today. This difference in inclusion simply represents a nuanced difference between our analyses, but it is important that searches are designed to adequately answer the question at hand. For instance, in applying the stated criteria given by the authors, we found three examples [7-9] of other studies of a single fruit or vegetable that were not included by the authors but appear to meet criteria, calling into question the thoroughness of the literature search.

Lastly, in trying to reproduce the authors work, we noted that the list of included studies in Table 1 includes Singh, 1992 [10] while the final analysis does not. The reviewer comments and author replies posted on line at the BMC Public Health journal website indicate the authors followed the reviewer recommendation to remove the Singh paper from the analysis, yet it remains in the published table and appears to also be counted in the number in the PRISMA diagram, Figure 1. According to the forest plots in Figure 2, 7 studies were included for weight outcomes and 5 studies were included for energy intake outcomes. Figure 1 does not match the included studies in the search results reported.

Conclusion:

It is clear that the report presented by the authors has several concerning errors, is not consistent within itself, and does not support the conclusion the paper states that these interventions, “…result in either a small reduction in body weight or reduced weight gain relative to controls.”[2] We respectfully request that the authors reply to these concerns in order to correct the record.

Kathryn A. Kaiser, Ph.D. Andrew W. Brown, Ph.D. David B. Allison, Ph.D. Office of Energetics, Nutrition and Obesity Research Center, School of Public Health, University of Alabama at Birmingham

References

\1.\ Kaiser KA, 2014 \2.\ Mytton OT, 2014 \3.\ Whybrow S, 2006 \4.\ http://handbook.cochrane.org/ \5.\ Weerts SE, 2011 \6.\ Christensen AS, 2013 \7.\ Lehtonen HM, 2010 \8.\ Thies F, 2012 \9.\ Vafa MR, 2011 \10.\ Singh RB, 1992

On 2015 Jul 08, Kathryn Kaiser commented:

Having recently published a paper on the same topic [1], we read this paper (published August, 2014) by Mytton and colleagues (the authors) with great interest. We identified several important issues that vitiate confidence in the paper by the authors [2].

Errors of calculation, stated protocol violations, and documentation:

Our first concern pertains to differences in calculations in a study we both included, as well as the irreproducibility of the standard errors (SEs) reported in the Figure 2a forest plot of effects on body weight in [2]. First, we believe there is a calculation error in how the authors derived the effect size and variance in [2] for Whybrow, 2006 [3], which we both included. For clarity, we note the correct citation for the Whybrow paper is 2006, though it is sometimes cited as 2007 in [2]. Per [4], in order to reduce the potential for the over influence of studies where there are two experimental groups (300 grams per day and 600 grams per day in the case of Whybrow, 2006 [3]) compared to only a single control group, the control number (in this case, n=14) should be divided in half and rounded if necessary to the nearest integer for meta-analysis. We both calculated the same mean differences (ours reported as standardized mean differences), but substantially different confidence intervals. In the original report of outcomes [3], the effects on body weight for both treatment groups were not significant, whereas in Figure 2a [2], the confidence intervals are represented as showing a statistically significant effect for both treatment groups.

After further examination of standard errors for the other studies in included by the authors, and being unable to recreate several of the values reported in the paper [2], we recomputed the meta-analytic statistics from the source papers after making corrections to these values (see list below for all corrected estimates). For the overall analysis, this resulted in a smaller effect [-0.54 (-1.05,-0.03)] than reported in [2]. In a more appropriate analysis and using corrected standard errors, we removed an outlier [5] that the authors had identified in the “Comparison with other studies” section on page 9 of [2]. This study had a drop-out rate of over 52 percent, did not conduct an intent-to-treat analysis, and included only 9 subjects in the final analysis. When removing this paper as suggested in [2] (but not done in Figure 2), the overall effect is -0.31 (-0.70, 0.07), p=0.11, with a drop in heterogeneity from 72 to 53%.

Recalculated forest plot of included studies, excluding [5] are as follows (Mean Difference in Kg, 95% Confidence Interval, Standard Error): Smith-Warner 2000: (-0.09, -0.40 to 0.22, 0.16); Whybrow 2006 (600g group): (-0.62, -1.52 to 0.28, 0.46); Whybrow 2006 (300g group): (-0.77, -1.73 to 0.19, 0.49); Basu 2010: (-0.90, -1.88 to 0.08, 0.50); Peterson 2011: (0.40, -0.05 to 0.85, 0.23); Dow 2012: (-0.50, -1.30 to 0.30, 0.41); Christensen 2013: (-0.90, -2.17 to 0.37, 0.65).

Concerning study inclusion based on the stated protocol, the authors were not consistent in the application of their criteria. The authors opted to include one study of individuals with newly diagnosed diabetes [6], despite their exclusion criterion of studies with “subjects with a medical illness that was liable to lead to weight loss or weight gain.”[2] Excluding only this study also results in non-significant findings [excluding [6]: -0.51 (-1.06, 0.03); excluding both the outlier [5] and [6] results in: -0.27 (-0.67, 0.13)]. In other words, correcting the apparent errors of inclusion and effect size calculations changes the conclusions as stated in the paper [2].

Other points of note:

Although some differences between our papers are the result of simple methodological differences, such as our limitation to longer than eight week studies versus the authors limitation to four weeks, other differences are more substantial. We note that the authors [2] did not use the term “obesity” in their list of keywords, which is an important omission in a review looking at weight outcomes. In addition, the authors included three studies that investigated changes in only a single fruit or vegetable, while we opted to include studies where the intervention was a variety of fruits and vegetables, in order to evaluate the more general public health message that is common today. This difference in inclusion simply represents a nuanced difference between our analyses, but it is important that searches are designed to adequately answer the question at hand. For instance, in applying the stated criteria given by the authors, we found three examples [7-9] of other studies of a single fruit or vegetable that were not included by the authors but appear to meet criteria, calling into question the thoroughness of the literature search.

Lastly, in trying to reproduce the authors work, we noted that the list of included studies in Table 1 includes Singh, 1992 [10] while the final analysis does not. The reviewer comments and author replies posted on line at the BMC Public Health journal website indicate the authors followed the reviewer recommendation to remove the Singh paper from the analysis, yet it remains in the published table and appears to also be counted in the number in the PRISMA diagram, Figure 1. According to the forest plots in Figure 2, 7 studies were included for weight outcomes and 5 studies were included for energy intake outcomes. Figure 1 does not match the included studies in the search results reported.

Conclusion:

It is clear that the report presented by the authors has several concerning errors, is not consistent within itself, and does not support the conclusion the paper states that these interventions, “…result in either a small reduction in body weight or reduced weight gain relative to controls.”[2] We respectfully request that the authors reply to these concerns in order to correct the record.

Kathryn A. Kaiser, Ph.D. Andrew W. Brown, Ph.D. David B. Allison, Ph.D. Office of Energetics, Nutrition and Obesity Research Center, School of Public Health, University of Alabama at Birmingham

References

\1.\ Kaiser KA, 2014 \2.\ Mytton OT, 2014 \3.\ Whybrow S, 2006 \4.\ http://handbook.cochrane.org/ \5.\ Weerts SE, 2011 \6.\ Christensen AS, 2013 \7.\ Lehtonen HM, 2010 \8.\ Thies F, 2012 \9.\ Vafa MR, 2011 \10.\ Singh RB, 1992

On 2015 Aug 14, Oliver T Mytton commented:

We note the comments for Kaiser et al. We are in the process of compiling a full response. When this is ready we will post it on the BMC Public Health website, linked directly to the article.

Oliver Mytton Honorary Specialty Registrar, Centre for Diet and Activity Research (CEDAR), MRC Epidemiology Unit, Cambridge.