On 2017 Jan 11, Kevin Hall commented:

Dr. Ludwig and his co- authors claimed in their 2012 JAMA article that “the main strength of our study was use of a controlled feeding protocol to establish weight stability following weight loss”. However, in the edited PubMed Commons comment below, Dr. Ludwig now reveals that the study did not establish weight stability but actually led to weight loss during all three test diets (VLC: -0.78 kg; LGI: -0.76 kg; LF: -0.23 kg). These newly revealed weight losses are qualitatively consistent with an overall state of negative energy balance as indicated by the reported energy intake and TEE measurements, although the mean weight losses remain quantitatively somewhat lower than might be expected based on the reported ~200-500 kcal/d negative energy balance. Furthermore, the 0.55 kg mean difference in weight loss during the VLC versus LF diets amounts to an approximate energy imbalance of ~150 kcal/d which is about half the reported mean difference in TEE. Without body composition measurements, it is impossible to be more definitive regarding the concordance between the mean reported TEE, energy balance, and weight loss.

On 2016 Jun 08, DAVID LUDWIG commented:

In the post below (dated June 7, 2016), Kevin Hall claims that the main findings of our JAMA 2012 study are false because of fundamental inconsistencies and measurement problems. On behalf of my coauthors Cara Ebbeling and Henry Feldman, I address each of Hall’s criticisms below.

1. Was total energy expenditure mismeasured? Hall bases his argument on the difference between calculated energy intake (in prepared meals) and total energy expenditure (TEE) of about 300 kcal/d. However, this observed difference, 10% of TEE, is quite small compared to reported discrepancies several fold that magnitude in outpatient behavioral studies in which participants consume self-prepared meals. Because our participants were not kept in a locked ward for the 7-month protocol, it is likely that some non-study foods were consumed – providing a simple explanation for the observed discrepancy. However, Hall offers no reason to believe that this small degree of non-compliance would invalidate results of the stable isotope measurement, let alone produce a systematic error favoring the very-low-carbohydrate. Indeed, multiple biomeasures of macronutrient composition (such as RQ, triglyceride, non-HDL-cholesterol, HDL-cholesterol) changed strongly and in the hypothesized directions – evidence that the feeding protocol produced substantive differences in dietary intakes as intended. Should we also dismiss those objective findings based on the possibility of marginal non-compliance? By this reasoning, no meaningful interventional nutrition research could ever be conducted for more than a month or two, the practical limits of human research under confinement. In fact, feeding protocols such as ours provide an important design alternative to the short-term locked-ward studies such as those of Hall (with poor generalizability) Hall KD, 2015 and dietary behavioral counseling trials (often with lack of attention to differentiation between diets). Furthermore, our estimates of the diet effects on TEE compare well with those of Hall himself, based on his recent metabolic ward study.

2. Does body weight on the test diets indicate internal inconsistency? Hall claims that the very-low-carbohydrate diet should have produced a 1 kg weight loss as a result of the 325 kcal/d greater TEE compared to the low-fat diet, and the lack of a significant difference in this regard leads to the “inescapable conclusion” of internal inconsistency and fundamental error. However, body weight is well recognized to be an imprecise measure of energy balance over the short term. Body weight can vary by 2 kg or more on a daily basis related to hydration status, amount of stool in the colon, and other physiological factors. Moreover, since body fat holds much more energy than lean tissue, energy balance may shift by thousands of calories without translation into weight change, if relatively small changes in body composition have occurred. (Diets that reduce insulin secretion have been shown to alter substrate partitioning favoring lean tissue, as for example Pawlak DB, 2004.) In addition, TEE was measured during the final 2 weeks of each test diet. Hall assumes the TEE differences emerged immediately, but the literature suggests that the process of fat adaptation may take 1 or 2 weeks Hawley JA, 2011 Vazquez JA, 1992 Veum VL, 2017. Finally, Hall assumes that the observed weight at the end of each test diet represents the effects of that diet alone, neglecting the cross-over design. In fact, the weight at any timepoint represents the cumulative effects of prior diet arms. Based on the randomized design, each test diet would come after at least one of the other diets two-thirds of the time. We analyzed change in body weight occurring during each test diet, and consistent with the TEE findings, results were numerically greater on the very-low-carbohydrate versus low-fat diet, by 0.55 kg (VLC: -0.78; LGI: -0.76; LF: -0.23 kg). (NB, this small overall weight loss averaged only 20 g/d -- if there were any minimal confounding as a result, it would have biased against the VLC and toward the null hypothesis.)

3. Is the resting energy expenditure finding meaningless? Resting energy expenditure (REE) was obtained through an independent method (indirect calorimetry by measurement of respiratory gases) and yielded a result consistent with TEE. However, Hall dismisses this finding too, arguing that the observed 67 kcal/day difference was too small to be meaningful. He disregards that this statistically significant difference would represent an entirely novel effect of dietary composition not recognized in the conventional approach to obesity treatment. By his own calculations, an energy gap of 1/10th this magnitude (30 kJ or about 7 kcal per day) “underlies the observed average weight gain” throughout the population Hall KD, 2011. Moreover, REE represents only one component of energy expenditure, which is why we also studied TEE (for which the observed difference was substantially larger). In any event, the aim of our relatively small study wasn’t to establish precise estimates of effect sizes, but rather to explore whether macronutrient composition might attenuate the biological adaptations to weight loss that antagonize successful weight loss maintenance.

4. Does dietary protein fatally confound study findings? The very-low-carbohydrate diet had 30% protein (consistent with the initial phase of the Atkins program, upon which this diet was modeled) compared to 20% for the other 2 diets. A protein difference of this magnitude can’t explain differences in REE in the fasting state, long after the thermic effects of food have dissipated. Nor could this difference explain the 325 kcal/day difference in TEE. Listed below are 10 studies involving differences in protein intake equal to or greater than that in our study, and also within the physiological range for dietary protein. In no case did TEE differ by more than 100kcal/day, and the average effect considering all studies was virtually null. Dulloo AG, 1999 Hochstenbach-Waelen A, 2009 Lejeune MP, 2006 Luscombe ND, 2003 Mikkelsen PB, 2000 Veldhorst MA, 2009 Veldhorst MA, 2010 Westerterp KR, 1999 Westerterp-Plantenga MS, 2009 Whitehead JM, 1996

5. Were the statistical methods faulty? Hall states that because we examined multiple secondary outcomes, the probability is “quite high” that any statistically significant results (and those for TEE in particular) were false positives, but this assertion lacks merit. We reported 22 secondary outcomes, including 20 in the table of study outcomes and 2 in the text, with statistical significance tests for diet trend ranging from p<0.0001 to p=0.78. We specified a threshold of p<0.05 for declaring significance, i.e., a 5% type I error rate. We can calculate the false discovery rate (FDR) according to the method of Benjamini and Hochberg, which takes into account the number of comparisons and their attained significance levels. This calculation reveals an overall 6.7% FDR, comparable in stringency to the 5% type I error rate. For the 22 tests of equality across the 3 diets (unordered, or “overall”), we calculate the FDR as 6.1%. For TEE, the risk of FDR is likely to be even lower, in view of its relatively robust statistical significance (p=0.003).

In summary, our feeding study obtained substantially greater differentiation between dietary treatments than conventional behavioral studies, as demonstrated by multiple biomeasures of compliance, allowing for a rigorous test of pre-specified study hypotheses. The outcomes were assessed with state-of-the-art techniques and analyzed according to accepted statistical methods. The manuscript passed rigorous peer-review at a selective journal. Hall’s claims of fatal problems involving the study findings are based on factual error and misinterpretation.

On 2016 Jun 07, Kevin Hall commented:

This interesting randomized controlled trial by Dr. David Ludwig’s group has been widely recognized as demonstrating a substantial metabolic advantage of carbohydrate restriction following a period of weight loss. The reported differences in total energy expenditure (TEE) amounted to as much as 325 kcal/d between isocaloric diets. However, in addition to the confounding differences in protein between the diets, there are several reasons to be skeptical about this conclusion. In particular, the data from this study are internally inconsistent at a fundamental level suggesting that some of the measurements were simply erroneous.

Following a run-in period of weight loss, subjects consumed three “weight-loss maintenance” diets in a randomized crossover fashion with each diet lasting one month. The very low carbohydrate (VLC) diet had ~50% more protein than both the low glycemic index (LGI) diet and the low fat (LF) diet, but energy intake was claimed to be held constant at ~2600 kcal/d. However, the measured TEE was between ~200-500 kcal/d greater than the reported energy intake for all diets. Therefore, the corresponding negative energy balance should have resulted in several kilograms of weight loss over the three month period consuming these diets. However, no such weight loss occurred since the mean body weight at the end of the run-in weight loss phase (105.0 kg minus 14.3 kg of lost weight = 90.7 kg) was slightly lower than the reported body weights during the different diets (91.5 kg for the LF diet; 91.1 kg for the LGI diet; and 91.2 kg for the VLC diet) which were not significantly different from each other.

What could be responsible for these inconsistencies in the data? It is highly likely that the energy intake measurements were inaccurate since diet adherence during outpatient feeding studies is typically poor even when all study foods are provided. If the energy intake measurements were consistently biased (such that energy intake was underestimated equally for all three diets) then the relative constancy of body weight suggests that TEE during the isocaloric VLC diet could not have been ~300 kcal/d greater than the LF diet. Such a difference in energy balance should have led to a cumulative difference in stored energy amounting to ~9000 kilocalories which translates to more than 1 kg of weight difference between the diets over the one month diet period. Since this weight difference was not observed, the inescapable conclusion is that the body weight, energy intake, and TEE data from this study are internally inconsistent and at least one of these measurements is fundamentally in error.

Since the body weight data are likely correct, either the subjects were eating ~300 kcal/d more during the VLC diet as compared to the LF diet (in which case the study was poorly controlled) or the TEE data were simply erroneous. The latter explanation is quite likely since the observed differences in TEE between the diets may have been statistical anomalies. In particular, the reported statistical analyses did not adequately address the multiple comparisons problem for this secondary study outcome which was one of 25 listed in the registration of this clinical trial. Therefore, the chance of obtaining a false positive for any one of these 25 secondary outcomes was quite high. Previous studies investigating the effects of isocaloric diets on energy expenditure have not seen such large effects, thereby adding further support to the conclusion that the TEE differences reported in this study were unlikely to be real.

Interestingly, the primary endpoint of this study was resting energy expenditure (REE) and the high protein VLC diet was the only diet showing a statistically significant difference compared to the LF diet. However, the magnitude of the REE effect was only ~67 kcal/d and was therefore clinically insignificant. The moderate carbohydrate, LGI diet with protein and calories matched to the LF diet failed to show a statistically significant difference in either REE or TEE despite a 34% decrease in carbohydrate compared to the LF diet. In other words, when comparing diets differing in protein, the primary REE outcome of the study showed a small effect that has been largely ignored and the secondary TEE outcome showed a large effect that was likely to be a false positive.

In conclusion, this study suffered from the same pitfalls that are typical of outpatient studies where poor diet adherence is the norm. Furthermore, the measurements were internally inconsistent and the reported beneficial effects of carbohydrate restriction on energy expenditure are likely to be incorrect.

On 2016 Jun 07, Kevin Hall commented:

This interesting randomized controlled trial by Dr. David Ludwig’s group has been widely recognized as demonstrating a substantial metabolic advantage of carbohydrate restriction following a period of weight loss. The reported differences in total energy expenditure (TEE) amounted to as much as 325 kcal/d between isocaloric diets. However, in addition to the confounding differences in protein between the diets, there are several reasons to be skeptical about this conclusion. In particular, the data from this study are internally inconsistent at a fundamental level suggesting that some of the measurements were simply erroneous.

Following a run-in period of weight loss, subjects consumed three “weight-loss maintenance” diets in a randomized crossover fashion with each diet lasting one month. The very low carbohydrate (VLC) diet had ~50% more protein than both the low glycemic index (LGI) diet and the low fat (LF) diet, but energy intake was claimed to be held constant at ~2600 kcal/d. However, the measured TEE was between ~200-500 kcal/d greater than the reported energy intake for all diets. Therefore, the corresponding negative energy balance should have resulted in several kilograms of weight loss over the three month period consuming these diets. However, no such weight loss occurred since the mean body weight at the end of the run-in weight loss phase (105.0 kg minus 14.3 kg of lost weight = 90.7 kg) was slightly lower than the reported body weights during the different diets (91.5 kg for the LF diet; 91.1 kg for the LGI diet; and 91.2 kg for the VLC diet) which were not significantly different from each other.

What could be responsible for these inconsistencies in the data? It is highly likely that the energy intake measurements were inaccurate since diet adherence during outpatient feeding studies is typically poor even when all study foods are provided. If the energy intake measurements were consistently biased (such that energy intake was underestimated equally for all three diets) then the relative constancy of body weight suggests that TEE during the isocaloric VLC diet could not have been ~300 kcal/d greater than the LF diet. Such a difference in energy balance should have led to a cumulative difference in stored energy amounting to ~9000 kilocalories which translates to more than 1 kg of weight difference between the diets over the one month diet period. Since this weight difference was not observed, the inescapable conclusion is that the body weight, energy intake, and TEE data from this study are internally inconsistent and at least one of these measurements is fundamentally in error.

Since the body weight data are likely correct, either the subjects were eating ~300 kcal/d more during the VLC diet as compared to the LF diet (in which case the study was poorly controlled) or the TEE data were simply erroneous. The latter explanation is quite likely since the observed differences in TEE between the diets may have been statistical anomalies. In particular, the reported statistical analyses did not adequately address the multiple comparisons problem for this secondary study outcome which was one of 25 listed in the registration of this clinical trial. Therefore, the chance of obtaining a false positive for any one of these 25 secondary outcomes was quite high. Previous studies investigating the effects of isocaloric diets on energy expenditure have not seen such large effects, thereby adding further support to the conclusion that the TEE differences reported in this study were unlikely to be real.

Interestingly, the primary endpoint of this study was resting energy expenditure (REE) and the high protein VLC diet was the only diet showing a statistically significant difference compared to the LF diet. However, the magnitude of the REE effect was only ~67 kcal/d and was therefore clinically insignificant. The moderate carbohydrate, LGI diet with protein and calories matched to the LF diet failed to show a statistically significant difference in either REE or TEE despite a 34% decrease in carbohydrate compared to the LF diet. In other words, when comparing diets differing in protein, the primary REE outcome of the study showed a small effect that has been largely ignored and the secondary TEE outcome showed a large effect that was likely to be a false positive.

In conclusion, this study suffered from the same pitfalls that are typical of outpatient studies where poor diet adherence is the norm. Furthermore, the measurements were internally inconsistent and the reported beneficial effects of carbohydrate restriction on energy expenditure are likely to be incorrect.

On 2016 Jun 08, DAVID LUDWIG commented:

In the post below (dated June 7, 2016), Kevin Hall claims that the main findings of our JAMA 2012 study are false because of fundamental inconsistencies and measurement problems. On behalf of my coauthors Cara Ebbeling and Henry Feldman, I address each of Hall’s criticisms below.

1. Was total energy expenditure mismeasured? Hall bases his argument on the difference between calculated energy intake (in prepared meals) and total energy expenditure (TEE) of about 300 kcal/d. However, this observed difference, 10% of TEE, is quite small compared to reported discrepancies several fold that magnitude in outpatient behavioral studies in which participants consume self-prepared meals. Because our participants were not kept in a locked ward for the 7-month protocol, it is likely that some non-study foods were consumed – providing a simple explanation for the observed discrepancy. However, Hall offers no reason to believe that this small degree of non-compliance would invalidate results of the stable isotope measurement, let alone produce a systematic error favoring the very-low-carbohydrate. Indeed, multiple biomeasures of macronutrient composition (such as RQ, triglyceride, non-HDL-cholesterol, HDL-cholesterol) changed strongly and in the hypothesized directions – evidence that the feeding protocol produced substantive differences in dietary intakes as intended. Should we also dismiss those objective findings based on the possibility of marginal non-compliance? By this reasoning, no meaningful interventional nutrition research could ever be conducted for more than a month or two, the practical limits of human research under confinement. In fact, feeding protocols such as ours provide an important design alternative to the short-term locked-ward studies such as those of Hall (with poor generalizability) Hall KD, 2015 and dietary behavioral counseling trials (often with lack of attention to differentiation between diets). Furthermore, our estimates of the diet effects on TEE compare well with those of Hall himself, based on his recent metabolic ward study.

2. Does body weight on the test diets indicate internal inconsistency? Hall claims that the very-low-carbohydrate diet should have produced a 1 kg weight loss as a result of the 325 kcal/d greater TEE compared to the low-fat diet, and the lack of a significant difference in this regard leads to the “inescapable conclusion” of internal inconsistency and fundamental error. However, body weight is well recognized to be an imprecise measure of energy balance over the short term. Body weight can vary by 2 kg or more on a daily basis related to hydration status, amount of stool in the colon, and other physiological factors. Moreover, since body fat holds much more energy than lean tissue, energy balance may shift by thousands of calories without translation into weight change, if relatively small changes in body composition have occurred. (Diets that reduce insulin secretion have been shown to alter substrate partitioning favoring lean tissue, as for example Pawlak DB, 2004.) In addition, TEE was measured during the final 2 weeks of each test diet. Hall assumes the TEE differences emerged immediately, but the literature suggests that the process of fat adaptation may take 1 or 2 weeks Hawley JA, 2011 Vazquez JA, 1992 Veum VL, 2017. Finally, Hall assumes that the observed weight at the end of each test diet represents the effects of that diet alone, neglecting the cross-over design. In fact, the weight at any timepoint represents the cumulative effects of prior diet arms. Based on the randomized design, each test diet would come after at least one of the other diets two-thirds of the time. We analyzed change in body weight occurring during each test diet, and consistent with the TEE findings, results were numerically greater on the very-low-carbohydrate versus low-fat diet, by 0.55 kg (VLC: -0.78; LGI: -0.76; LF: -0.23 kg). (NB, this small overall weight loss averaged only 20 g/d -- if there were any minimal confounding as a result, it would have biased against the VLC and toward the null hypothesis.)

3. Is the resting energy expenditure finding meaningless? Resting energy expenditure (REE) was obtained through an independent method (indirect calorimetry by measurement of respiratory gases) and yielded a result consistent with TEE. However, Hall dismisses this finding too, arguing that the observed 67 kcal/day difference was too small to be meaningful. He disregards that this statistically significant difference would represent an entirely novel effect of dietary composition not recognized in the conventional approach to obesity treatment. By his own calculations, an energy gap of 1/10th this magnitude (30 kJ or about 7 kcal per day) “underlies the observed average weight gain” throughout the population Hall KD, 2011. Moreover, REE represents only one component of energy expenditure, which is why we also studied TEE (for which the observed difference was substantially larger). In any event, the aim of our relatively small study wasn’t to establish precise estimates of effect sizes, but rather to explore whether macronutrient composition might attenuate the biological adaptations to weight loss that antagonize successful weight loss maintenance.

4. Does dietary protein fatally confound study findings? The very-low-carbohydrate diet had 30% protein (consistent with the initial phase of the Atkins program, upon which this diet was modeled) compared to 20% for the other 2 diets. A protein difference of this magnitude can’t explain differences in REE in the fasting state, long after the thermic effects of food have dissipated. Nor could this difference explain the 325 kcal/day difference in TEE. Listed below are 10 studies involving differences in protein intake equal to or greater than that in our study, and also within the physiological range for dietary protein. In no case did TEE differ by more than 100kcal/day, and the average effect considering all studies was virtually null. Dulloo AG, 1999 Hochstenbach-Waelen A, 2009 Lejeune MP, 2006 Luscombe ND, 2003 Mikkelsen PB, 2000 Veldhorst MA, 2009 Veldhorst MA, 2010 Westerterp KR, 1999 Westerterp-Plantenga MS, 2009 Whitehead JM, 1996

5. Were the statistical methods faulty? Hall states that because we examined multiple secondary outcomes, the probability is “quite high” that any statistically significant results (and those for TEE in particular) were false positives, but this assertion lacks merit. We reported 22 secondary outcomes, including 20 in the table of study outcomes and 2 in the text, with statistical significance tests for diet trend ranging from p<0.0001 to p=0.78. We specified a threshold of p<0.05 for declaring significance, i.e., a 5% type I error rate. We can calculate the false discovery rate (FDR) according to the method of Benjamini and Hochberg, which takes into account the number of comparisons and their attained significance levels. This calculation reveals an overall 6.7% FDR, comparable in stringency to the 5% type I error rate. For the 22 tests of equality across the 3 diets (unordered, or “overall”), we calculate the FDR as 6.1%. For TEE, the risk of FDR is likely to be even lower, in view of its relatively robust statistical significance (p=0.003).

In summary, our feeding study obtained substantially greater differentiation between dietary treatments than conventional behavioral studies, as demonstrated by multiple biomeasures of compliance, allowing for a rigorous test of pre-specified study hypotheses. The outcomes were assessed with state-of-the-art techniques and analyzed according to accepted statistical methods. The manuscript passed rigorous peer-review at a selective journal. Hall’s claims of fatal problems involving the study findings are based on factual error and misinterpretation.

On 2017 Jan 11, Kevin Hall commented:

Dr. Ludwig and his co- authors claimed in their 2012 JAMA article that “the main strength of our study was use of a controlled feeding protocol to establish weight stability following weight loss”. However, in the edited PubMed Commons comment below, Dr. Ludwig now reveals that the study did not establish weight stability but actually led to weight loss during all three test diets (VLC: -0.78 kg; LGI: -0.76 kg; LF: -0.23 kg). These newly revealed weight losses are qualitatively consistent with an overall state of negative energy balance as indicated by the reported energy intake and TEE measurements, although the mean weight losses remain quantitatively somewhat lower than might be expected based on the reported ~200-500 kcal/d negative energy balance. Furthermore, the 0.55 kg mean difference in weight loss during the VLC versus LF diets amounts to an approximate energy imbalance of ~150 kcal/d which is about half the reported mean difference in TEE. Without body composition measurements, it is impossible to be more definitive regarding the concordance between the mean reported TEE, energy balance, and weight loss.