Abstract
The question of whether the ratio of dietary carbohydrates to fat substantially impacts total energy expenditure (TEE) or body fat has been investigated for decades, with most studies pointing to no clinically meaningful effect. However, a recent study by Ebbeling et al. (https://doi.org/10.1136/bmj.k4583) reported substantial differences in TEE between diets varying in their ratio of carbohydrate to fat. The original pre-registered statistical analysis plan for the primary study outcome of Ebbeling et al. addressed the question of whether the reduction in TEE during weight loss maintenance compared to the pre-weight loss baseline depended on the dietary carbohydrate to fat ratio. However, the final analysis plan was modified to make the diet comparisons with the TEE measurements collected in the immediate post-weight loss period rather than at the pre-weight loss baseline. Here, we reanalyzed the data according to the original plan we found that the TEE differences were no longer statistically significant between the diet groups and the nominal diet differences of ∼100 kcal/d were much smaller than the ∼250 kcal/d differences reported in the publication. In other words, when conducting the analysis originally planned by the authors we found that the significant increases in TEE with the low carbohydrate diet that were reported by Ebbeling et al. disappeared. Furthermore, the TEE effect modification by baseline insulin secretion also disappeared.
Introduction
The question of whether the ratio of dietary carbohydrates to fat substantially impacts total energy expenditure (TEE) or body fat has been investigated for decades, with most studies pointing to no clinically meaningful effect 1. However, a recent study by Ebbeling et al. reported substantial differences in TEE between diets varying in their ratio of carbohydrate to fat 2. This study was remarkable in that it provided the subjects with all their food and employed the gold-standard doubly labeled water method to measure free-living TEE during a prolonged 20-week period of weight loss maintenance. In a model of open and transparent communication that facilitated our analysis below, the authors provided access to the original clinical protocol, all protocol amendments, much of the individual subject data, statistical analysis plan, and well-documented statistical analysis code on the Open Science Framework website.
The original pre-registered statistical analysis plan for the primary study outcome of Ebbeling et al. addressed the question of whether the reduction in TEE during weight loss maintenance compared to the pre-weight loss baseline depended on the dietary carbohydrate to fat ratio – a design similar to a previous study by many of the same authors 3. However, the final analysis plan was modified to make the diet comparisons with the TEE measurements collected in the immediate post-weight loss period rather than at the pre-weight loss baseline. Here, we reanalyzed the data according to the original plan we found that the TEE differences were no longer statistically significant between the diet groups and the nominal diet differences of ∼100 kcal/d were much smaller than the ∼250 kcal/d differences reported in the publication. In other words, when conducting the analyses originally planned by the authors we found that the significant increases in TEE with the low carbohydrate diet that were reported by Ebbeling et al. disappeared. Furthermore, the TEE effect modification by baseline insulin secretion also disappeared.
Methods
We downloaded the individual subject data and SAS statistical analysis code on the Open Science Framework website (https://osf.io/rvbuy/). We used SAS (version 9.4). To ensure that we understood the well-documented SAS code, we first successfully reproduced the results reported by Ebbeling et al. regarding the primary TEE outcome and its effect modification by baseline insulin secretion when using the immediately post-weight loss anchor as the point of comparison. We then modified the SAS code to use the pre-weight loss baseline as the point of comparison. The data are reported as mean±SE.
Results and Discussion
Unlike most diet trials that test the effects of counseling people to change their diets, Ebbeling et al. provided the subjects with all their food and reported several biomarkers indicating that the different diet groups likely consumed significantly different amounts of dietary carbohydrate. However, the authors did not report that the total energy intake provided to the subjects was substantially less than TEE during the weight loss maintenance period such that subjects in the intention to treat group consumed (mean ± SE) 460 ± 46 kcal/d less than they expended (p<0.0001) and the subjects in the per-protocol group who maintained body weight to within ±2 kg of their post-weight loss value at randomization consumed 422 ± 47 kcal/d less than expended (p<0.0001). While weight stability is not necessarily indicative of unchanging body energy stores, and the measured body fat changes have yet to be reported, such large energy deficits indicate that the subjects were likely consuming a substantial amount of unaccounted food despite the controlled-feeding design.
When using the original statistical plan to compare the effect of the weight loss maintenance diets to the pre-weight loss baseline TEE, we found no significant diet differences in the reduction of TEE in the per protocol group; with the low, moderate, and high carbohydrate groups decreasing TEE by 262 ±72 kcal/d, 254±75 kcal/d, and 356±80 kcal/d, respectively, compared with the pre-weight loss baseline period (p=0.59 for the test of equivalence between the diets). The mean absolute weight losses at 10 and 20 weeks compared to the pre-weight loss baseline were well-matched and within 250 g between all diet groups (p>0.9), so differences in mean weight loss could not have obscured any diet effects. In the intention to treat group, we also found no significant differences in TEE between diet groups compared with the pre-weight loss baseline period; with the low, moderate, and high carbohydrate groups decreasing TEE by 240 ±64 kcal/d, 322±66 kcal/d, and 356±67 kcal/d, respectively (p=0.43 for the test of equivalence between the diets). Pairwise comparisons of TEE diet differences with respect to the pre-weight loss baseline were not significant between diets in either the intention to treat (p>0.21) or the per protocol groups (p>0.35) (Figure 1A). Similar results were obtained using weight-normalized TEE data or when considering only the final 20-week time point as was planned in the original protocol rather than the average of the 10-week and 20-week TEE values as reported in the publication (not shown).
One possible reason why the final analysis plan of Ebbeling et al. led to a substantial apparent TEE increase with the low carbohydrate diet in the per-protocol group was the unlucky event that the immediate post-weight loss period led to TEE values that happened to decrease by 392±71 kcal/d in the low-carbohydrate group, whereas TEE decreased by only 271±73 kcal/d and 282±75 kcal/d in the moderate and high carbohydrate groups, respectively (Figure 1B). Despite these measurements being obtained prior to diet randomization, and not being statistically significantly different, the ∼100 kcal/d greater TEE decrease in immediately post-weight loss in the low carbohydrate group compared to the other diet groups makes it possible that simple regression to the mean resulted in the subsequent reported increases in TEE in the low-carbohydrate group when using the post-weight loss anchor point. Indeed, there was no significant TEE difference between the moderate and high carbohydrate groups even using the post-weight loss TEE anchor as specified in the final analysis plan, but this comparison was not reported by Ebbeling et al.
The substantial effect modification of TEE by baseline insulin secretion observed by Ebbeling et al. in the per protocol group when using the post-weight loss TEE measurement as the anchor point was no longer significant when using the pre-weight loss TEE as the anchor point (p=0.36 for the test of equivalence between the diets). While not statistically significant, TEE was nominally 383±196 kcal/d greater in the low carbohydrate group than the high carbohydrate group for those subjects in the highest insulin secretion tertile (p=0.053). Normalizing TEE by body weight also did not result in a significant overall TEE effect modification by baseline insulin secretion (p=0.29 for the test of equivalence between the diets), but the TEE difference between the low and high carbohydrate diets in the highest insulin secretion tertile was 386±173 kcal/d (p=0.03).
Did the measured components of energy expenditure corroborate the suggested TEE differences between low and high carbohydrate groups in the per-protocol subjects in this highest insulin secretion tertile? They did not. Differences in resting energy expenditure (−32±49 kcal/d; p=0.52), total physical activity (45754±47821 counts/d; p=0.34), moderate to vigorous physical activity (−5±6 min/d; p=0.4), sedentary time (−9±30 min/d; p=0.77), skeletal muscle work efficiency at 10W (1±0.9 %; p=0.27), 25W (1.2±1.1 %; p=0.28) and 50W (0.5±0.8 %; p=0.48) were all not-significantly different between the low and high carbohydrate diets when compared to the pre-weight loss baseline. Nevertheless, we cannot rule out possible differences in the thermic effect of food, sleeping energy expenditure, or some other unmeasured factor contributing to TEE. Alternatively, the apparent TEE diet differences in this high insulin secretion group may have been due to chance. None of the p-values above have been adjusted for multiple comparisons.
As justification for our reanalysis, we note that most of the history of the study (7 of 8 versions of the protocol spanning from 2014-2016) the planned primary outcome calculations used the pre-weight loss TEE baseline as the anchor point for the subsequent diet comparisons during weight loss maintenance. Prior to unmasking the randomization blind, but after all cohorts had completed the trial, the final protocol amendment in 2017 altered the previously planned statistical analysis to use the post-weight loss TEE measurement as the anchor point to make the subsequent diet comparisons.
The reasons for the change in the analysis plan were not provided in the protocol amendment or the final statistical analysis plan, but the Supplemental Materials in the final publication provided three reasons. First, the post-weight loss TEE measurement was chosen as the new anchor point because it occurred closer to the point of diet randomization. Second, the pre-weight loss TEE measurement would be “strongly confounded by weight loss”. How this might happen and why the post weight loss measure would not be similarly affected is difficult to imagine. Finally, Ebbeling et al. argued that the pre-weight loss baseline would have been inappropriate because the TEE measurements were insufficiently accurate or precise and therefore the study would be under-powered. However, the power calculations in the protocol were based on TEE data from a pilot study using the pre-weight loss TEE measurements as the basis for comparing how different diets affected the absolute reduction in TEE during weight loss maintenance 3. The pilot study did not measure TEE in the period immediately post-weight loss and therefore could not have been used to power the recent study in question.
Despite a request by the BMJ Editors to report the results of their original analysis plan, Ebbeling et al. refused because they were “concerned that the additional analysis would provide no meaningful biological insights – that is, no useful information about the nature of the relationship between dietary composition and energy expenditure. Rather, inclusion of the additional analysis would tend to elevate and give undue attention to an error, and therefore potentially cause confusion.”
We believe that the revised analysis plan has caused confusion and that the original statistical analysis plan that used pre-weight loss TEE as the anchor point is preferable for several reasons. First, it specifically addresses the question of whether the typical reduction in TEE that accompanies maintenance of lost weight depends on the carbohydrate to fat ratio of the weight loss maintenance diet. Second, the revised plan is potentially confounded by the substantial adaptive thermogenesis that occurs immediately post-weight loss that typically becomes less severe after a period of energy balance and weight loss maintenance 4,5. Finally, the pre-weight loss baseline TEE measurements were obtained in the situation where the doubly labeled water method is routinely employed: free-living people maintaining their habitual weight. Ideally, a post-weight loss TEE measurement should have first stabilized subjects at the lower body weight for several weeks prior to dosing with doubly labeled water. In contrast, the post-weight loss TEE measurements conducted by Ebbeling et al. were obtained during the same 2-week weight stabilization period when diet calories were being progressively increased at a rate determined by each individual subject’s recent rate of weight loss. While the doubly labeled water method generally provides a robust and valid estimate of TEE, this situation of simultaneous refeeding immediately post-weight loss potentially introduces uncertainty into the conversion of CO2 production into TEE. For example, the daily respiratory quotient during this period was clearly not equal to the food quotient as was assumed by Ebbeling et al. While such an effect can be appropriately modeled 6, this was not done in their TEE calculations.
In conclusion, when analyzed using the original statistical plan that was not confounded by the immediate post-weight loss period, the data of Ebbeling et al. do not support the conclusion that the ratio of dietary carbohydrate to fat affects the reduction in TEE during weight loss maintenance. While there are many reasons people could benefit from consuming healthy low carbohydrate diets 7, such diets are unlikely to substantially offset the usual reduction in TEE during maintenance of lost weight.