On 2015 Aug 06, David Keller commented:

Possible bias in the reporting of test results for this weight-loss product

The main author of this study reports conflicts of interest with regard to the green-plant membrane weight-loss product being tested, which is sold under the brand name "Appethyl" [1]. Her positions as scientific adviser, part-owner and member of the board of directors of companies that produce Appethyl could bias her evaluation of its efficacy. Results from her research are presented in promotional materials for Appethyl [2] and quoted in the popular media [3].

An example of possible bias involves two subjects who dropped out of this study after they were randomized to the control group. Their weight data was simply deleted, yielding an average weight loss in the control group of 3.5 kg. Intention-to-treat analysis, performed by carrying forward the starting weight in Table 1 for each of the control group drop-outs, yields an average weight loss in the control group of 3.9 kg. A smaller weight loss in the control group makes the Appethyl-treated group's average weight loss appear larger and more significant, by comparison to placebo. The U.S. Food and Drug Administration requires intention-to-treat analysis in studies of products submitted for their approval [4], which would be required if Appethyl is to be offered in the U.S.

The NIH suggests the following measures be taken to ameliorate financial conflicts of interest [5]:

"Change of personnel or personnel responsibilities, or disqualification of personnel from participation in all or a portion of the research;

"Reduction or elimination of the financial interest (e.g., sale of an equity interest); or

"Severance of relationships that create financial conflicts."

A safe and effective weight loss product derived entirely from natural green plant sources is a welcome possibility. The existence of significant conflicts of interest requires great care to prevent bias in the reporting of experimental results for Appethyl.

References:

1: Science Direct website, accessed on 8/6/2015, at the following URL: http://www.sciencedirect.com/science/article/pii/S0195666314003493

2: Online press release published 3/10/2014, "New Clinical [sic] Shows That Greenleaf Medical's Appethyl(TM) Decreases Urge to Eat Sweet Foods", SOURCE: Greenleaf Medical AB, accessed on 8/20/2015 at http://www.marketwatch.com/story/new-clinical-shows-that-greenleaf-medicals-appethyltm-decreases-urge-to-eat-sweet-foods-2014-03-10

3: Commercial sale of Appethyl on Amazon.com, accessed on 9/12/2015, at: http://www.amazon.com/Appethyl-30-ct/dp/B00RM5KY48/ref=cm_cr_pr_product_top?ie=UTF8

4: Guideline for the Format and Content of the Clinical and Statistical Sections of Applications, Center for Drug Evaluation and Research, Food and Drug Administration, Department of Health and Human Services, 5600 Fishers Lane Rockville, Maryland 20857 (301) 443-4330. 1988 Jul 24

5: ELECTRONIC CODE OF FEDERAL REGULATIONS, Subpart F—Promoting Objectivity in Research, accessed on 8/20/2015 at http://www.ecfr.gov/cgi-bin/text-idx?c=ecfr&SID=992817854207767214895b1fa023755d&rgn=div5&view=text&node=42:1.0.1.4.23&idno=42#sp42.1.50.f

On 2015 Sep 14, David Keller commented:

Blinding of study subjects was not assessed, and likely failed, causing Pygmalion effect

In this study, the control group subjects were given a beverage containing 2.8 grams of rapeseed oil and 50 grams of blueberry "soup". The active intervention group subjects were given the same beverage, with the addition of 5 grams of a commercial green spinach extract powder called "Appethyl". No assessment was made of whether the subjects remained blinded after seeing and tasting these beverages, which must have differed markedly in appearance and taste. It is difficult to imagine that the study subjects would not quickly figure out who was getting the placebo beverage, and who was getting the active intervention beverage. Once this was known, uncontrolled Pygmalion effect [1] is likely in a weight loss clinical trial. The Pygmalion effect is defined as enhanced benefit that occurs when a subject becomes aware that she is receiving the active experimental intervention. Pygmalion effect is closely related to placebo effect; in the limit, if the product being studied is actually ineffective, then the Pygmalion effect becomes equal to the placebo effect.

Clinical studies should evaluate the quality of the blinding achieved and maintained throughout the study, in order to exclude or quantify the biases and and confounding factors introduced when subjects know whether they received placebo or the experimental therapy. Assessment of blinding quality could have been obtained by asking the subjects to provide their best guess as to whether they received the placebo beverage with or without added Appethyl, and tracking those results over time. With perfect blinding, all subjects should guess with the random accuracy of a fairly flipped coin, on the average (50%). With complete unblinding, all subjects guess correctly whether they received placebo or the therapy being studied.

Perhaps Higgs and Geary can reveal further heretofore unpublished data to settle this question.

References:

1: Draper,S.W. The Hawthorne, Pygmalion, Placebo and other effects of expectation, published on 5/14/2002, and visited on 5/14/2015, at:<br> http://www.psy.gla.ac.uk/~steve/hawth.html#Hawthorne overall

2: Bang H, Ni L, Davis CE. Assessment of blinding in clinical trials. Control Clin Trials. 2004 Apr;25(2):143-56. PubMed PMID: 15020033.

On 2015 Aug 04, David Keller commented:

The data in Table 1 contain important discrepancies, which require discussion & correction

Table 1 states that the average starting weight for the 19 control subjects who began the trial was 80.2 kg, and the average ending weight was 76.3 kg for the 17 control subjects who finished the trial. Given this data, one can calculate the average weight loss of 3.9 kg by intention-to-treat analysis using the "last data point carried forward" technique. The authors of this paper chose to report the per-protocol average weight loss, which is impossible to calculate from Table 1, because it is missing the average starting weight of the 17 control patients who completed the study. None of this was disclosed anywhere in the paper, nor was it disclosed that the weight loss results were not analyzed by intention-to-treat analysis, which preserves the statistical qualities of the original randomization. Deleting control patients from a study degrades its statistical randomness and increases the effect of confounding factors. That is why intention-to-treat analysis is the scientific standard for analyzing data from clinical trials.

Further, there is no way for readers to check these calculations or the integrity of the paper's conclusions, due to the data missing from Table 1. This is an important point, because dropping the data from the 2 control patients decreased the average weight loss by control group subjects from 3.9 kg to 3.5 kg. This, in turn, increased the apparent average net weight loss among subjects treated with Appethyl.

The editors of Appetite should encourage readers to report published errors, because such reports are the last line of defense of the truth in scientific publishing. False counter-charges of error made against such readers, such as in their reply to me, will only discourage readers from pointing out published errors in the future.

On 2015 Aug 04, David Keller commented:

The authors DID specify the data variability as SEM, not SD, as falsely claimed by Higgs and Geary

The following quote is taken from the last paragraph of the Materials and Methods section of this paper: "In figures and text, data are expressed as the mean,  plus or minus the standard error". There is no other statement in this paper defining the data format. Interpreted using the data format specified by the authors, the study's reported weight loss with Appethyl is not significantly better than achieved with placebo, which I reported in an earlier comment. In their reply above, Higgs & Geary, the co-editors of Appetite, state that the study's final results are actually reported in the format of the mean plus or minus the standard deviation, a format not specified anywhere in the paper. It is only by substituting the use of the undocumented SD variability format that the weight loss with Appethyl is made to appear significantly better than with placebo. If, indeed, the authors decided to change data formats, that should have been documented in the paper. Rather than thanking me for identifying this important error in the paper, the editors of Appetite attack my conclusion that Appethyl is statistically ineffective, which is the only possible interpretation of the data using the format specified by the authors. My observations exposed an important error in this paper, which they should be grateful to have corrected by a reader.

On 2015 Aug 04, Suzanne Higgs commented:

Dr. David Keller commented that, contrary to the authors' conclusion, green-plant membrane consumption did not significantly reduce body weight in this twelve-week trial, that the control weight-loss data in Table 1 and in the abstract are discrepant, that an intention-to-treat analysis was not done, and that work may have been biased by the authors' financial interests. The authors did not consistently describe the numbers of subjects in the control group and did not explicitly identify whether the variabilities reported in Table 1 were SD or SEM. Dr. Keller’s suspicions regarding the significance of the weight-change data arise from this lack of clarity. In fact, the variabilities reported in the abstract and Table 1 are mean ± SEM and the mean (95% CI) three-month weight losses are 5.0 (3.97 – 6.03) kg for the 19 subjects receiving green-plant-membrane and 3.5 (2.41 – 4.59) kg for the 17 of the 19 initial control subjects who finished the study, and these two means are significantly different by t-test. A correction to the original article has now been published that clarifies these issues. The analytic approach, which was not an intention-to-treat approach, and how the missing data were handed are described in the methods and results section. Finally, the senior author fully reported her conflicts of interest to the satisfaction of the journal. Appetite appreciates Dr Keller's close study of this report.

Suzanne Higgs, PhD

Nori Geary, PhD

Co-Executive Editors, Appetite

On 2015 Jun 29, David Keller commented:

Results of this study are reported inconsistently and without intention-to-treat analysis

The weight-loss results of this study are inconsistently reported as the mean value, plus or minus the standard error of the mean, or as the mean value plus or minus the standard deviation. This uncertainty makes it impossible for the reader to evaluate the results of the experiment, and the senior investigator (C. Erlanson-Albertsson) has acknowledged this error and described plans to correct it [1].

In addition, there is an apparent discrepancy between the average weight loss in the control group, calculated from Table 1 of the paper as 3.9 kg, but reported in the abstract as 3.5 kg. The editor of Appetite has acknowledged that this difference is due to the fact that the results are reported by per-protocol analysis, rather than by intention-to-treat analysis, and that the paper does not disclose that fact. Addressing the lack of intention-to-treat analysis, she pointed out that "the study is not registered as a clinical trial in the USA, and it was not funded by the US government, so does not fall under its guidelines." [2] However, according to the Cochrane Reviews, "An intention-to-treat (ITT) analysis is often recommended as the least biased way to estimate intervention effects in randomized trials" [3]

Reference

1: Erlanson-Albertsson C, private correspondence, August 7, 2015

2: Higgs, S, private correspondence, August 10, 2015

3: Cochrane Handbook, Section 8.13.1, Rationale for concern about bias, accessed online on 9/10/2015 at the following website: http://handbook.cochrane.org/chapter_8/8_13_1_rationale_for_concern_about_bias.htm

On 2015 Jun 29, David Keller commented:

Results of this study are reported inconsistently and without intention-to-treat analysis

The weight-loss results of this study are inconsistently reported as the mean value, plus or minus the standard error of the mean, or as the mean value plus or minus the standard deviation. This uncertainty makes it impossible for the reader to evaluate the results of the experiment, and the senior investigator (C. Erlanson-Albertsson) has acknowledged this error and described plans to correct it [1].

In addition, there is an apparent discrepancy between the average weight loss in the control group, calculated from Table 1 of the paper as 3.9 kg, but reported in the abstract as 3.5 kg. The editor of Appetite has acknowledged that this difference is due to the fact that the results are reported by per-protocol analysis, rather than by intention-to-treat analysis, and that the paper does not disclose that fact. Addressing the lack of intention-to-treat analysis, she pointed out that "the study is not registered as a clinical trial in the USA, and it was not funded by the US government, so does not fall under its guidelines." [2] However, according to the Cochrane Reviews, "An intention-to-treat (ITT) analysis is often recommended as the least biased way to estimate intervention effects in randomized trials" [3]

Reference

1: Erlanson-Albertsson C, private correspondence, August 7, 2015

2: Higgs, S, private correspondence, August 10, 2015

3: Cochrane Handbook, Section 8.13.1, Rationale for concern about bias, accessed online on 9/10/2015 at the following website: http://handbook.cochrane.org/chapter_8/8_13_1_rationale_for_concern_about_bias.htm

On 2015 Aug 06, David Keller commented:

Possible bias in the reporting of test results for this weight-loss product

The main author of this study reports conflicts of interest with regard to the green-plant membrane weight-loss product being tested, which is sold under the brand name "Appethyl" [1]. Her positions as scientific adviser, part-owner and member of the board of directors of companies that produce Appethyl could bias her evaluation of its efficacy. Results from her research are presented in promotional materials for Appethyl [2] and quoted in the popular media [3].

An example of possible bias involves two subjects who dropped out of this study after they were randomized to the control group. Their weight data was simply deleted, yielding an average weight loss in the control group of 3.5 kg. Intention-to-treat analysis, performed by carrying forward the starting weight in Table 1 for each of the control group drop-outs, yields an average weight loss in the control group of 3.9 kg. A smaller weight loss in the control group makes the Appethyl-treated group's average weight loss appear larger and more significant, by comparison to placebo. The U.S. Food and Drug Administration requires intention-to-treat analysis in studies of products submitted for their approval [4], which would be required if Appethyl is to be offered in the U.S.

The NIH suggests the following measures be taken to ameliorate financial conflicts of interest [5]:

"Change of personnel or personnel responsibilities, or disqualification of personnel from participation in all or a portion of the research;

"Reduction or elimination of the financial interest (e.g., sale of an equity interest); or

"Severance of relationships that create financial conflicts."

A safe and effective weight loss product derived entirely from natural green plant sources is a welcome possibility. The existence of significant conflicts of interest requires great care to prevent bias in the reporting of experimental results for Appethyl.

References:

1: Science Direct website, accessed on 8/6/2015, at the following URL: http://www.sciencedirect.com/science/article/pii/S0195666314003493

2: Online press release published 3/10/2014, "New Clinical [sic] Shows That Greenleaf Medical's Appethyl(TM) Decreases Urge to Eat Sweet Foods", SOURCE: Greenleaf Medical AB, accessed on 8/20/2015 at http://www.marketwatch.com/story/new-clinical-shows-that-greenleaf-medicals-appethyltm-decreases-urge-to-eat-sweet-foods-2014-03-10

3: Commercial sale of Appethyl on Amazon.com, accessed on 9/12/2015, at: http://www.amazon.com/Appethyl-30-ct/dp/B00RM5KY48/ref=cm_cr_pr_product_top?ie=UTF8

4: Guideline for the Format and Content of the Clinical and Statistical Sections of Applications, Center for Drug Evaluation and Research, Food and Drug Administration, Department of Health and Human Services, 5600 Fishers Lane Rockville, Maryland 20857 (301) 443-4330. 1988 Jul 24

5: ELECTRONIC CODE OF FEDERAL REGULATIONS, Subpart F—Promoting Objectivity in Research, accessed on 8/20/2015 at http://www.ecfr.gov/cgi-bin/text-idx?c=ecfr&SID=992817854207767214895b1fa023755d&rgn=div5&view=text&node=42:1.0.1.4.23&idno=42#sp42.1.50.f