On 2017 May 03, Peter Hajek commented:

Meta-analyses do not normally use different approaches than a widely accepted standard and this field has one (see over a dozen Cochrane meta-analyses, Russell Standard, and any other norm in this field). As far as I know, no meta-analysis or individual study over the past 20 years or so included completers only. As explained earlier, the key point is that ‘missingness’ in this field is not considered random. If among 100 smokers who had treatment only 10 answer follow-up calls and report abstinence, the success rate is considered to be 10% rather than 100% as you would report it.

Re: studies that exclude treatment successes, imagine a treatment with good efficacy that helps 50% of patients, but only the 50% that were not helped are followed-up. These treatment failures may have worse outcomes than a random comparator group (they could have been treatment resistant e.g. because they have a more severe condition or other adverse circumstances). Your approach would interpret the finding as showing that the treatment is not only ineffective, but that it causes harm – when in fact it shows no such thing and is simply an artefact of the selection bias.

I appreciate that getting these things right can be difficult and would leave this alone if it was not such an important topic open to ideological misuse. And I agree that more studies are needed for the definitive answers to emerge.

On 2017 Apr 17, Regina El Dib commented:

In his comment, Dr. Hajek states that ‘in smoking cessation trials, drop-outs are classified as non-abstainers;. The approach to dealing with missing data in meta-analysis is differnet from that in trials. A survey of the methods literature identified four proposed approaches for dealing with missing outcome data when conducting a meta-analysis (https://www.ncbi.nlm.nih.gov/pubmed/26202162). All approaches recommended the use of a complete case analysis as the primary analysis. This is exactly how we conducted our meta-analysis (figure 5 in the published paper); the pooled relative ratio (RR) was 2.03 (95% CI 0.94 to 4.38) for smoking cessation with ENDS relative to ENNDS.

The same proposed approaches recommended additional sensitivity analyses using different imputation methods. The main purpose of these additional analyses is to assess the extent to which missing data may be biasing the findings of the primary analysis (https://www.ncbi.nlm.nih.gov/pubmed/23451162). Accordingly, we have conducted two sensitivity analyses respectively assuming that all participants with missing data had success or failure in smoking cessation. When assuming success, the pooled RR was 0.95 (95% CI 0.76 to 1.18, p=0.63) with ENDS relative to ENNDS; when assuming failure, the pooled RR was 2.27 (95% CI 1.04 to 4.95, p=0.04). This dramatic variation in the results when making different assumptions is clearly an indicator that the missingness of data is associated with a risk of bias, and that decreases our confidence in the results. We have already reflected that judgment in our risk of bias assessment of these two studies, in table 4 and figure 2; and in our assessment of the quality of evidence in table 7.

Even if we were going to consider the RR of 2.27 as the best effect estimate (i.e., assuming all those with missing data had failure with smoking cessation), the findings would not be supporting the effectiveness of e-cigarettes on smoking cessation. Indeed, the included trials do not address that question, and our review found no study comparing e-cigarettes to no e-cigarettes. The included trials compare two forms of e-cigarettes.

When assessing an intervention A (e.g., e-cigarettes) that has two types A1 (e.g., ENDS) and A2 (e.g., ENNDS), it would be important to first compare A (A1 and/or A2) to the standard intervention (e.g., no intervention or nicotine replacement therapy (NRT)), before comparing A1 to A2. If A1 and A2 are inferior to the standard intervention with A1 being less inferior than A2 (but still inferior to the standard intervention), focusing on the comparison of A1 to A2 (and ignoring the comparison to the standard intervention) will show that A1 is better than A2. That could also falsely suggest that at least A1 (and maybe A2) is favorable. Therefore, a recommendation of A1 vs. A2 should be considered only if A is already recommended over the standard intervention (i.e. A is non inferior to the standard intervention).

Dr. Hajek also criticizes the inclusion of studies that recruited smokers who used e-cigarettes in the past but continue to smoke. When discussing treatment and examining evidence, we refer to effectiveness (known as pragmatic; a treatment that works under real-world conditions). This includes (among other criteria) the inclusion all participants who have the condition of interest, regardless of their anticipated risk, responsiveness, comorbidities or past compliance. Therefore, the inclusion of studies that recruited smokers who used e-cigarettes in the past but continue to smoke had a role in portraying the impact of ENDS and ENNDS in cigarette smokers on long-term tobacco use.

On 2017 Mar 02, Wasim Maziak commented:

None

On 2017 Mar 02, Peter Hajek commented:

Wow! Let me try to explain these points in turn.

Re excluding drop-outs: Imagine 100 people get real treatment and 40 quit, 100 get placebo and 20 quit. All those who were successful attend for follow up (they feel good, grateful, get praised) but many less among treatment failures are willing to face the music (feel that they disappointed clinicians, may be told off, or feel that treatment was rubbish). If the same proportion of failures attend in each group (or if none attend), the success rate among attenders will be identical for both study arms, despite the real quit rates being 40% vs 20%. Check https://www.ncbi.nlm.nih.gov/pubmed/15733243 I cannot think of a mechanism through which this could act in the opposite way as you assert.

Re including irrelevant studies, your response provided no explanation for doing this.

Re. your statements about 'industry people' who should conduct independent science, I do not have and never had any links with any tobacco or e-cigarette manufacturers; and have published dozens of studies on smoking cessation treatments. I believe that anti-vaping activism presented as science needs challenging because misinforming smokers keeps them smoking and undermining much less risky alternatives to cigarettes protects cigarettes monopoly and harms public health.

On 2017 Mar 02, Wasim Maziak commented:

None

On 2017 Mar 01, Peter Hajek commented:

The conclusion that further trials of e-cigarettes are needed is correct, but there are two major problems with this review of the work that has been done so far.

In smoking cessation trials, drop-outs are classified as non-abstainers because treatment failures are less likely to engage in further contact than treatment successes. Practically all smoking cessation trials and reviews published in the last 10 years or so use this approach. Removing drop-outs from the sample, as was done here, dilutes any treatment effect.

The second serious issue is the inclusion of studies that recruited smokers who used e-cigarettes in the past but continue to smoke. Such studies have a higher proportion of treatment failures in the ‘tried e-cig cohort’ and so have less quitting in this subgroup, but they provide no useful information on the efficacy of e-cigarettes. Saying that they provide low quality evidence is wrong – they provide no evidence at all.

The narrative that some studies show that vaping helps quitting and some that it hinders misrepresents the evidence. No study showed that vaping hinders quitting smoking. The two RCTs with long-term outcome, if analysed in an unbiased way, show a positive effect despite controlling for sensorimotor effects and using low nicotine delivery products.

On 2017 Mar 01, Peter Hajek commented:

The conclusion that further trials of e-cigarettes are needed is correct, but there are two major problems with this review of the work that has been done so far.

In smoking cessation trials, drop-outs are classified as non-abstainers because treatment failures are less likely to engage in further contact than treatment successes. Practically all smoking cessation trials and reviews published in the last 10 years or so use this approach. Removing drop-outs from the sample, as was done here, dilutes any treatment effect.

The second serious issue is the inclusion of studies that recruited smokers who used e-cigarettes in the past but continue to smoke. Such studies have a higher proportion of treatment failures in the ‘tried e-cig cohort’ and so have less quitting in this subgroup, but they provide no useful information on the efficacy of e-cigarettes. Saying that they provide low quality evidence is wrong – they provide no evidence at all.

The narrative that some studies show that vaping helps quitting and some that it hinders misrepresents the evidence. No study showed that vaping hinders quitting smoking. The two RCTs with long-term outcome, if analysed in an unbiased way, show a positive effect despite controlling for sensorimotor effects and using low nicotine delivery products.

On 2017 Mar 02, Peter Hajek commented:

Wow! Let me try to explain these points in turn.

Re excluding drop-outs: Imagine 100 people get real treatment and 40 quit, 100 get placebo and 20 quit. All those who were successful attend for follow up (they feel good, grateful, get praised) but many less among treatment failures are willing to face the music (feel that they disappointed clinicians, may be told off, or feel that treatment was rubbish). If the same proportion of failures attend in each group (or if none attend), the success rate among attenders will be identical for both study arms, despite the real quit rates being 40% vs 20%. Check https://www.ncbi.nlm.nih.gov/pubmed/15733243 I cannot think of a mechanism through which this could act in the opposite way as you assert.

Re including irrelevant studies, your response provided no explanation for doing this.

Re. your statements about 'industry people' who should conduct independent science, I do not have and never had any links with any tobacco or e-cigarette manufacturers; and have published dozens of studies on smoking cessation treatments. I believe that anti-vaping activism presented as science needs challenging because misinforming smokers keeps them smoking and undermining much less risky alternatives to cigarettes protects cigarettes monopoly and harms public health.

On 2017 Apr 17, Regina El Dib commented:

In his comment, Dr. Hajek states that ‘in smoking cessation trials, drop-outs are classified as non-abstainers;. The approach to dealing with missing data in meta-analysis is differnet from that in trials. A survey of the methods literature identified four proposed approaches for dealing with missing outcome data when conducting a meta-analysis (https://www.ncbi.nlm.nih.gov/pubmed/26202162). All approaches recommended the use of a complete case analysis as the primary analysis. This is exactly how we conducted our meta-analysis (figure 5 in the published paper); the pooled relative ratio (RR) was 2.03 (95% CI 0.94 to 4.38) for smoking cessation with ENDS relative to ENNDS.

The same proposed approaches recommended additional sensitivity analyses using different imputation methods. The main purpose of these additional analyses is to assess the extent to which missing data may be biasing the findings of the primary analysis (https://www.ncbi.nlm.nih.gov/pubmed/23451162). Accordingly, we have conducted two sensitivity analyses respectively assuming that all participants with missing data had success or failure in smoking cessation. When assuming success, the pooled RR was 0.95 (95% CI 0.76 to 1.18, p=0.63) with ENDS relative to ENNDS; when assuming failure, the pooled RR was 2.27 (95% CI 1.04 to 4.95, p=0.04). This dramatic variation in the results when making different assumptions is clearly an indicator that the missingness of data is associated with a risk of bias, and that decreases our confidence in the results. We have already reflected that judgment in our risk of bias assessment of these two studies, in table 4 and figure 2; and in our assessment of the quality of evidence in table 7.

Even if we were going to consider the RR of 2.27 as the best effect estimate (i.e., assuming all those with missing data had failure with smoking cessation), the findings would not be supporting the effectiveness of e-cigarettes on smoking cessation. Indeed, the included trials do not address that question, and our review found no study comparing e-cigarettes to no e-cigarettes. The included trials compare two forms of e-cigarettes.

When assessing an intervention A (e.g., e-cigarettes) that has two types A1 (e.g., ENDS) and A2 (e.g., ENNDS), it would be important to first compare A (A1 and/or A2) to the standard intervention (e.g., no intervention or nicotine replacement therapy (NRT)), before comparing A1 to A2. If A1 and A2 are inferior to the standard intervention with A1 being less inferior than A2 (but still inferior to the standard intervention), focusing on the comparison of A1 to A2 (and ignoring the comparison to the standard intervention) will show that A1 is better than A2. That could also falsely suggest that at least A1 (and maybe A2) is favorable. Therefore, a recommendation of A1 vs. A2 should be considered only if A is already recommended over the standard intervention (i.e. A is non inferior to the standard intervention).

Dr. Hajek also criticizes the inclusion of studies that recruited smokers who used e-cigarettes in the past but continue to smoke. When discussing treatment and examining evidence, we refer to effectiveness (known as pragmatic; a treatment that works under real-world conditions). This includes (among other criteria) the inclusion all participants who have the condition of interest, regardless of their anticipated risk, responsiveness, comorbidities or past compliance. Therefore, the inclusion of studies that recruited smokers who used e-cigarettes in the past but continue to smoke had a role in portraying the impact of ENDS and ENNDS in cigarette smokers on long-term tobacco use.

On 2017 May 03, Peter Hajek commented:

Meta-analyses do not normally use different approaches than a widely accepted standard and this field has one (see over a dozen Cochrane meta-analyses, Russell Standard, and any other norm in this field). As far as I know, no meta-analysis or individual study over the past 20 years or so included completers only. As explained earlier, the key point is that ‘missingness’ in this field is not considered random. If among 100 smokers who had treatment only 10 answer follow-up calls and report abstinence, the success rate is considered to be 10% rather than 100% as you would report it.

Re: studies that exclude treatment successes, imagine a treatment with good efficacy that helps 50% of patients, but only the 50% that were not helped are followed-up. These treatment failures may have worse outcomes than a random comparator group (they could have been treatment resistant e.g. because they have a more severe condition or other adverse circumstances). Your approach would interpret the finding as showing that the treatment is not only ineffective, but that it causes harm – when in fact it shows no such thing and is simply an artefact of the selection bias.

I appreciate that getting these things right can be difficult and would leave this alone if it was not such an important topic open to ideological misuse. And I agree that more studies are needed for the definitive answers to emerge.