On 2016 May 02, Clive Bates commented:

It is worth drawing out the most telling criticism made by Hajek, McRobbie and Bullen (above) in their response in the Lancet Respiratory Medicine (above).

There are other problems—such as selective inclusion of studies, and selective reporting of data from studies that were included—and limitations the authors acknowledge in the text but ignore in their conclusions. Detailed criticism of the methods is, however, not needed, because lumping incongruous studies together—which were mostly not designed to evaluate the efficacy of e-cigarettes, and contain no useful information on this topic unless misinterpreted—makes no scientific sense in the first place.

The fundamental problem really is of lumping together completely different studies designed to observe different behaviours in different populations with different outcome measures (mostly not to see how well e-cigarettes help smokers quit). This problem is fatal, and has not been addressed convincingly anywhere, and cannot be, by the authors. Meta-anlaysis is fine for pooling, for example, several drug trials conducted in more with almost identical methodology, but not as it used here.

Once all the studies that should not be 'lumped together' are not lumped together, we come back to something more like the Cochrane review: Can electronic cigarettes help people stop smoking or reduce the amount they smoke, and are they safe to use for this purpose?, which is tentatively positive, but weak given the small numbers of relevant studies.

On 2016 May 01, Peter Hajek commented:

There are several serious problems with this review, see http://www.thelancet.com/pdfs/journals/lanres/PIIS2213-2600(16)30024-8.pdf

The most obvious issue is that the result is based on studies that have no bearing on whether e-cigarettes are effective or not. This is because vapers who successfully quit smoking were excluded and only those who failed to do so were retained. The studies were not at fault, they were just not set up to evaluate quit rates in smokers who try and not try vaping. The fault is with misinterpreting their results. The letter in LRM referenced above provides more details.

On 2016 Apr 25, Stanton A Glantz commented:

A detailed response to the "expert" criticism is available here: http://tobacco.ucsf.edu/our-new-meta-analysis-entire-relevant-literature-shows-e-cigarettes-used-are-associated-less-not-more-quit#comment-17171

On 2016 Apr 24, Clive Bates commented:

This meta-analysis has been subject to considerable criticism from the moment of its publication, see Expert reaction to meta-analysis looking at e-cigarette use and smoking cessation via the Science Media Centre.

For example, Professor Robert West, Professor of Health Psychology at University College London, commented: “Publication of this study represents a major failure of the peer review system in this journal.”

A pre-publication version of this meta-analysis was severely criticised in evidence to the U.S. Food and Drug Administration by experts at the Truth Initiative, which describes itself as "America’s largest non-profit public health organization dedicated to making tobacco use a thing of the past". In the Truth Initiative submission to FDA the examination of the methodological issues begins on page 8 and the following comment appears on page 12, referring to the analysis subsequently published in Lancet Respiratory Medicine.

"While the majority of the studies we reviewed are marred by poor measurement of exposures and unmeasured confounders, many of them have been included in a meta-analysis that claims to show that smokers who use e-cigarettes are less likely to quit smoking compared to those who do not. [73] This meta-analysis simply lumps together the errors of inference from these correlations. As described in detail above, quantitatively synthesizing heterogeneous studies is scientifically inappropriate and the findings of such meta-analyses are therefore invalid."

I hope that in due course The Lancet Respiratory Medicine will publish a critique and reconsider its decision to publish this paper. In the meantime, my own critique is available on my blog.

I would also like to draw readers' attention to a thoughtful discussion of the failings of this meta-analysis by Carl V. Phillips on his blog

On 2016 Apr 24, Clive Bates commented:

This meta-analysis has been subject to considerable criticism from the moment of its publication, see Expert reaction to meta-analysis looking at e-cigarette use and smoking cessation via the Science Media Centre.

For example, Professor Robert West, Professor of Health Psychology at University College London, commented: “Publication of this study represents a major failure of the peer review system in this journal.”

A pre-publication version of this meta-analysis was severely criticised in evidence to the U.S. Food and Drug Administration by experts at the Truth Initiative, which describes itself as "America’s largest non-profit public health organization dedicated to making tobacco use a thing of the past". In the Truth Initiative submission to FDA the examination of the methodological issues begins on page 8 and the following comment appears on page 12, referring to the analysis subsequently published in Lancet Respiratory Medicine.

"While the majority of the studies we reviewed are marred by poor measurement of exposures and unmeasured confounders, many of them have been included in a meta-analysis that claims to show that smokers who use e-cigarettes are less likely to quit smoking compared to those who do not. [73] This meta-analysis simply lumps together the errors of inference from these correlations. As described in detail above, quantitatively synthesizing heterogeneous studies is scientifically inappropriate and the findings of such meta-analyses are therefore invalid."

I hope that in due course The Lancet Respiratory Medicine will publish a critique and reconsider its decision to publish this paper. In the meantime, my own critique is available on my blog.

I would also like to draw readers' attention to a thoughtful discussion of the failings of this meta-analysis by Carl V. Phillips on his blog