On 2015 Nov 02, R Andrew Moore commented:

Peter has a number of problems, and it is difficult to cover them all in a short space. He has not reported his concerns through the Cochrane Feedback mechanism.

Our searches did identify a large number of unpublished trials. We deplore this, but are powerless to change things. In a previous review for topical NSAIDs we contacted all identified manufacturers for published or unpublished data. The yield was small (as others have found), but some unpublished studies were brought into the public domain. Waiting for all studies and clinical trial reports of all studies would take forever, and could probably never be achieved. We believe that many of these unpublished studies relate to drugs and/or formulations that have never been manufactured commercially. While these would be of interest in determining what does and doesn’t work, they would have little clinical relevance.

Peter’s main issue appears to be heterogeneity. Tests for heterogeneity are problematical anyway (Pain. 2000 85:415-24), and the I square of 92% was for all topical diclofenac formulations combined. We demonstrated in the review that the (at least five) different diclofenac formulations produced different results from one another using L’Abbé plots and in our detailed analyses showing large variations in efficacy between them. In the circumstance, a high I square for all combined (clinical heterogeneity) is to be expected. The bulk of the studies on diclofenac were published in the last five years, were of decent quality, and moderate to large size. There were older data for ketoprofen, but again major differences between formulations.

Trying to determine publication bias using funnel plots or other measures is something of a lost cause (Journal of Clinical Epidemiology 2000;53:207-16). It is especially so with small number of trials (Journal of Clinical Epidemiology 2000 53: 477-484), and making sense of funnel plots is anything but easy for most people (Journal of Clinical Epidemiology 2005 58: 894-901). A useless method seems an odd choice to make to criticise our review.

There are very good scientific reasons why drug and formulation may play a big part in the efficacy of topical NSAIDs. This is also the case for oral analgesics used in acute pain, where formulation improvements generating rapid absorption confers greater efficacy. Simple lumping strategies may have been permissible in past systematic review methodology, but a more forensic approach is needed now and in the future. This is what we have attempted to do in this latest review.

Comparisons with Peter’s 2010 review seem inappropriate since that review was based on other reviews that are now out of date.

On 2015 Oct 28, Peter Gøtzsche commented:

The authors found that the results were missing from 5900 patients. Furthermore, there was extreme heterogeneity in their meta-analyses, e.g. I square was 92% for the diclofenac trials, which were the most common ones, and there was extreme funnel plot asymmetry, with the largest trials showing the smallest effects (the authors didn’t show funnel plots but I constructed one for diclofenac). Moreover, the trials were industry funded, of relatively poor quality, and the authors analysed published data, not data from clinical study reports, and did not try to obtain all the missing trials and data from the manufacturers.

The authors concluded that topical NSAIDs are effective in providing pain relief but also cautioned that the large amounts of unpublished data “could influence results in updates of this review.” They certainly could. I believe it is plain wrong to perform meta-analyses on the authors’ data. When I most recently reviewed this area for the BMJ in 2010, I concluded that we don't know whether topical NSAIDs are beneficial (1).

1 Gøtzsche PC. NSAIDs. Clin Evid (Online). 2010 Jun 28.

On 2015 Oct 28, Peter Gøtzsche commented:

The authors found that the results were missing from 5900 patients. Furthermore, there was extreme heterogeneity in their meta-analyses, e.g. I square was 92% for the diclofenac trials, which were the most common ones, and there was extreme funnel plot asymmetry, with the largest trials showing the smallest effects (the authors didn’t show funnel plots but I constructed one for diclofenac). Moreover, the trials were industry funded, of relatively poor quality, and the authors analysed published data, not data from clinical study reports, and did not try to obtain all the missing trials and data from the manufacturers.

The authors concluded that topical NSAIDs are effective in providing pain relief but also cautioned that the large amounts of unpublished data “could influence results in updates of this review.” They certainly could. I believe it is plain wrong to perform meta-analyses on the authors’ data. When I most recently reviewed this area for the BMJ in 2010, I concluded that we don't know whether topical NSAIDs are beneficial (1).

1 Gøtzsche PC. NSAIDs. Clin Evid (Online). 2010 Jun 28.