On 2015 May 29, Michal Kicinski commented:

I thank Dr. Hilda Bastian for her interest in our recent study (Kicinski M, 2015). I strongly believe that post-publication comments very often raise important issues and help the readers to better understand the merits of a study and its limitations. However, I was disappointed to see that the comments of dr. Hilda Bastian do not correspond with the content of our study. For this reason, I feel obliged to clarify a number of issues.

The study of Ioannidis JP, 2007 points out one of the limitations of a large part of publication bias methods based on the asymmetry of the funnel plot, namely that they do not take between-study heterogeneity into account. This is indeed an important limitation of these methods, as also discussed by other researchers (Song F, 2010). However, please note that we did not rely on the asymmetry of the funnel plot in our analysis. Additionally, please note that our model is just an extension of the standard random effects meta-analysis model, which is a valid approach when between-study variability is present. In fact, the study of Ioannidis JP, 2007 is one of the contributions that motivates our approach to model publication bias since our model takes heterogeneity into account.

Dr. Hilda Bastian correctly points out that our study is not the first study on publication bias. There are many valuable studies on this topic and we discussed those most relevant to our research questions in our article. The contribution of our study is that we analyzed a very large number of meta-analyses using a model with strong theoretical foundations. Our study is the largest study on publication bias in meta-analyses to date. Please note that previous studies, e.g., Ioannidis JP, 2007, which Dr. Hilda Bastian mentioned, considered small study effects, a phenomenon that may have many different causes, including publication bias (Song F, 2010, Sterne JA, 2011). Another merit of our study is that we estimated the association between the size of publication bias and the publication year of the studies included in the meta-analyses.

I completely agree that the best solution to the problem of publication bias is the complete reporting of study results. In fact, our findings showing that publication bias is smaller in the meta-analyses of more recent studies support the effectiveness of the measures used to reduce publication bias in clinical trials. I strongly advocate the introduction of new policies aimed to completely eliminate reporting biases from clinical trials and, as written in our article, the implementation of measures to reduce publication bias in research domains other than clinical trials, such as observational studies and preclinical research.

Although we did not investigate the use of publication bias methods in the meta-analyses from the Cochrane Library, it is clear from previous research that the potential presence of publication bias is often ignored by researchers performing meta-analyses and that the methods accounting for publication bias based on the statistical significance are hardly ever used (Song F, 2010, Onishi A, 2014). When publication bias is present in a meta-analysis, ignoring the problem leads to biased estimates of the effect size (Normand SL, 1999). Therefore, similar to others (Sterne JA, 2011), we argue that researchers should investigate the presence of publication bias and perform sensitivity analyses taking publication bias into account. One difficulty with the use of publication bias methods is that they require researchers to make certain assumptions about the nature of publication bias. For example, the trim and fill method defines publication bias as suppression of a certain number of most extreme negative studies (Duval S, 2000). The use of the Egger’s test (Egger M, 1997) as a publication bias detection tool requires researchers to make the assumption that publication bias leads to a negative association between effect size and precision. The performance of a certain publication bias method depends on whether or not the method’s assumptions are met. For example, it has been demonstrated that publication bias detection tests based on the funnel are characterized by a very low power when publication bias based on the statistical significance is present and the mean effect size equals zero (Kicinski M, 2014). Publication bias based on the statistical significance is the best-documented form of publication bias (Song F, 2009, Dwan K, 2013), The results of our study add to this body of evidence. Therefore, we argue that publication bias tools designed to handle publication bias based on the statistical significance should be used by researchers.

In the tweet with the link to her comment on PubMed, Dr. Hilda Bastian wrote on the 25th of May: ‘27% of cochranecollab reviews over-estimate effects cos of publication bias? Hmm.’ Please note that our study did not investigate the proportion of meta-analyses that overestimate effects. In fact, the objectives of our study were completely different. We estimated the ratio of the probability of including statistically significant outcomes favoring treatment to the probability of including other outcomes in the meta-analyses of efficacy and the ratio of the probability of including results showing no evidence of adverse effects to the probability of including results demonstrating the presence of adverse effects in the meta-analyses of safety.

On 2015 May 25, Hilda Bastian commented:

This is an interesting study. But it's a rather enthusiastic self-assessment of a method not validated by other researchers, and some perspective is useful in thinking about the conclusions.

Kicinski M, 2015 is neither the first, nor the largest study, of publication bias (PB) in meta-analyses, and the presence of publication bias in them is well-known. These authors used a scraper they have made available on Github to extract meta-analyses from Cochrane reviews. They looked at reviews with placebo or "no treatment" control groups and 10 or more included studies. Whether or not these results are applicable to interventions with active or usual care control groups is unknown.

For perspective here: Ioannidis JP, 2007 considered PB in 1,669 Cochrane reviews, ultimately analyzing 6,873 meta-analyses. A half of the meta-analyses had no statistically significant results in them, so the problem identified here could not have applied to them. Ioannidis JP, 2007 concluded that only 5% of the full set of Cochrane reviews would qualify for the use of asymmetry tests, and only 12% of those with a larger number of events and participants. They found very little concordance between different asymmetry tests - only around 3-4%. A more important problem according to Ioannidis JP, 2007 was the misapplication and misinterpretation of statistical tests, not under use. False-positives are a problem with tests for PB when there is clinical heterogeneity. Ioannidis JP, 2007 conclude that the only viable solution to the problem of PB is full reporting of results.

Kicinski M, 2015 conclude that statistical tools for PB are under-utilized, but the extent to which PB is assessed was not part of their study. Although PB itself may be decreasing over time, assessment of PB is increasing, even if the methods for exploring it are still problematic:

• Palma S, 2005 found that PB was assessed in 11% of trials between 1990 and 2002, increasing from 3% in 1998 to 19% in 2002 (less frequently in Cochrane reviews than others).
• Moher D, 2007 found that about 23% of systematic reviews in 2004 assessed PB (32% in Cochrane reviews, 18% in others).
• Riley RD, 2011 found that only 9% of reviews from one Cochrane group assessed PB.
• van Enst WA, 2014 found that most systematic reviews of diagnostic test accuracy in 2011/2012 mentioned the issue, with 41% measuring PB.

In assessing only the meta-analyses themselves, and not the reviews that included them, it's not possible to know, as the authors point out, to what extent other studies were included, but without data that could be pooled. An issue not raised by Kicinski M, 2015 are trials reported only in conference abstracts, and thus with minimal data. Cochrane reviews often include studies reported in conference abstracts only, and those are apparently more likely to have non-statistically significant results (Scherer RW, 2007) - as well as relatively little data for the multiple meta-analyses in a review.

It's important to consider the review, and not just the effect summaries within meta-analyses, because the conclusions of the systematic review should reflect the body of the evidence, not only the meta-analyses. Over-favorable results in a meta-analysis shouldn't be equated with over-favorable conclusions about effectiveness in a review (although unfortunately it often will). We shouldn't jump to conclusions about effect sizes from meta-analyses alone. They can be skewed by clinical heterogeneity and small study size as well as (or instead of) publication bias, and the devil may be more in the interpretation than the calculations.

Disclosure: I work on projects related to systematic reviews at the NCBI (National Center for Biotechnology Information, U.S. National Library of Medicine).

On 2015 May 25, Hilda Bastian commented:

This is an interesting study. But it's a rather enthusiastic self-assessment of a method not validated by other researchers, and some perspective is useful in thinking about the conclusions.

Kicinski M, 2015 is neither the first, nor the largest study, of publication bias (PB) in meta-analyses, and the presence of publication bias in them is well-known. These authors used a scraper they have made available on Github to extract meta-analyses from Cochrane reviews. They looked at reviews with placebo or "no treatment" control groups and 10 or more included studies. Whether or not these results are applicable to interventions with active or usual care control groups is unknown.

For perspective here: Ioannidis JP, 2007 considered PB in 1,669 Cochrane reviews, ultimately analyzing 6,873 meta-analyses. A half of the meta-analyses had no statistically significant results in them, so the problem identified here could not have applied to them. Ioannidis JP, 2007 concluded that only 5% of the full set of Cochrane reviews would qualify for the use of asymmetry tests, and only 12% of those with a larger number of events and participants. They found very little concordance between different asymmetry tests - only around 3-4%. A more important problem according to Ioannidis JP, 2007 was the misapplication and misinterpretation of statistical tests, not under use. False-positives are a problem with tests for PB when there is clinical heterogeneity. Ioannidis JP, 2007 conclude that the only viable solution to the problem of PB is full reporting of results.

Kicinski M, 2015 conclude that statistical tools for PB are under-utilized, but the extent to which PB is assessed was not part of their study. Although PB itself may be decreasing over time, assessment of PB is increasing, even if the methods for exploring it are still problematic:

• Palma S, 2005 found that PB was assessed in 11% of trials between 1990 and 2002, increasing from 3% in 1998 to 19% in 2002 (less frequently in Cochrane reviews than others).
• Moher D, 2007 found that about 23% of systematic reviews in 2004 assessed PB (32% in Cochrane reviews, 18% in others).
• Riley RD, 2011 found that only 9% of reviews from one Cochrane group assessed PB.
• van Enst WA, 2014 found that most systematic reviews of diagnostic test accuracy in 2011/2012 mentioned the issue, with 41% measuring PB.

In assessing only the meta-analyses themselves, and not the reviews that included them, it's not possible to know, as the authors point out, to what extent other studies were included, but without data that could be pooled. An issue not raised by Kicinski M, 2015 are trials reported only in conference abstracts, and thus with minimal data. Cochrane reviews often include studies reported in conference abstracts only, and those are apparently more likely to have non-statistically significant results (Scherer RW, 2007) - as well as relatively little data for the multiple meta-analyses in a review.

It's important to consider the review, and not just the effect summaries within meta-analyses, because the conclusions of the systematic review should reflect the body of the evidence, not only the meta-analyses. Over-favorable results in a meta-analysis shouldn't be equated with over-favorable conclusions about effectiveness in a review (although unfortunately it often will). We shouldn't jump to conclusions about effect sizes from meta-analyses alone. They can be skewed by clinical heterogeneity and small study size as well as (or instead of) publication bias, and the devil may be more in the interpretation than the calculations.

Disclosure: I work on projects related to systematic reviews at the NCBI (National Center for Biotechnology Information, U.S. National Library of Medicine).