On 2016 May 09, Swapnil Hiremath commented:

We read the meta-analysis review by Subramaniam et. al. with interest. The investigators undertook a herculean effort by including multiple treatments for the prevention of contrast induced-acute kidney injury (CI-AKI) within one report, and should be congratulated for their efforts. However, some of the results are open to being poorly understood or misinterpreted as presented, and we encourage readers to reflect on a few other aspects here.

• Firstly, the focus on the surrogate outcome of change in creatinine is perplexing, when greater emphasis could be placed on the many trials which report clinically important outcomes (need for renal replacement therapy, cardiac events and mortality).

• Secondly, it can be difficult to capture or identify important points of heterogeneity between between studies for various treatments in the context of such a large undertaking. These points of heterogeneity have been characterized in more focused meta-analyses of these CI-AKI prevention strategies. We are concerned that these points of heterogeneity have not been adequately recognized in the present report and may have implications on the authors conclusions. It has been noted that for many CI-AKI prevention trials, if not the majority for some treatments,the sample sizes are quite small with extremely large treatment effects. When these small studies have been followed by larger ones, it has become evident that these early promising results were not readily reproducible. As such, analysis of trials by sample size has shown that the purported treatment benefit is the greatest in smaller studies, and often not statistically significant in larger trials. Unfortunately, the larger studies are often considerably smaller in number, and the meta-analytic framework does not give them adequate weight. It may argued that meta-analyses were not intended for marked imbalances of this sort, and if calculated, the summary estimates may not be accurate. Observing consistency between treatment effects in large and small studies would be reassuring. Unfortunately, there is no such consistency for many CI-AKI prevention treatments. This is particularly notable for N-acetylcysteine (NAC) and sodium bicarbonate. Multiple clinical trials were performed after initial reports of positive clinical trial results of sodium bicarbonate for the prevention of CI-AKI. The studies largely showed no significant benefit compared with sodium bicarbonate fluid administration. Not unexpectedly, this practice has largely been abandoned by most practitioners.

• Similarly, the efficacy of NAC was explored in a large and possibly the best quality trial, which was done with high-dose NAC, included 2308 patients with a hazard ratio of 1.00 for the primary outcome. Even in subgroups at higher risk for CI-AKI, diabetics and estimated GFR less than 60, no significant benefit was observed. The result for the effectiveness of low-dose NAC, which includes older, smaller trials of lower methodological quality (36 trials,4874 participants), as compared to high-dose NAC (usually larger, higher quality trials, 18 trials, 4336 participants), makes little biological sense. The literature on statin therapy is similarly flawed. The largest clinical trial with almost 3000 patients, was about six times larger than the next largest clinical trial. While this study did show a statistically significant benefit for statin treatment versus no statin treatment, the interpretation is complicated given that the majority of patients enrolled were at very low risk for CI-AKI because the majority had CKD stage I or II and were at low risk for CI-AKI. When the subset of approximately 500 patients with estimated GFR & less than 60 were analyzed, no significant benefit for statin therapy was observed.

• Presentation of the results as NAC compared to intravenous (IV) saline (in Table 1) is also misleading, given that this was a comparison of patients who received NAC and IV saline compared to those who received IV saline alone (or with placebo). IV fluid expansion is the most effective measure reported in the literature, and to report NAC as being potentially superior to it may result in preferential use of NAC instead of IV saline, a concern which was also highlighted by Drs Weisbord and Pavelsky in a response to the journal.

• Lastly, in the present report, stratification by contrast type further complicates the interpretation by introducing an additional subgroup in the analyses, and increases the risk for a type II error.

These aspects of the literature can be readily lost when seen through the meta-analytical lens. The aim of a meta-analysis is not solely to generate a summary estimate, but provide a structured framework to explore and understand sources of heterogeneity, both statistical and clinical. Identifying important heterogeneity between studies can at times provide more insight than the summary estimates. Unfortunately, this can be very time-consuming, in particular when multiple treatments are under review, and requires expertise in the treatments being evaluated.

Somjot Brar; Kaiser Permanenete, Los Angeles

Ayub Akbari, Swapnil Hiremath; University of Ottawa

(This is a longer version of the comment posted online on the Annals website for this same study)

On 2016 May 09, Swapnil Hiremath commented:

We read the meta-analysis review by Subramaniam et. al. with interest. The investigators undertook a herculean effort by including multiple treatments for the prevention of contrast induced-acute kidney injury (CI-AKI) within one report, and should be congratulated for their efforts. However, some of the results are open to being poorly understood or misinterpreted as presented, and we encourage readers to reflect on a few other aspects here.

• Firstly, the focus on the surrogate outcome of change in creatinine is perplexing, when greater emphasis could be placed on the many trials which report clinically important outcomes (need for renal replacement therapy, cardiac events and mortality).

• Secondly, it can be difficult to capture or identify important points of heterogeneity between between studies for various treatments in the context of such a large undertaking. These points of heterogeneity have been characterized in more focused meta-analyses of these CI-AKI prevention strategies. We are concerned that these points of heterogeneity have not been adequately recognized in the present report and may have implications on the authors conclusions. It has been noted that for many CI-AKI prevention trials, if not the majority for some treatments,the sample sizes are quite small with extremely large treatment effects. When these small studies have been followed by larger ones, it has become evident that these early promising results were not readily reproducible. As such, analysis of trials by sample size has shown that the purported treatment benefit is the greatest in smaller studies, and often not statistically significant in larger trials. Unfortunately, the larger studies are often considerably smaller in number, and the meta-analytic framework does not give them adequate weight. It may argued that meta-analyses were not intended for marked imbalances of this sort, and if calculated, the summary estimates may not be accurate. Observing consistency between treatment effects in large and small studies would be reassuring. Unfortunately, there is no such consistency for many CI-AKI prevention treatments. This is particularly notable for N-acetylcysteine (NAC) and sodium bicarbonate. Multiple clinical trials were performed after initial reports of positive clinical trial results of sodium bicarbonate for the prevention of CI-AKI. The studies largely showed no significant benefit compared with sodium bicarbonate fluid administration. Not unexpectedly, this practice has largely been abandoned by most practitioners.

• Similarly, the efficacy of NAC was explored in a large and possibly the best quality trial, which was done with high-dose NAC, included 2308 patients with a hazard ratio of 1.00 for the primary outcome. Even in subgroups at higher risk for CI-AKI, diabetics and estimated GFR less than 60, no significant benefit was observed. The result for the effectiveness of low-dose NAC, which includes older, smaller trials of lower methodological quality (36 trials,4874 participants), as compared to high-dose NAC (usually larger, higher quality trials, 18 trials, 4336 participants), makes little biological sense. The literature on statin therapy is similarly flawed. The largest clinical trial with almost 3000 patients, was about six times larger than the next largest clinical trial. While this study did show a statistically significant benefit for statin treatment versus no statin treatment, the interpretation is complicated given that the majority of patients enrolled were at very low risk for CI-AKI because the majority had CKD stage I or II and were at low risk for CI-AKI. When the subset of approximately 500 patients with estimated GFR & less than 60 were analyzed, no significant benefit for statin therapy was observed.

• Presentation of the results as NAC compared to intravenous (IV) saline (in Table 1) is also misleading, given that this was a comparison of patients who received NAC and IV saline compared to those who received IV saline alone (or with placebo). IV fluid expansion is the most effective measure reported in the literature, and to report NAC as being potentially superior to it may result in preferential use of NAC instead of IV saline, a concern which was also highlighted by Drs Weisbord and Pavelsky in a response to the journal.

• Lastly, in the present report, stratification by contrast type further complicates the interpretation by introducing an additional subgroup in the analyses, and increases the risk for a type II error.

These aspects of the literature can be readily lost when seen through the meta-analytical lens. The aim of a meta-analysis is not solely to generate a summary estimate, but provide a structured framework to explore and understand sources of heterogeneity, both statistical and clinical. Identifying important heterogeneity between studies can at times provide more insight than the summary estimates. Unfortunately, this can be very time-consuming, in particular when multiple treatments are under review, and requires expertise in the treatments being evaluated.

Somjot Brar; Kaiser Permanenete, Los Angeles

Ayub Akbari, Swapnil Hiremath; University of Ottawa

(This is a longer version of the comment posted online on the Annals website for this same study)