On 2017 Sep 15, Robin Kok commented:

Regarding bias, this 'review' is an exercise in confirmation bias.

The rationale and execution of this systematic review are confusing at best, and misleading at worst. The authors invoke some metric of 'strength of evidence', which consists of tallying up positive conclusions from meta-analyses. For example, they conclude: "Protocols improve anxiety in adults: 8/9 meta-analyses", which we are to believe is strong evidence for cognitive bias modification (CBM).

However:

a) many of these meta-analyses draw from the same pool of primary studies, resulting in a great deal of overlap. Yet the authors treat these as if they are independent data points - demonstrably, they are not. If one would perform 10 meta-analyses on 40 studies, then it should come as no surprise that these meta-analyses report comparable outcomes. This greatly exaggerates positive effects by counting single studies multiple times. It is disconcerting that the peer reviewers did not spot this serious and elementary flaw. Then again, given that the entire process of submission, reviews and editorial acceptance took just over a month, reviewers and/or editor(s) must have been particularly pleased with this manuscript to favour it with an expedient review process.

b) many of these meta-analyses - and the primary outcomes from which they draw data - fail even the most basic quality criteria for rigorously conducted meta-analyses and randomised controlled trials.

c) many of these meta-analyses and primary studies reported therein were conducted by investigators with an investigator allegiance or other conflicts of interest. Studies performed by independent researchers often fail to find effects.

d) there is evidence for extreme publication bias in the field of CBM, as apparent by both statistical indicators (which the authors mostly ignore), as well as by the great number of trials in official trial registries which are still awaiting publication - some for almost 10 years. The reader can easily find an abundance of file-drawered CBM studies in the WHO, ISRCTN, ANZCTR, and Clinicaltrials.gov trial registries.

In short, this review is strongly biased in favour of cognitive bias modification and serves only to promote CBM as a 'promising therapy'. It adds to an already overwhelming number of positively biased reviews and analyses which essentially draw from the same pool of biased literature, strengthening the echo-chamber effect that seriously diminishes the scientific credibility of the CBM field. Reviews such as these are no more than opinion/promotion pieces in disguise, and should be interpreted with extreme caution as a reader unwary of these issues might easily conclude that there is overwhelming evidence for the efficacy of CBM - which there is not.

If there is any value in CBM, it deserves fair testing - which means pre-registered, transparent replication efforts by disinterested, independent researchers. The disturbing phenomenon where biased CBM trials are recycled in reviews and meta-analyses in quick succession by biased investigators is the academic equivalent of an echo-chamber. And if CBM is to be seen as a serious alternative to psychotherapy, it is to be held to the same standards of methodology and transparency and step away from the cargo cult science we have seen so far.

DECLARATION OF INTEREST: I have co-authored two (mostly critical) meta-analyses included in this review.

On 2017 Sep 15, Robin Kok commented:

Regarding bias, this 'review' is an exercise in confirmation bias.

The rationale and execution of this systematic review are confusing at best, and misleading at worst. The authors invoke some metric of 'strength of evidence', which consists of tallying up positive conclusions from meta-analyses. For example, they conclude: "Protocols improve anxiety in adults: 8/9 meta-analyses", which we are to believe is strong evidence for cognitive bias modification (CBM).

However:

a) many of these meta-analyses draw from the same pool of primary studies, resulting in a great deal of overlap. Yet the authors treat these as if they are independent data points - demonstrably, they are not. If one would perform 10 meta-analyses on 40 studies, then it should come as no surprise that these meta-analyses report comparable outcomes. This greatly exaggerates positive effects by counting single studies multiple times. It is disconcerting that the peer reviewers did not spot this serious and elementary flaw. Then again, given that the entire process of submission, reviews and editorial acceptance took just over a month, reviewers and/or editor(s) must have been particularly pleased with this manuscript to favour it with an expedient review process.

b) many of these meta-analyses - and the primary outcomes from which they draw data - fail even the most basic quality criteria for rigorously conducted meta-analyses and randomised controlled trials.

c) many of these meta-analyses and primary studies reported therein were conducted by investigators with an investigator allegiance or other conflicts of interest. Studies performed by independent researchers often fail to find effects.

d) there is evidence for extreme publication bias in the field of CBM, as apparent by both statistical indicators (which the authors mostly ignore), as well as by the great number of trials in official trial registries which are still awaiting publication - some for almost 10 years. The reader can easily find an abundance of file-drawered CBM studies in the WHO, ISRCTN, ANZCTR, and Clinicaltrials.gov trial registries.

In short, this review is strongly biased in favour of cognitive bias modification and serves only to promote CBM as a 'promising therapy'. It adds to an already overwhelming number of positively biased reviews and analyses which essentially draw from the same pool of biased literature, strengthening the echo-chamber effect that seriously diminishes the scientific credibility of the CBM field. Reviews such as these are no more than opinion/promotion pieces in disguise, and should be interpreted with extreme caution as a reader unwary of these issues might easily conclude that there is overwhelming evidence for the efficacy of CBM - which there is not.

If there is any value in CBM, it deserves fair testing - which means pre-registered, transparent replication efforts by disinterested, independent researchers. The disturbing phenomenon where biased CBM trials are recycled in reviews and meta-analyses in quick succession by biased investigators is the academic equivalent of an echo-chamber. And if CBM is to be seen as a serious alternative to psychotherapy, it is to be held to the same standards of methodology and transparency and step away from the cargo cult science we have seen so far.

DECLARATION OF INTEREST: I have co-authored two (mostly critical) meta-analyses included in this review.