On 2016 Jan 22, Stefan Hofmann commented:

Stefan G. Hofmann, Nora Esser, and Giovanbattista Andreoli:

The study by Leichsenring and colleagues highlights the importance of considering the quality of the studies that are included in a meta-analysis when evaluating the results. The Cochrane Collaboration’s Tool (Higgins et al., 2011) is a commonly-used instrument to quantify the risk of bias using the following criteria: allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting. We analyzed the 64 randomized controlled trials of manual-guided PDT for specific mental disorders that were used in the review by Leichsenring et al (see Table 1). Thirty studies showed risk biases in sequence generation, 54 in allocation concealment, and 31 in the blinding conditions. Only one of the studies showed no obvious biases. Our results suggest that the studies included in Leichsenring’s meta-analysis were of poor quality, essentially invalidating the authors’ results and making the findings meaningless. Table 1: http://issuu.com/gvand/docs/quality_ratings_of_studies_in_leich/1 Table 2: http://issuu.com/gvand/docs/description_and_results_of_studies/1 References: Higgins, J.P., Altman, D.G., Gøtzsche, P.C., Jüni, P., Moher, D., Oxman, A.D., Savovic, J., Schulz, K.F., Weeks, L., Sterne, A.C., Cochrane Bias Methods Group, Cochrane Statistical Methods Group (2011). The Cochrane Collaboration´s tool for assessing risk of bias in randomised trials. RESEARCH METHODS & REPORTING, 343.)

On 2016 Apr 22, Falk Leichsenring commented:

For a response to Hofmann, see http://ebmh.bmj.com/cgi/content/full/eb-2016-102372?ijkey=giCBgiE7JTlTg&keytype=ref&siteid=bmjjournals

On 2016 Mar 25, Stefan Hofmann commented:

For a more detailed critique and response, see: http://www.ncbi.nlm.nih.gov/pubmed/27009055

On 2016 Feb 27, Falk Leichsenring commented:

We absolutely agree: strong conclusions require strong evidence

Falk Leichsenring, Patrick Luyten, Mark J. Hilsenroth, Allan Abbass, Jacques P. Barber, John R. Keefe, Frank Leweke, Sven Rabung, Christiane Steinert

Hofmann [1] asked us to provide evidence - here it is.

In our first response [2] to Hofmann et al. [3], we showed that their commentary on the quality of studies of PDT is not consistent with empirical evidence from quality research performed in adversarial collaboration between PDT and CBT researchers, which found no significant differences in quality between studies of PDT and CBT [4, 5]. Furthermore, we noted that Gerber et al. did not find significant correlations between methodological quality and outcome in studies of PDT [5]. These results (i.e., evidence) are inconsistent with Hofmann et al.’s conclusions [3], regardless of whether Hofmann et al.’s ratings [3] are methodologically sound.

In addition, we emphasized that Hofmann et al. failed to demonstrate any evidence that the quality of the 64 RCTs leads to results in favor of PDT. In a similar way Bhar and Beck suggested that the lack of difference in outcome between CBT and PDT found by Leichsenring, Rabung, and Leibing [6] was due to poor treatment integrity [7]. However, using Bhar and Beck’s own integrity ratings, their assertion was not corroborated by empirical data [8]. It is of note that these meta-analyses [6, 8] included researchers from both CBT (E.L.) and PDT (e.g. F.L.).

In our first response, we also observed that the authors failed to provide basic data on interrater reliability, raters’ training, the rating procedures, attempts to address allegiance effects, or blinding of raters [2]. The authors also did not include researchers of both approaches among the raters, as done by Gerber et al. and Thoma et al. [4, 5]. In addition, we noted that Hofmann et al. based their conclusions of poor methodological quality on "unclear" designations of quality [2]. Most authors would have attempted to contact the original authors of a study before asserting that procedural information was unclear and making strong conclusions about study quality.

Hofmann et al. [3] drew strong conclusions about the quality of our review using extreme terms such as "invalidating the authors´ results" and "making the findings meaningless" using nonstandard procedures of questionable quality. For strong conclusions, strong evidence is required. Yet, Hofmann et al. failed to provide it. For a commentary aiming to address study quality, it is puzzling to apply procedures of such poor quality.

We are raising these issues again since Dr. Hofmann did not address them in his response [1]. Instead of doing so, Dr. Hofmann stated [1]: ”As their only defense, the authors argue that CBT is also poorly supported." In this way, he is simply ignoring the evidence we provided and the methodological shortcomings of his commentary we had pointed out [2]. Further, we did not question that CBT is an efficacious treatment. We just pointed out that the available evidence shows that the quality of CBT studies is no better than that of PDT studies [4, 5].

We also did not intend to attack Dr. Hofmann on a personal level, but rather intended to provide evidence that he repeatedly applied double standards when judging studies of CBT as compared to those of PDT [2, 9, p. 49-51]. We respectfully asked that if he chooses to write about PDT (e.g., comment on a meta-analysis, conduct a meta-analysis, or conduct a study involving PDT), that he considers involving a psychodynamic researcher in the process. This invitation still stands.

Dr. Hofmann emphasized that CBT is widely disseminated in the UK. This is true, but PDT is recommended by treatment guidelines and implemented in the National Health Service in the UK as well. This is also true in other countries. In Germany, for instance, PDT is as frequently used as CBT [10]. The Scientific Board for Psychotherapy (Wissenschaftlicher Beirat Psychotherapie; WBP) is the paramount body in Germany for assessing the scientific status of psychotherapeutic interventions. For this purpose, standardized and transparent criteria are used. Based on a careful evaluation by the WBP, both CBT and PDT were acknowledged as scientific and efficacious forms of psychotherapy (www.wbpsychotherapie.de). It is noteworthy that the WBP is composed of researchers from diverse psychotherapeutic orientations (e.g. CBT, PDT, and systemic therapy). The studies of PDT were evaluated by CBT researchers, and vice versa. The conclusions by a balanced expert institution such as the WBP are incompatible with those by Hofmann et al. [3].

We all should be happy that a variety of psychotherapeutic treatments exist that are beneficial to patients. Future research should address the question of which patients benefit most from which treatments, and why.

Declaring the evidence of a whole treatment approach as "meaningless" is not supported by the preponderance of evidence, and is counter-productive to this goal.

References

1 Hofmann SG: Show us the data! Pubmed Commons Feb 17 2016 2 Leichsenring F, Luyten P, Hilsenroth MJ, Abbass A, Barber JP, Keefe JR, Leweke F, Rabung S, Steinert C: Once again: Double standards in psychotherapy research - response to hofmann et al. PubMed Commons 2016 3 Hofmann SG, Eser N, Andreoli G: Comment from pubmed commons. . January 23rd, 12:28am UTC 2016 2016 4 Thoma NC, McKay D, Gerber AJ, Milrod BL, Edwards AR, Kocsis JH: A quality-based review of randomized controlled trials of cognitive-behavioral therapy for depression: An assessment and metaregression. Am J Psychiatry 2012;169 5 Gerber AJ, Kocsis JH, Milrod BL, Roose SP, Barber JP, Thase ME, Perkins P, Leon AC: A quality-based review of randomized controlled trials of psychodynamic psychotherapy. Am J Psychiatry 2011;168:19-28. 6 Leichsenring F, Rabung S, Leibing E: The efficacy of short-term psychodynamic psychotherapy in specific psychiatric disorders: A meta-analysis. Arch Gen Psychiatry 2004;61:1208-1216. 7 Bhar SS, Beck AT: Treatment integrity of studies that compare short-term psychodynamic psychotherapy with cognitive-behavior therapy Clin Psychol-Sci Pr 2009;16:370-378. 8 Leichsenring F, Salzer S, Hilsenroth M, Leibing E, Leweke F, Rabung S: Treatment integrity: An unresolved issue in psychotherapy research. Curr Psych Rev 2011;7 313-321. 9 Leichsenring F, Rabung S: Double standards in psychotherapy research. Psychother Psychosom 2011;80:48-51. 10 Albani C, Blaser G, Geyer M, Schmutzer G, Brähler E: Outpatient psychotherapy in germany from the patient perspective: Part 1: Health care situation Psychotherapeut 2010;55:503-514.

On 2016 Feb 17, Stefan Hofmann commented:

Show us the data! The authors wrote in the abstract that psychodynamic therapy is as efficacious as treatments established in efficacy. Strong conclusions require strong evidence. Dr. Leichsenring and colleagues were unable to provide the reader with such evidence. Our earlier commentary of the article by Leichsenring and colleagues reported data suggesting that the majority of studies included in their review were of low quality. It is the responsibility of Leichsenring and colleagues to provide the reader with evidence that their conclusions are justified. Instead of providing the reader with such evidence, the authors chose to attack me on a personal level. Whether or not Dr. Leichsenring and colleagues believe that I am adequate to serve as a reviewer or editor of scientific journals or grants and collaborate with others is completely unrelated to the weaknesses of their study. As their only defense, the authors argue that CBT is also poorly supported. The authors are incorrect. The supporting evidence of CBT is overwhelmingly large. Our own review identified 269 meta-analytic studies of CBT. We observed that the quality of studies that entered some of these meta-analyses were not uniformly high. However, some of them were of high quality (e.g., Hofmann & Smits, 2008). Because CBT has such a solid empirical basis, many countries, including the UK, disseminate CBT on a large-scale basis, e.g., http://www.iapt.nhs.uk/. It should be noted that this dissemination is not limited to CBT but also includes other empirically supported treatments. Polemics and personal attacks on my scientific integrity are not the real problem. The biggest concern in my view is that these disputes distract from the real issue. They confuse our patients and policy makers, inhibit scientific progress, and inflict harm by withholding effective treatments. References 1. Hofmann, S. G., Asnaani, A., Vonk, J. J., Sawyer, A. T., & Fang, A. (2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research, 36, 427-440. doi 10.1007/s10608-012-9476-1 2. Hofmann, S. G. & Smits, J. A. J. (2008). Cognitive-behavioral therapy for adult anxiety disorders: A meta-analysis of randomized placebo-controlled trials. Journal of Clinical Psychiatry, 69, 621-632. doi: 10.4088/JCP.v69n0415

On 2016 Jan 29, Falk Leichsenring commented:

Once again: Double standards in psychotherapy research - response to Hofmann et al.

Falk Leichsenring, Patrick Luyten, Mark J. Hilsenroth, Allan Abbass, Jacques P. Barber, John R. Keefe, Frank Leweke, Sven Rabung, Christiane Steinert

Referring to a recent review on psychodynamic therapy (PDT) [1], Hofmann et al. [2] criticize the quality of studies included in this review. The authors conclude that the "poor quality" of studies of PDT "invalidates" the results of this review making them "meaningless" [2]. The comment by Hofmann et al. [2] deserves some response.

• The conclusions drawn by Hofmann et al. [2] are inconsistent with present research. As shown by an independent research group including proponents of both CBT and PDT working respectfully together, the quality of studies of PDT does not differ significantly from that of studies of CBT which fell into the lower range of adequate quality [3, p. 22, 4]. Most of the studies included by Leichsenring et al. [1] were also included in this comparison [3, 4]. However, Hofmann has not described the respective CBT studies as “meaningless”.

Furthermore, the comment by Hofmann et al. [2] suffers from several shortcomings.

• The authors are incorrect when referring to our publication as a meta-analysis. In fact it was a systematic review [1]. This is of note since possible shortcomings in individual studies would not invalidate the review as a whole.

• The authors draw a highly generalizing conclusion without any differentiation, for example, by disorders or degree of risk.

• For their ratings, Hofmann et al. [2] did not report basic data on the number and training of raters, on blinding, or interrater-reliability. For the best minimization of bias, raters of both approaches would have been included as done by Gerber et al. and Thoma et al. [3, 4]. Thus, the quality of the procedures applied by Hofmann et al. [2] themselves is questionable.

• The conclusions by Hofmann et al. [2] are based mostly on “unclear” designations, not clear flaws. In fact, an “unclear” risk of bias indicates that the design feature could be both worse or better than described in the article. What is most concerning is that the authors did not make any effort to resolve the “unclear” assignments by carefully reading the papers or contacting their authors. Many assignments are obviously clear from the studies [e.g. 5, 6].

• In addition, even if there are flaws, Hofmann has not shown that these particular flaws lead to results in favor of PDT (rather than e.g., greater error in effect estimates overall). Several meta-analyses did neither find significant correlations between ratings of methodological quality and outcome [4, 7] nor between treatment integrity (assessed by prominent CBT researchers, e.g. Aaron T. Beck) and differences in outcome between CBT and PDT [8].

• If the "poor quality" of PDT studies "invalidates" [2] the results reported by Leichsenring et al. [1] making them "meaningless", this would equally apply to meta-analyses carried out by CBT researchers who included several of these same studies. Tolin [9], for example, included 10 studies also included by Leichsenring et al. [1]. Hofmann, however, has never critically commented on these meta-analyses - which are interpreted as supporting the efficacy of CBT - thus, again applying a double standard.

• Hofmann was repeatedly shown to apply double standards when judging studies of CBT vs. PDT [10, p. 49-51].

(a) In a previous meta-analysis, for example, Hofmann [11, p. 180] claimed that the quality of studies included by him was "considerably better" than that of studies of PDT, e.g. in the meta-analysis by Leichsenring and Rabung [12]. Hofmann et al. [11] reported a mean Jadad score of 1.23, whereas the mean Jadad score in the meta-analysis by Leichsenring et al.[12] was 1.96 (0 = poor; 5 = excellent).

(b) Hofmann [11]criticized the meta-analysis by Leichsenring et al. [12] for including heterogeneous studies. However, between-effect hterogeneity was in the low to medium range [10, 12]. In his own meta-analysis, Hofmann [11] did not even test for heterogeneity before combining data of randomized controlled and observational studies [10, 11].

• Thus, from a scientific perspective, it is questionable whether a strong proponent of CBT who has publicly demonstrated that he is an opponent of PDT [e.g. 13] is able to provide unbiased conclusions about PDT.

Given the author’s very negative publicly expressed opinions about PDT, and the way he conducted this critique of the Leichsenring et al. review [1] it appears that biases can lead to a lack of even-handedness in regard to the evaluation of psychodynamic studies. Thus, we respectfully would ask that if he chooses to write about PDT(e.g., comment on a meta-analysis, conduct a meta-analysis, or conduct a study involving PDT), that he involve a psychodynamic researcher in the process (i.e., implement a version of adversarial collaboration [14], and also that he recuse himself from being involved as an editor, or reviewer, in regard to research involving PDT.

We would welcome the collaboration of CBT researchers in researching psychotherapy and synthesizing the results from trials. We have done so several times [e.g. 15, 16].

Given the present crisis of replicability of research [17], biased and tendentious statements as those by Hofmann bear the risk of damaging all psychotherapy research equally in the eyes of the public.

References

1 Leichsenring F et al.: Psychodynamic therapy meets evidence-based medicine: a systematic review using updated criteria. Lancet Psychiatry 2015;2:648-660. 2 Hofmann SG et al.: Comment from PubMed Commons. January 23rd, 12:28am UTC 2016 3 Thoma NC et al.: A quality-based review of randomized controlled trials of cognitive-behavioral therapy for depression: an assessment and metaregression. Am J Psychiatry 2012;169 4 Gerber AJ et al.: A quality-based review of randomized controlled trials of psychodynamic psychotherapy. Am J Psychiatry 2011;168:19-28. 5 Barber J et al.: Short-term dynamic psychotherapy versus pharmacotherapy for major depressive disorder: a randomized, placebo-controlled trial. J Clin Psychiatry 2012;73:66-73. 6 Crits-Christoph P et al.: Psychosocial treatments for cocaine dependence: National Institute on Drug Abuse Collaborative Cocaine Treatment Study. Arch Gen Psychiatry 1999;56:493-502. 7 Keefe JR et al.: A meta-analytic review of psychodynamic therapies for anxiety disorders. Clin Psychol Rev 2014;34:309-323. 8 Leichsenring F et al.: Treatment integrity: An unresolved issue in psychotherapy research. Curr Psych Rev 2011;7 313-321. 9 Tolin DF: Is cognitive-behavioral therapy more effective than other therapies? A meta-analytic review. Clin Psychol Rev 2010;30:710-720. 10 Leichsenring F et al.: Double standards in psychotherapy research. Psychother Psychosom 2011;80:48-51. 11 Hofmann SG et al.: The effect of mindfulness-based therapy on anxiety and depression: a meta-analytic review J Consult Clin Psychol 2010; 78 169-183. 12 Leichsenring F et al.: Effectiveness of long-term psychodynamic psychotherapy: a meta-analysis. JAMA 2008;300:1551-1565. 13 Rief W et al.: [Saving psychoanalysis. At any cost?]. Nervenarzt 2009;80:593-597. 14 Mellers B. et al.: Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychol Sc 2001;12:269-275. 15 Leichsenring F et al.: The efficacy of short-term psychodynamic psychotherapy in specific psychiatric disorders: a meta-analysis. Arch Gen Psychiatry 2004;61:1208-1216. 16 Leichsenring F et al.: Psychodynamic therapy and CBT in social anxiety disorder: a multicenter randomized controlled trial. Am J Psychiatry 2013;170:759-767. 17 Open Science Collaboration: PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 2015;349:aac4716.

On 2016 Jan 22, Stefan Hofmann commented:

Stefan G. Hofmann, Nora Esser, and Giovanbattista Andreoli:

The study by Leichsenring and colleagues highlights the importance of considering the quality of the studies that are included in a meta-analysis when evaluating the results. The Cochrane Collaboration’s Tool (Higgins et al., 2011) is a commonly-used instrument to quantify the risk of bias using the following criteria: allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting. We analyzed the 64 randomized controlled trials of manual-guided PDT for specific mental disorders that were used in the review by Leichsenring et al (see Table 1). Thirty studies showed risk biases in sequence generation, 54 in allocation concealment, and 31 in the blinding conditions. Only one of the studies showed no obvious biases. Our results suggest that the studies included in Leichsenring’s meta-analysis were of poor quality, essentially invalidating the authors’ results and making the findings meaningless. Table 1: http://issuu.com/gvand/docs/quality_ratings_of_studies_in_leich/1 Table 2: http://issuu.com/gvand/docs/description_and_results_of_studies/1 References: Higgins, J.P., Altman, D.G., Gøtzsche, P.C., Jüni, P., Moher, D., Oxman, A.D., Savovic, J., Schulz, K.F., Weeks, L., Sterne, A.C., Cochrane Bias Methods Group, Cochrane Statistical Methods Group (2011). The Cochrane Collaboration´s tool for assessing risk of bias in randomised trials. RESEARCH METHODS & REPORTING, 343.)