On 2015 Sep 17, Ghassan El-Baalbaki commented:

authors of this comment are: Loose, T.^1,3 , Ashrah, L.^1, Romero, M.^1, Bégin, J.¹ and El-Baalbaki, G.^1,2 Affiliations, Université du Québec À Montréal^1, McGill University^2, Université de Nantes³ .

In this study, the authors examine the efficacy of Acceptance and Commitment Therapy (ACT) in treating partner aggression. After entering into communication with the first author, she confirmed that this study corresponds to her doctoral thesis (Zarling, 2013). We read her thesis in order to gain further knowledge about the published study, and we found two important aspects of the work that need to be brought into light: 1) we find it difficult to conclude that partner aggression was specifically reduced, because of threats to internal validity. 2) There seems to be inaccuracies in their statistical calculations.

The primary objective was to demonstrate that ACT reduces intimate partner aggression better than the placebo control group. The authors strongly suggest that ACT was indeed able to do so. However, several points call this statement into question.

First of all, it is worth noting that partner aggression is never specifically assessed. One may think that the Conflict Tactics Scales-2-Physical Assault Scale (CTS-2) did assess this construct (Straus, Hamby, Boney-McCoy & Sugarman, 1996), but when taking into account the exact content of the administered questionnaire that was figured in her thesis, it can be seen that the instructions were modified in order to include aggression towards “people we care about” (Zarling, 2013, p.132) instead of specifically towards a partner. This modification is not mentioned in the published article. Hence, participants could have hypothetically committed aggressive acts exclusively towards a non-partner, such as a child or a friend. It is thus hard to conclude that aggressive behavior, specifically among couples, decreased as a result of therapy.

Furthermore, after modifying the instructions, one of the items even became incoherent. More specifically, when taking into the modified instructions of the questionnaire, the item stated “When upset or in a disagreement with someone you care about … have you became angry enough to frighten your partner?” (italics added)(Zarling, 2013, p. 132). It seems that this question was destined to ask if the participant became angry enough to frighten someone that they « care about » and not their « partner ». For example, if the participants were (or became) single, they could have interpreted this question as if they were angry enough to frighten someone that is not necessarily their partner. It is unclear how the authors wanted the participants to interpret this item. The way that the construct of partner aggression was assessed calls into question the internal validity of the study.

It remains questionable why the authors neglected to assess attrition of couples during the study. Over the course of the study, it is possible that aggressive behavior decreased more in the ACT group as a result of more couples breaking up in this treatment condition than in the placebo control group. Even if the authors carried out both completer analysis and intent to treat analysis and the results did not differ on measured variables, the attrition of couples was not measured. Hence, it is impossible to know if the groups differed on partner attrition. It remains possible that participants continued to aggress former partners, but again this cannot be asserted simply because it was not measured. Assessing partner attrition would have helped neutralize a pertinent threat to internal validity. Hence, internal validity was limited by not taking into account partner attrition.

Lastly, it is worth bringing up that the first author and two other investigators co-led all administered interventions (p. 201). Even if the authors state this in their article, after looking at the detailed explanation of the recruitment process of investigators, this choice does not seem to be justified. It seems that the first author was able to recruit investigators at no extra cost and it is questionable as to why she did not recruit three instead of two, thus avoiding a potential threat to internal validity. The explanation provided in her thesis, but not the article, also states that all the manipulation checks were carried out by the first author and supervised by the second and third authors (Zarling, 2013, p.61). Because of the implication of the authors in the experimental procedure, the results are potentially attributable to the experimenter expectancy effect. This is another threat to internal validity that evokes the tendency for researchers to unintentionally bias the results of their investigations in accord with their hypotheses. In taking into account the previously stated points, it seems difficult to conclude that this study shows that ACT specifically reduces partner aggression better than a placebo control group.

On another note, the statistical validity of the study can be questioned:

Throughout the results section of the paper (pp. 205–207), the authors report many beta values (β) for intercepts and slopes. If these are indeed beta values, they should vary between approximately 1 and -1 (Cohen, Cohen, West & Aiken, 2002). However in many cases, reported values are outside of this range. For example the authors state “The ACT participants also reported significantly less psychological aggression, β = 2.05, SE = 0.71, t(97) = 8.33, p < .001, and physical aggression, β = 2.21, SE = 0.65, t(97) = 8.19, p < .001, at the 6-month follow-up assessment” (p. 8). This leads us to think that the authors are actually reporting non-standardized coefficients (B) because B values can vary outside of this range. However, if this is the case, then the t values reported become incoherent. t values should equal B/(2SE) (we retained the value of 2 after rounding up from 1.96 which corresponds to 95% of a normal distribution (Christensen, 1986)), but reported t values do not correspond to this calculation. For example, let’s take another look at the previous citation and assume that it actually refers to B values. According to our calculations, t = B/(2SE) = 2.05/(2*0.71)=1.44, but the authors report that t = 8.33, p < .001. More simply stated,if the authors are indeed reporting β values, then many of the β values seem to be impossible (outside of the ± 1 range), but if they are reporting B values, the t values seem to be impossible according to our calculations. We were wondering how these seemingly paradoxical results could be explained.

In conclusion, even if this research topic is of great interest, two main points should to be taken into account. First of all, it is hard to state that ACT is an effective treatment to reduce partner aggression because 1) partner aggression was never specifically assessed 2) partner attrition was not measured 3) of potential bias due to the experimenter expectancy effect. Secondly, it is hard to draw conclusions from the statistics figured in the study because 1) if β values were indeed being reported, then they lie outside of the ± 1 range and 2) if B values were actually reported, then the t values become incoherent.

References

Christensen, H. B. (1986). La statistique: démarche pédagogique programmée. Boucherville: Gaëtan Morin.

Cohen, J., Cohen P., West, S. G. et Aiken, L. S. (2002). Applied multiple regression/correlation analysis for the behavioral sciences. New York: Routledge.

Straus, M. A., Hamby, S. L., Boney-McCoy, S., & Sugarman, D. B. (1996). The revised conflict tactics scales (CTS2): Development and preliminary psychometric data. Journal of Family Issues, 17, 283–316.

Zarling, A. (2013). A preliminary trial of ACT skills training for aggressive behavior. University of Iowa.

On 2015 Sep 17, Ghassan El-Baalbaki commented:

authors of this comment are: Loose, T.^1,3 , Ashrah, L.^1, Romero, M.^1, Bégin, J.¹ and El-Baalbaki, G.^1,2 Affiliations, Université du Québec À Montréal^1, McGill University^2, Université de Nantes³ .

In this study, the authors examine the efficacy of Acceptance and Commitment Therapy (ACT) in treating partner aggression. After entering into communication with the first author, she confirmed that this study corresponds to her doctoral thesis (Zarling, 2013). We read her thesis in order to gain further knowledge about the published study, and we found two important aspects of the work that need to be brought into light: 1) we find it difficult to conclude that partner aggression was specifically reduced, because of threats to internal validity. 2) There seems to be inaccuracies in their statistical calculations.

The primary objective was to demonstrate that ACT reduces intimate partner aggression better than the placebo control group. The authors strongly suggest that ACT was indeed able to do so. However, several points call this statement into question.

First of all, it is worth noting that partner aggression is never specifically assessed. One may think that the Conflict Tactics Scales-2-Physical Assault Scale (CTS-2) did assess this construct (Straus, Hamby, Boney-McCoy & Sugarman, 1996), but when taking into account the exact content of the administered questionnaire that was figured in her thesis, it can be seen that the instructions were modified in order to include aggression towards “people we care about” (Zarling, 2013, p.132) instead of specifically towards a partner. This modification is not mentioned in the published article. Hence, participants could have hypothetically committed aggressive acts exclusively towards a non-partner, such as a child or a friend. It is thus hard to conclude that aggressive behavior, specifically among couples, decreased as a result of therapy.

Furthermore, after modifying the instructions, one of the items even became incoherent. More specifically, when taking into the modified instructions of the questionnaire, the item stated “When upset or in a disagreement with someone you care about … have you became angry enough to frighten your partner?” (italics added)(Zarling, 2013, p. 132). It seems that this question was destined to ask if the participant became angry enough to frighten someone that they « care about » and not their « partner ». For example, if the participants were (or became) single, they could have interpreted this question as if they were angry enough to frighten someone that is not necessarily their partner. It is unclear how the authors wanted the participants to interpret this item. The way that the construct of partner aggression was assessed calls into question the internal validity of the study.

It remains questionable why the authors neglected to assess attrition of couples during the study. Over the course of the study, it is possible that aggressive behavior decreased more in the ACT group as a result of more couples breaking up in this treatment condition than in the placebo control group. Even if the authors carried out both completer analysis and intent to treat analysis and the results did not differ on measured variables, the attrition of couples was not measured. Hence, it is impossible to know if the groups differed on partner attrition. It remains possible that participants continued to aggress former partners, but again this cannot be asserted simply because it was not measured. Assessing partner attrition would have helped neutralize a pertinent threat to internal validity. Hence, internal validity was limited by not taking into account partner attrition.

Lastly, it is worth bringing up that the first author and two other investigators co-led all administered interventions (p. 201). Even if the authors state this in their article, after looking at the detailed explanation of the recruitment process of investigators, this choice does not seem to be justified. It seems that the first author was able to recruit investigators at no extra cost and it is questionable as to why she did not recruit three instead of two, thus avoiding a potential threat to internal validity. The explanation provided in her thesis, but not the article, also states that all the manipulation checks were carried out by the first author and supervised by the second and third authors (Zarling, 2013, p.61). Because of the implication of the authors in the experimental procedure, the results are potentially attributable to the experimenter expectancy effect. This is another threat to internal validity that evokes the tendency for researchers to unintentionally bias the results of their investigations in accord with their hypotheses. In taking into account the previously stated points, it seems difficult to conclude that this study shows that ACT specifically reduces partner aggression better than a placebo control group.

On another note, the statistical validity of the study can be questioned:

Throughout the results section of the paper (pp. 205–207), the authors report many beta values (β) for intercepts and slopes. If these are indeed beta values, they should vary between approximately 1 and -1 (Cohen, Cohen, West & Aiken, 2002). However in many cases, reported values are outside of this range. For example the authors state “The ACT participants also reported significantly less psychological aggression, β = 2.05, SE = 0.71, t(97) = 8.33, p < .001, and physical aggression, β = 2.21, SE = 0.65, t(97) = 8.19, p < .001, at the 6-month follow-up assessment” (p. 8). This leads us to think that the authors are actually reporting non-standardized coefficients (B) because B values can vary outside of this range. However, if this is the case, then the t values reported become incoherent. t values should equal B/(2SE) (we retained the value of 2 after rounding up from 1.96 which corresponds to 95% of a normal distribution (Christensen, 1986)), but reported t values do not correspond to this calculation. For example, let’s take another look at the previous citation and assume that it actually refers to B values. According to our calculations, t = B/(2SE) = 2.05/(2*0.71)=1.44, but the authors report that t = 8.33, p < .001. More simply stated,if the authors are indeed reporting β values, then many of the β values seem to be impossible (outside of the ± 1 range), but if they are reporting B values, the t values seem to be impossible according to our calculations. We were wondering how these seemingly paradoxical results could be explained.

In conclusion, even if this research topic is of great interest, two main points should to be taken into account. First of all, it is hard to state that ACT is an effective treatment to reduce partner aggression because 1) partner aggression was never specifically assessed 2) partner attrition was not measured 3) of potential bias due to the experimenter expectancy effect. Secondly, it is hard to draw conclusions from the statistics figured in the study because 1) if β values were indeed being reported, then they lie outside of the ± 1 range and 2) if B values were actually reported, then the t values become incoherent.

References

Christensen, H. B. (1986). La statistique: démarche pédagogique programmée. Boucherville: Gaëtan Morin.

Cohen, J., Cohen P., West, S. G. et Aiken, L. S. (2002). Applied multiple regression/correlation analysis for the behavioral sciences. New York: Routledge.

Straus, M. A., Hamby, S. L., Boney-McCoy, S., & Sugarman, D. B. (1996). The revised conflict tactics scales (CTS2): Development and preliminary psychometric data. Journal of Family Issues, 17, 283–316.

Zarling, A. (2013). A preliminary trial of ACT skills training for aggressive behavior. University of Iowa.