On 2017 Nov 25, Tom Kindlon commented:

More data demonstrating why the SF-36 physical functioning threshold of 60+ is problematic

The PACE trial's principal investigators (PIs) argue that scores of 60 or higher on the SF-36 physical function subscale (SF-36 PF) fall within the "normal range" and use this threshold as one component of their revised definition of recovery from Chronic Fatigue Syndrome (CFS), as well as a post hoc measure in the main PACE trial paper[1,2].

Referring to the normal ranges for physical function and fatigue used in the PACE trial, one of the trial's PIs, Trudie Chalder, stated that “twice as many people on graded exercise therapy and cognitive behaviour therapy got back to normal”[3]. This same terminology, "back to normal", was also used by the PIs in a patient information leaflet[4].

The use of this phrase, which implies full restoration of health, merits closer scrutiny following publication of a new paper containing data from CFS patients assessed at one of the PACE trial's treatment centres, the Chronic Fatigue Research and Treatment Unit in London, UK, between November 2007 and January 2014[5].

The paper, whose corresponding author is Trudie Chalder, reports that 39.6% of the patients diagnosed with CFS (228/576) had SF-36 PF scores of more than 60; i.e, despite having a score within the normal range, they were judged as sufficiently disabled to be diagnosed with CFS. Note that the data given were for more than 60 on the SF-36 PF so the figure for 60 or more would likely be higher again.

Moreover, Dutch researchers recently reported that 39% of patients with CFS had SF-36 PF scores of 65 or higher[6], and in the PACE trial itself patients could start with a SF-36 PF score of 65 at baseline[7]. Also, as we reported in a reanalysis of the PACE Trial data, in a large British community sample 90% of people aged 18–59 without a long-term illness or disability actually score 90 or higher[8].

Taken together, these data suggest that the derived normal range threshold for physical function is too low to serve as a meaningful indicator of recovery and does not, in fact, represent getting "back to normal".

References:

1 White PD1, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Bavinton J, Angus BJ, Murphy G, Murphy M, O'Dowd H, Wilks D, McCrone P, Chalder T, Sharpe M; PACE trial management group. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011; 377: 611–690

2 White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M. Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychol Med. 2013;43(10):2227–2235.

3 Boseley, S. Study finds therapy and exercise best for ME. The Guardian. Feb 18, 2011 https://www.theguardian.com/society/2011/feb/18/study-exercise-therapy-me-treatment

4 PACE participants newsletter 4. February 2011. http://www.wolfson.qmul.ac.uk/images/pdfs/participantsnewsletter4.pdf

5 Ryan EG, Vitoratou S, Goldsmith KA, Chalder T. Psychometric properties and factor structure of a shortened version of the Cognitive Behavioural Responses Questionnaire (CBRQ). Psychosom Med. 2017 Oct 10. doi: 10.1097/PSY.0000000000000536. [Epub ahead of print]

6 Janse A, Nikolaus S, Wiborg JF, Heins M, van der Meer JWM, Bleijenberg G, Tummers M, Twisk J, Knoop H. Long-term follow-up after cognitive behaviour therapy for chronic fatigue syndrome. J Psychosom Res, 2017. doi: 10.1016/j.jpsychores.2017.03.016.

7 White, PD, Sharpe, MC, Chalder, T et al. Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol. 2007; 7: 6

8 Wilshire CE, Kindlon T, Matthees A, et al. Can patients with chronic fatigue syndrome really recover after graded exercise or cognitive behavioural therapy? A critical commentary and preliminary re-analysis of the PACE trial. Fatigue. 2017;5:1–4.

On 2016 Sep 27, Alem Matthees commented:

A preliminary re-analysis of ‘recovery’ in the PACE trial based on the published protocol

Much has been written about the problems with the revised recovery criteria in the PACE trial and the poorly or erroneously justified deviations from the published protocol after the trial was over. There have been calls for a re-analysis of recovery using the protocol-specified recovery criteria. While a formal publication is still pending, a preliminary re-analysis of 'recovery' using individual participant data has recently been released on Virology Blog [1].

http://www.virology.ws/2016/09/21/no-recovery-in-pace-trial-new-analysis-finds/

Summary: The PACE trial tested interventions for chronic fatigue syndrome, but the published ‘recovery’ rates were based on thresholds that deviated substantially from the published trial protocol. Individual participant data on a selection of measures has recently been released under the Freedom of Information Act, enabling the re-analysis of recovery rates in accordance with the thresholds specified in the published trial protocol. The recovery rate using these thresholds is 3.1% for specialist medical care alone; for the adjunctive therapies it is 6.8% for cognitive behavioural therapy, 4.4% for graded exercise therapy, and 1.9% for adaptive pacing therapy. This re-analysis demonstrates that the previously reported recovery rates were inflated by an average of four-fold. Furthermore, in contrast with the published paper by the trial investigators, the recovery rates in the cognitive behavioural therapy and graded exercise therapy groups are not significantly higher than with specialist medical care alone. The implications of these findings are discussed.

Update 1:

When asked about the re-analysis for an article in The Times, Professor Peter White (lead PACE trial investigator) did not dispute the methodology of the re-analysis, but added that the argument was about the definition of recovery. He stated: "We thought people who rated their health as 'much better' or 'very much better' should be included. They used 'very much better'." [2]

We used the recovery criteria as established by Prof. White and colleagues in the published PACE trial protocol. However, including those who rated themselves "much better" makes little difference to the number of participants classified as recovered when using the other protocol-specified recovery criteria, even when imputing the missing participant-rated CGI scores with doctor-rated scores, which tend to be more optimistic than the participant-rated scores: SMC, 5 to 6; APT, 3 (unchanged); CBT, 11 to 13; GET, 7 to 9. Our conclusion remains the same, no therapy group has a (statistically) significantly higher rate of recovery than for SMC alone, for either intention-to-treat or available-case. Almost all the participants who rated themselves "much better" failed to meet the remaining protocol-specified recovery criteria.

The comment from Prof. White does not address the major changes to other criteria. The revised "normal range" for fatigue and physical function overlaps with trial eligibility criteria for severe disabling fatigue, whereas previously there was a significant gap. Not meeting Oxford CFS criteria in the revised recovery criteria is not what it sounds: participants were counted as not meeting Oxford CFS criteria if they had a CFQ (bimodal) fatigue score of less than 6 or a SF-36 physical function score of more than 65, irrespective of whether they still met Oxford CFS criteria or not. Approximately half of those who 'no longer met Oxford CFS criteria' according to the revised recovery criteria still actually met Oxford CFS criteria. Feeling "much better" is not necessarily the same as recovered and can be a result of changes not relating to fatigue or physical function. None of the revised recovery criteria, alone or combined, convincingly reflect being recovered. Over one-third meeting all the revised recovery criteria still met Oxford CFS criteria.

Contrary to the impression given by Professor George Lewith's statement for The Times [2], we did not "torture the data until it proves what they believe", we followed the published PACE trial protocol before it was changed after the trial was over, and we made this clear in our article. While the article in The Times states that Prof. Lewith was not involved in the original research, perhaps it should be noted that he has co-authored a paper with the co-principal PACE trial investigators using PACE trial data [3].

Individuals who defended the revised PACE trial recovery criteria, including in the UK House of Lords [4], argued that the results were impressive or meaningful because it means no longer having CFS. Unfortunately, the PACE trial data shows or confirms that these people have been misled.

Update 2:

In an article in The Guardian, Prof. White asserts that "The authors got their figures by tweaks such as increasing the pass-grade for what counted as recovery, and excluding patients who had reported themselves as 'much better'." [5]

This implies that we fiddled with the recovery criteria to get the results we wanted. That is false and misleading; again, we simply used the thresholds established by Prof. White himself (and colleagues) in their own published trial protocol, before they changed it after the trial was over. Furthermore, counting "much better" towards recovery makes no significant difference to the results; CBT and GET still do not significantly increase recovery rates.

References

1) Matthees A, Kindlon T, Maryhew C, Stark P, Levin B. A preliminary analysis of ‘recovery’ from chronic fatigue syndrome in the PACE trial using individual participant data. Virology Blog. 21 September 2016. http://www.virology.ws/wp-content/uploads/2016/09/preliminary-analysis.pdf

2) Whipple T. Exercise and therapy cure for ME is 'seriously flawed'. The Times. 28 September 2016. http://www.thetimes.co.uk/article/b0c9d588-84d8-11e6-9270-cf26736cb244

3) Lewith G, Stuart B, Chalder T, McDermott C, White PD. Complementary and alternative healthcare use by participants in the PACE trial of treatments for chronic fatigue syndrome. J Psychosom Res. 2016 Aug;87:37-42. doi: 10.1016/j.jpsychores.2016.06.005. Epub 2016 Jun 10. PMID: 27411750.

4) PACE Trial: Chronic Fatigue Syndrome/Myalgic Encephalomyelitis. House of Lords Hansard. Volume 743. 06 February 2013. https://hansard.parliament.uk/Lords/2013-02-06/debates/130206114000195/PACETrialChronicFatigueSyndromeMyalgicEncephalomyelitis

5) White P. If my team’s research on ME is rejected, the patients will suffer. The Guardian. 30 September 2016. https://www.theguardian.com/commentisfree/2016/sep/30/me-chronic-fatigue-syndrome-patients-suffer-put-off-treatments-our-research

On 2016 Mar 31, Lily Chu commented:

Dr. Rebecca Goldin of STATS.org, connected to the American Statistical Association, recently published a critique of this paper at the link below:

http://www.stats.org/pace-research-sparked-patient-rebellion-challenged-medicine/

On 2016 Feb 15, Sam Carter commented:

Exploring changes to PACE trial outcome measures using anonymised data from the FINE trial.

When the results of the PACE trial were published (1, 2) it was noted that the primary outcome measures and the definition of "recovery" described in the trial's published protocol (3) had been abandoned and replaced with markedly less stringent criteria.

The fully anonymised data set from the FINE trial(4), considered to be the PACE trial's "sister" study, makes it possible to explore how these changes may have affected the reported efficacy of the PACE trial's interventions.

At week 20 (assessment 2), 18 FINE trial participants met PACE trial post-hoc recovery thresholds (SF36 PF ≥ 60 and CFQ Likert ≤ 18) compared to only 3 participants who met the stricter, protocol-defined recovery thresholds (SF36 PF ≥ 85 and CFQ bimodal ≤ 3). Therefore, at assessment 2, the post-hoc changes increased the "recovery" rate by a factor of 6.

By week 70 (assessment 3), between 10 and 12 of the original 18 had relapsed so that they no longer met the post-hoc recovery thresholds (data are missing for two participants). Such a high rate of relapse within a year shows that the post-hoc recovery thresholds, said to represent a "strict criterion for recovery" in a Comment (5) which accompanied the original publication of PACE trial results, are neither strict nor reliable indicators of sustained wellbeing.

Regarding the Chalder fatigue questionnaire, White et al wrote that "we changed the original bimodal scoring of the Chalder fatigue questionnaire (range 0–11) to Likert scoring to more sensitively test our hypotheses of effectiveness" (1). However, data from the FINE trial show that Likert and bimodal scores are often contradictory and thus call into question White et al's assumption that Likert scoring is necessarily more sensitive than bimodal scoring.

For example, of the 33 FINE trial participants who met the post-hoc PACE trial recovery threshold for fatigue at week 20 (Likert CFQ score ≤ 18), 10 had a bimodal CFQ score ≥ 6 so would still be fatigued enough to enter the PACE trial and 16 had a bimodal CFQ score ≥ 4 which is the accepted definition of abnormal fatigue.

Therefore, for this cohort, if a person met the PACE trial post-hoc recovery threshold for fatigue at week 20 they had approximately a 50% chance of still having abnormal levels of fatigue and a 30% chance of being fatigued enough to enter the PACE trial.

A further problem with the Chalder fatigue questionnaire is illustrated by the observation that the bimodal score and Likert score of 10 participants moved in opposite directions at consecutive assessments i.e. one scoring system showed improvement whilst the other showed deterioration.

Moreover, it can be seen that some FINE trial participants were confused by the wording of the questionnaire itself. For example, a healthy person should have a Likert score of 11 out of 33, yet 17 participants recorded a Likert CFQ score of 10 or less at some point (i.e. they reported less fatigue than a healthy person), and 5 participants recorded a Likert CFQ score of 0.

The discordance between Likert and bimodal scores and the marked increase in those meeting post-hoc recovery thresholds suggest that White et al's deviation from their protocol-specified analysis is likely to have profoundly affected the reported efficacy of the PACE trial interventions.

An independent re-analysis of anonymised PACE trial data as described in its published protocol is urgently required to quantify the effects of the revised outcome and recovery criteria.

References

(1) White PD et al (2011) Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet Mar 5;377(9768):823-36.<br> (2) White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M (2013) Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychol Med. Oct;43(10):2227-35.<br> (3) White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R (2007) Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol Mar 8;7:6.<br> (4) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4685991/bin/pone.0144623.s002.dta<br> (5) Bleijenberg G, Knoop H. (2011) Chronic fatigue syndrome: where to PACE from here? Lancet. Mar 5;377(9768):786-8.

On 2015 Oct 25, Tom Kindlon commented:

A lot of the focus of the critique is this paper (i.e. Recovery from chronic fatigue syndrome after treatments given in the PACE trial).

The critique is spread over three pieces: http://www.virology.ws/2015/10/21/trial-by-error-i/ ; http://www.virology.ws/2015/10/22/trial-by-error-ii/ ; http://www.virology.ws/2015/10/23/trial-by-error-iii/

On 2015 Oct 23, Lily Chu commented:

For a critique of the PACE trial and papers related to it, see:

http://www.virology.ws/2015/10/21/trial-by-error-i/

On 2014 Sep 20, Tom Kindlon commented:

Letter published criticising aspects of the PACE Trial recovery paper and criteria:

Kindlon T, Baldwin A. Response to: reports of recovery in chronic fatigue syndrome may present less than meets the eye. Evid Based Mental Health doi:10.1136/eb-2014-101961 http://ebmh.bmj.com/content/early/2014/09/19/eb-2014-101961.extract

On 2014 Apr 28, Tom Kindlon commented:

Easy-to-understand video criticising the (adjusted) recovery definition:

A former mathematics teacher has made an easy-to-understand video explaining and criticising the PACE Trial's recovery criteria:

http://youtu.be/d_7J5ELjArU

On 2013 Oct 29, Tom Kindlon commented:

(contd.)

References

Cella M, Chalder T (2010). Measuring fatigue in clinical and community settings. Journal of Psychosomatic Research 69, 17–22.

Chalder T, Berelowitz G, Hirsch S, Pawlikowska T, Wallace P, Wessely S (1993). Development of a fatigue scale. Journal of Psychosomatic Research 37, 147–153.

Evans S (2007). When and how can endpoints be changed after initiation of a randomized clinical trial? PLoS Clin Trials 2, e18.

Lerdal A, Wahl A, Rustøen T, Hanestad BR, Moum T (2005). Fatigue in the general population: a translation and test of the psychometric properties of the Norwegian version of the fatigue severity scale. Scandinavian Journal of Public Health 33, 123-30.

McAteer A, Elliott AM, Hannaford PC (2011). Ascertaining the size of the symptom iceberg in a UK-wide community based survey. British Journal of General Practice 61, e1– e11.

Stulemeijer M, de Jong LW, Fiselier TJ, Hoogveld SW, Bleijenberg G (2005). Cognitive behaviour therapy for adolescents with chronic fatigue syndrome: randomised controlled trial. BMJ 330, 7481–7486.

Tummers M, Knoop H, van Dam A, Bleijenberg G (2012) . Implementing a minimal intervention for chronic fatigue syndrome in a mental health centre: a randomized controlled trial. Psychological Medicine 42, 2205-15

van't Leven M, Zielhuis GA, van der Meer JW, Verbeek AL, Bleijenberg G (2010). Fatigue and chronic fatigue syndrome-like complaints in the general population. European Journal of Public Health 20, 251-7

Wearden AJ, Dowrick C, Chew-Graham C, Bentall RP, Morriss RK, Peters S, Riste L, Richardson G, Lovell K, Dunn G; Fatigue Intervention by Nurses Evaluation (FINE) trial writing group and the FINE trial group (2010). Fatigue Intervention by Nurses Evaluation (FINE) trial writing group and the FINE trial group. Nurse led, home based self help treatment for patients in primary care with chronic fatigue syndrome: randomised controlled trial. BMJ 340, c1777.

White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M; PACE Trial Management Group (2013). Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychological Medicine Jan 31:1-9. [Epub ahead of print]

White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R; PACE trial group (2007). Protocol for the PACE trial : a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BioMed Central Neurology 7, 6.

On 2013 Oct 29, Tom Kindlon commented:

Changes to the recovery criteria have not improved their validity

When one publishes a protocol for a trial, as the PACE Trial investigators have done (White et al. 2007), there needs to be compelling reasons to deviate from it (Evans, 2007). White et al. (2013) claim that the revised recovery definition is conservative, with the changes being made to "more accurately reflect recovery". Is this true with regard to the Chalder Fatigue Questionnaire (CFQ) and SF-36 physical functioning (SF36 PF) criteria?

The new CFQ criterion, a score of 18 or less (Likert scoring), was chosen because it represented the mean plus 1 standard deviation in a community sample (Cella & Chalder, 2010). The CFQ scores were not normally distributed but we know that only 13.6% of the sample scored higher than 18. However, it does not follow that this threshold represents a reliable cut-off for fatigue-caseness as fatigue problems are common in the general population. For example, in the paper the authors referenced when discussing symptoms in the general population (McAteer et al. 2011), 41.3% reported “feeling tired/run down” while 23.1% of a representative sample of the Norwegian population had high levels of fatigue (Lerdal et al. 2005). That is to say, it is quite possible that more than 13.6% of the sample in Cella & Chalder (2010) were experiencing significant fatigue problems.

The recovery criteria described in the protocol require a score of 3 or less (bimodal scoring) which is a validated definition for the absence of fatigue (Chalder et al. 1993). Although exact translation between Likert and bimodal scores is not possible, it can be shown that such a score is stricter than the new criteria because it translates to a Likert score between 6 and 17. Therefore, when compared against the established definition of fatigue-caseness, a Likert score of 18 always indicates the presence of abnormal levels of fatigue.

Furthermore, the trial's entry criterion for fatigue, a CFQ bimodal score of 6 or higher, translates to a Likert score between 12 and 23 meaning that participants could have baseline scores which were already 18 or less so that no improvement was required for them to recover according to the new criteria. Indeed, 17.6% of patients diagnosed with CFS at the Chronic Fatigue Unit at the South London and Maudsley NHS Trust had scores of 18 or less on the CFQ before treatment for their fatigue.

For SF-36 PF scores, the protocol required a score of ≥85 for recovery, whilst the newer criteria require a score of ≥60. Again, participants could score 60 or more at baseline which suggests the new criterion is neither conservative nor "more accurately reflects recovery".

Also, while I have not undertaken an exhaustive search, in all the other trials that I am aware of that used the SF36 PF to operationalize CFS criteria, a score of 60 would have been sufficiently low to meet each trial's requirements for a diagnosis of CFS (e.g. Stulemeijer et al. 2005; Tummers et al. 2012; van't Leven et al. 2010; Wearden et al. 2010).

White et al. (2013) used the formula of mean minus one standard deviation (sd) from data on the UK general population from Bowling et al. (1999) to derive the threshold of SF-36 PF ≥60. However, CFS is not unique in causing reductions in this domain, with Bowling et al. noting that 22% in the same survey reported a long-term health problem while 16% reported having an acute illness. Moreover 28.6% were aged 65 or more; population norms from these age groups are of questionable relevance to the PACE Trial cohort (mean (sd) age at baseline: 38 (12)).

In summary, the CFQ and SF-36 PF criteria that constitute White and colleagues' new definition of recovery have been revised such that they are less strict than those contained in the published protocol. These changes suggest that it is not safe to conclude that the new criteria are either conservative or more accurately reflect recovery than those published in the trial's protocol.

On 2013 Oct 29, Tom Kindlon commented:

Changes to the recovery criteria have not improved their validity

When one publishes a protocol for a trial, as the PACE Trial investigators have done (White et al. 2007), there needs to be compelling reasons to deviate from it (Evans, 2007). White et al. (2013) claim that the revised recovery definition is conservative, with the changes being made to "more accurately reflect recovery". Is this true with regard to the Chalder Fatigue Questionnaire (CFQ) and SF-36 physical functioning (SF36 PF) criteria?

The new CFQ criterion, a score of 18 or less (Likert scoring), was chosen because it represented the mean plus 1 standard deviation in a community sample (Cella & Chalder, 2010). The CFQ scores were not normally distributed but we know that only 13.6% of the sample scored higher than 18. However, it does not follow that this threshold represents a reliable cut-off for fatigue-caseness as fatigue problems are common in the general population. For example, in the paper the authors referenced when discussing symptoms in the general population (McAteer et al. 2011), 41.3% reported “feeling tired/run down” while 23.1% of a representative sample of the Norwegian population had high levels of fatigue (Lerdal et al. 2005). That is to say, it is quite possible that more than 13.6% of the sample in Cella & Chalder (2010) were experiencing significant fatigue problems.

The recovery criteria described in the protocol require a score of 3 or less (bimodal scoring) which is a validated definition for the absence of fatigue (Chalder et al. 1993). Although exact translation between Likert and bimodal scores is not possible, it can be shown that such a score is stricter than the new criteria because it translates to a Likert score between 6 and 17. Therefore, when compared against the established definition of fatigue-caseness, a Likert score of 18 always indicates the presence of abnormal levels of fatigue.

Furthermore, the trial's entry criterion for fatigue, a CFQ bimodal score of 6 or higher, translates to a Likert score between 12 and 23 meaning that participants could have baseline scores which were already 18 or less so that no improvement was required for them to recover according to the new criteria. Indeed, 17.6% of patients diagnosed with CFS at the Chronic Fatigue Unit at the South London and Maudsley NHS Trust had scores of 18 or less on the CFQ before treatment for their fatigue.

For SF-36 PF scores, the protocol required a score of ≥85 for recovery, whilst the newer criteria require a score of ≥60. Again, participants could score 60 or more at baseline which suggests the new criterion is neither conservative nor "more accurately reflects recovery".

Also, while I have not undertaken an exhaustive search, in all the other trials that I am aware of that used the SF36 PF to operationalize CFS criteria, a score of 60 would have been sufficiently low to meet each trial's requirements for a diagnosis of CFS (e.g. Stulemeijer et al. 2005; Tummers et al. 2012; van't Leven et al. 2010; Wearden et al. 2010).

White et al. (2013) used the formula of mean minus one standard deviation (sd) from data on the UK general population from Bowling et al. (1999) to derive the threshold of SF-36 PF ≥60. However, CFS is not unique in causing reductions in this domain, with Bowling et al. noting that 22% in the same survey reported a long-term health problem while 16% reported having an acute illness. Moreover 28.6% were aged 65 or more; population norms from these age groups are of questionable relevance to the PACE Trial cohort (mean (sd) age at baseline: 38 (12)).

In summary, the CFQ and SF-36 PF criteria that constitute White and colleagues' new definition of recovery have been revised such that they are less strict than those contained in the published protocol. These changes suggest that it is not safe to conclude that the new criteria are either conservative or more accurately reflect recovery than those published in the trial's protocol.

On 2014 Apr 28, Tom Kindlon commented:

Easy-to-understand video criticising the (adjusted) recovery definition:

A former mathematics teacher has made an easy-to-understand video explaining and criticising the PACE Trial's recovery criteria:

http://youtu.be/d_7J5ELjArU

On 2014 Sep 20, Tom Kindlon commented:

Letter published criticising aspects of the PACE Trial recovery paper and criteria:

Kindlon T, Baldwin A. Response to: reports of recovery in chronic fatigue syndrome may present less than meets the eye. Evid Based Mental Health doi:10.1136/eb-2014-101961 http://ebmh.bmj.com/content/early/2014/09/19/eb-2014-101961.extract

On 2015 Oct 23, Lily Chu commented:

For a critique of the PACE trial and papers related to it, see:

http://www.virology.ws/2015/10/21/trial-by-error-i/

On 2016 Feb 15, Sam Carter commented:

Exploring changes to PACE trial outcome measures using anonymised data from the FINE trial.

When the results of the PACE trial were published (1, 2) it was noted that the primary outcome measures and the definition of "recovery" described in the trial's published protocol (3) had been abandoned and replaced with markedly less stringent criteria.

The fully anonymised data set from the FINE trial(4), considered to be the PACE trial's "sister" study, makes it possible to explore how these changes may have affected the reported efficacy of the PACE trial's interventions.

At week 20 (assessment 2), 18 FINE trial participants met PACE trial post-hoc recovery thresholds (SF36 PF ≥ 60 and CFQ Likert ≤ 18) compared to only 3 participants who met the stricter, protocol-defined recovery thresholds (SF36 PF ≥ 85 and CFQ bimodal ≤ 3). Therefore, at assessment 2, the post-hoc changes increased the "recovery" rate by a factor of 6.

By week 70 (assessment 3), between 10 and 12 of the original 18 had relapsed so that they no longer met the post-hoc recovery thresholds (data are missing for two participants). Such a high rate of relapse within a year shows that the post-hoc recovery thresholds, said to represent a "strict criterion for recovery" in a Comment (5) which accompanied the original publication of PACE trial results, are neither strict nor reliable indicators of sustained wellbeing.

Regarding the Chalder fatigue questionnaire, White et al wrote that "we changed the original bimodal scoring of the Chalder fatigue questionnaire (range 0–11) to Likert scoring to more sensitively test our hypotheses of effectiveness" (1). However, data from the FINE trial show that Likert and bimodal scores are often contradictory and thus call into question White et al's assumption that Likert scoring is necessarily more sensitive than bimodal scoring.

For example, of the 33 FINE trial participants who met the post-hoc PACE trial recovery threshold for fatigue at week 20 (Likert CFQ score ≤ 18), 10 had a bimodal CFQ score ≥ 6 so would still be fatigued enough to enter the PACE trial and 16 had a bimodal CFQ score ≥ 4 which is the accepted definition of abnormal fatigue.

Therefore, for this cohort, if a person met the PACE trial post-hoc recovery threshold for fatigue at week 20 they had approximately a 50% chance of still having abnormal levels of fatigue and a 30% chance of being fatigued enough to enter the PACE trial.

A further problem with the Chalder fatigue questionnaire is illustrated by the observation that the bimodal score and Likert score of 10 participants moved in opposite directions at consecutive assessments i.e. one scoring system showed improvement whilst the other showed deterioration.

Moreover, it can be seen that some FINE trial participants were confused by the wording of the questionnaire itself. For example, a healthy person should have a Likert score of 11 out of 33, yet 17 participants recorded a Likert CFQ score of 10 or less at some point (i.e. they reported less fatigue than a healthy person), and 5 participants recorded a Likert CFQ score of 0.

The discordance between Likert and bimodal scores and the marked increase in those meeting post-hoc recovery thresholds suggest that White et al's deviation from their protocol-specified analysis is likely to have profoundly affected the reported efficacy of the PACE trial interventions.

An independent re-analysis of anonymised PACE trial data as described in its published protocol is urgently required to quantify the effects of the revised outcome and recovery criteria.

References

(1) White PD et al (2011) Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet Mar 5;377(9768):823-36.<br> (2) White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M (2013) Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychol Med. Oct;43(10):2227-35.<br> (3) White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R (2007) Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol Mar 8;7:6.<br> (4) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4685991/bin/pone.0144623.s002.dta<br> (5) Bleijenberg G, Knoop H. (2011) Chronic fatigue syndrome: where to PACE from here? Lancet. Mar 5;377(9768):786-8.

On 2016 Mar 31, Lily Chu commented:

Dr. Rebecca Goldin of STATS.org, connected to the American Statistical Association, recently published a critique of this paper at the link below:

http://www.stats.org/pace-research-sparked-patient-rebellion-challenged-medicine/

On 2016 Sep 27, Alem Matthees commented:

A preliminary re-analysis of ‘recovery’ in the PACE trial based on the published protocol

Much has been written about the problems with the revised recovery criteria in the PACE trial and the poorly or erroneously justified deviations from the published protocol after the trial was over. There have been calls for a re-analysis of recovery using the protocol-specified recovery criteria. While a formal publication is still pending, a preliminary re-analysis of 'recovery' using individual participant data has recently been released on Virology Blog [1].

http://www.virology.ws/2016/09/21/no-recovery-in-pace-trial-new-analysis-finds/

Summary: The PACE trial tested interventions for chronic fatigue syndrome, but the published ‘recovery’ rates were based on thresholds that deviated substantially from the published trial protocol. Individual participant data on a selection of measures has recently been released under the Freedom of Information Act, enabling the re-analysis of recovery rates in accordance with the thresholds specified in the published trial protocol. The recovery rate using these thresholds is 3.1% for specialist medical care alone; for the adjunctive therapies it is 6.8% for cognitive behavioural therapy, 4.4% for graded exercise therapy, and 1.9% for adaptive pacing therapy. This re-analysis demonstrates that the previously reported recovery rates were inflated by an average of four-fold. Furthermore, in contrast with the published paper by the trial investigators, the recovery rates in the cognitive behavioural therapy and graded exercise therapy groups are not significantly higher than with specialist medical care alone. The implications of these findings are discussed.

Update 1:

When asked about the re-analysis for an article in The Times, Professor Peter White (lead PACE trial investigator) did not dispute the methodology of the re-analysis, but added that the argument was about the definition of recovery. He stated: "We thought people who rated their health as 'much better' or 'very much better' should be included. They used 'very much better'." [2]

We used the recovery criteria as established by Prof. White and colleagues in the published PACE trial protocol. However, including those who rated themselves "much better" makes little difference to the number of participants classified as recovered when using the other protocol-specified recovery criteria, even when imputing the missing participant-rated CGI scores with doctor-rated scores, which tend to be more optimistic than the participant-rated scores: SMC, 5 to 6; APT, 3 (unchanged); CBT, 11 to 13; GET, 7 to 9. Our conclusion remains the same, no therapy group has a (statistically) significantly higher rate of recovery than for SMC alone, for either intention-to-treat or available-case. Almost all the participants who rated themselves "much better" failed to meet the remaining protocol-specified recovery criteria.

The comment from Prof. White does not address the major changes to other criteria. The revised "normal range" for fatigue and physical function overlaps with trial eligibility criteria for severe disabling fatigue, whereas previously there was a significant gap. Not meeting Oxford CFS criteria in the revised recovery criteria is not what it sounds: participants were counted as not meeting Oxford CFS criteria if they had a CFQ (bimodal) fatigue score of less than 6 or a SF-36 physical function score of more than 65, irrespective of whether they still met Oxford CFS criteria or not. Approximately half of those who 'no longer met Oxford CFS criteria' according to the revised recovery criteria still actually met Oxford CFS criteria. Feeling "much better" is not necessarily the same as recovered and can be a result of changes not relating to fatigue or physical function. None of the revised recovery criteria, alone or combined, convincingly reflect being recovered. Over one-third meeting all the revised recovery criteria still met Oxford CFS criteria.

Contrary to the impression given by Professor George Lewith's statement for The Times [2], we did not "torture the data until it proves what they believe", we followed the published PACE trial protocol before it was changed after the trial was over, and we made this clear in our article. While the article in The Times states that Prof. Lewith was not involved in the original research, perhaps it should be noted that he has co-authored a paper with the co-principal PACE trial investigators using PACE trial data [3].

Individuals who defended the revised PACE trial recovery criteria, including in the UK House of Lords [4], argued that the results were impressive or meaningful because it means no longer having CFS. Unfortunately, the PACE trial data shows or confirms that these people have been misled.

Update 2:

In an article in The Guardian, Prof. White asserts that "The authors got their figures by tweaks such as increasing the pass-grade for what counted as recovery, and excluding patients who had reported themselves as 'much better'." [5]

This implies that we fiddled with the recovery criteria to get the results we wanted. That is false and misleading; again, we simply used the thresholds established by Prof. White himself (and colleagues) in their own published trial protocol, before they changed it after the trial was over. Furthermore, counting "much better" towards recovery makes no significant difference to the results; CBT and GET still do not significantly increase recovery rates.

References

1) Matthees A, Kindlon T, Maryhew C, Stark P, Levin B. A preliminary analysis of ‘recovery’ from chronic fatigue syndrome in the PACE trial using individual participant data. Virology Blog. 21 September 2016. http://www.virology.ws/wp-content/uploads/2016/09/preliminary-analysis.pdf

2) Whipple T. Exercise and therapy cure for ME is 'seriously flawed'. The Times. 28 September 2016. http://www.thetimes.co.uk/article/b0c9d588-84d8-11e6-9270-cf26736cb244

3) Lewith G, Stuart B, Chalder T, McDermott C, White PD. Complementary and alternative healthcare use by participants in the PACE trial of treatments for chronic fatigue syndrome. J Psychosom Res. 2016 Aug;87:37-42. doi: 10.1016/j.jpsychores.2016.06.005. Epub 2016 Jun 10. PMID: 27411750.

4) PACE Trial: Chronic Fatigue Syndrome/Myalgic Encephalomyelitis. House of Lords Hansard. Volume 743. 06 February 2013. https://hansard.parliament.uk/Lords/2013-02-06/debates/130206114000195/PACETrialChronicFatigueSyndromeMyalgicEncephalomyelitis

5) White P. If my team’s research on ME is rejected, the patients will suffer. The Guardian. 30 September 2016. https://www.theguardian.com/commentisfree/2016/sep/30/me-chronic-fatigue-syndrome-patients-suffer-put-off-treatments-our-research

On 2017 Nov 25, Tom Kindlon commented:

More data demonstrating why the SF-36 physical functioning threshold of 60+ is problematic

The PACE trial's principal investigators (PIs) argue that scores of 60 or higher on the SF-36 physical function subscale (SF-36 PF) fall within the "normal range" and use this threshold as one component of their revised definition of recovery from Chronic Fatigue Syndrome (CFS), as well as a post hoc measure in the main PACE trial paper[1,2].

Referring to the normal ranges for physical function and fatigue used in the PACE trial, one of the trial's PIs, Trudie Chalder, stated that “twice as many people on graded exercise therapy and cognitive behaviour therapy got back to normal”[3]. This same terminology, "back to normal", was also used by the PIs in a patient information leaflet[4].

The use of this phrase, which implies full restoration of health, merits closer scrutiny following publication of a new paper containing data from CFS patients assessed at one of the PACE trial's treatment centres, the Chronic Fatigue Research and Treatment Unit in London, UK, between November 2007 and January 2014[5].

The paper, whose corresponding author is Trudie Chalder, reports that 39.6% of the patients diagnosed with CFS (228/576) had SF-36 PF scores of more than 60; i.e, despite having a score within the normal range, they were judged as sufficiently disabled to be diagnosed with CFS. Note that the data given were for more than 60 on the SF-36 PF so the figure for 60 or more would likely be higher again.

Moreover, Dutch researchers recently reported that 39% of patients with CFS had SF-36 PF scores of 65 or higher[6], and in the PACE trial itself patients could start with a SF-36 PF score of 65 at baseline[7]. Also, as we reported in a reanalysis of the PACE Trial data, in a large British community sample 90% of people aged 18–59 without a long-term illness or disability actually score 90 or higher[8].

Taken together, these data suggest that the derived normal range threshold for physical function is too low to serve as a meaningful indicator of recovery and does not, in fact, represent getting "back to normal".

References:

1 White PD1, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Bavinton J, Angus BJ, Murphy G, Murphy M, O'Dowd H, Wilks D, McCrone P, Chalder T, Sharpe M; PACE trial management group. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011; 377: 611–690

2 White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M. Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychol Med. 2013;43(10):2227–2235.

3 Boseley, S. Study finds therapy and exercise best for ME. The Guardian. Feb 18, 2011 https://www.theguardian.com/society/2011/feb/18/study-exercise-therapy-me-treatment

4 PACE participants newsletter 4. February 2011. http://www.wolfson.qmul.ac.uk/images/pdfs/participantsnewsletter4.pdf

5 Ryan EG, Vitoratou S, Goldsmith KA, Chalder T. Psychometric properties and factor structure of a shortened version of the Cognitive Behavioural Responses Questionnaire (CBRQ). Psychosom Med. 2017 Oct 10. doi: 10.1097/PSY.0000000000000536. [Epub ahead of print]

6 Janse A, Nikolaus S, Wiborg JF, Heins M, van der Meer JWM, Bleijenberg G, Tummers M, Twisk J, Knoop H. Long-term follow-up after cognitive behaviour therapy for chronic fatigue syndrome. J Psychosom Res, 2017. doi: 10.1016/j.jpsychores.2017.03.016.

7 White, PD, Sharpe, MC, Chalder, T et al. Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol. 2007; 7: 6

8 Wilshire CE, Kindlon T, Matthees A, et al. Can patients with chronic fatigue syndrome really recover after graded exercise or cognitive behavioural therapy? A critical commentary and preliminary re-analysis of the PACE trial. Fatigue. 2017;5:1–4.