On 2013 Oct 23, Chris Del Mar commented:

We appraised this using the FAST mnemonic.

‘F’ind seemed odd: few hits for such a search, (~1000 hits, not including some presumed overlap); ‘A’ppraisal was apparently secure: only RCTs were included, (however little was offered in the form of critically appraising the included studies for methodological rigour); ‘S’ynthesis reaped a major flaw: the studies were meta-analysed in forest plots by changes from baseline – not against the controls. This explains the greater effect sizes than expected, and probably represents a fatal flaw: in effect, only the intervention arms of the RCTs were included in a before-after comparison; ‘T’ransfer of the results was therefore not really applicable.

What should we do?

1 this is a very good teaching tool (a bad example, to see if students can pick the flaws, and a lesson that not all SRs are good!);

2 perhaps this topic needs systematically reviewing properly (if not done already);

3 should we write to the journal to set out the flaw? Perhaps we are probably too late to send in a corrective response.

On 2013 Oct 23, Chris Del Mar commented:

We appraised this using the FAST mnemonic.

‘F’ind seemed odd: few hits for such a search, (~1000 hits, not including some presumed overlap); ‘A’ppraisal was apparently secure: only RCTs were included, (however little was offered in the form of critically appraising the included studies for methodological rigour); ‘S’ynthesis reaped a major flaw: the studies were meta-analysed in forest plots by changes from baseline – not against the controls. This explains the greater effect sizes than expected, and probably represents a fatal flaw: in effect, only the intervention arms of the RCTs were included in a before-after comparison; ‘T’ransfer of the results was therefore not really applicable.

What should we do?

1 this is a very good teaching tool (a bad example, to see if students can pick the flaws, and a lesson that not all SRs are good!);

2 perhaps this topic needs systematically reviewing properly (if not done already);

3 should we write to the journal to set out the flaw? Perhaps we are probably too late to send in a corrective response.