On 2017 Jul 14, Daniel Weeks commented:

Concerns about the analyses presented in the Han et al (2017) paper have been described by Frank Harrell in his "Statistical Thinking" blog in the "Improper Subgrouping" section of the post entitled "Statistical Errors in the Medical Literature". He points out that this paper "makes the classic statistical error of attempting to learn about differences in treatment effectiveness by subgrouping rather than by correctly modeling interactions. They compounded the error by not adjusting for covariates when comparing treatments in the subgroups, and even worse, by subgrouping on a variable for which grouping is ill-defined and information-losing: age.". For further details and additional concerns, please see the blog post.

On 2017 Jul 05, David Keller commented:

Rationale for performing a per-protocol analysis of this study's data

The percentage of control patients who were switched to statin treatment by their personal physicians reached 29.0 % by the end of the study. Meanwhile, non-adherence to statin therapy grew to 22.2 % in the active treatment group.

Intention-to-treat (ITT) analysis assigns the events experienced by a patient (e.g. death, heart attack) to the group to which the patient was initially randomized, regardless of whether the patient actually took the study medicine (if he was randomized to take it), or whether he took the medicine off-protocol (despite being randomized to the control group). The net effect of these "crossover" events, from control to active treatment, or vice-versa, is to weaken the apparent benefit of the medication, as calculated using ITT. This is good, because as a sort of "worst-case scenario" evaluation, we are assured that, if our patients actually take the study medication, they should probably benefit at least as much as the patients did in the clinical trial.

However, in a case where we are evaluating whether there is really no benefit to a medication, we should also consider a best-case scenario evaluation, because if the medicine is not beneficial even when it is evaluated in a manner which is more highly sensitive for detecting benefit, we can be that much more certain that the study medication has no role in treating the study population.

Per-protocol analysis assigns the outcomes and events experienced by a patient based on his actual behavior during the study. If he crossed over from the control group to active treatment and then had a good outcome, that good outcome would be attributed by per-protocol analysis to the effects of active treatment, not to control treatment. Conversely, if a patient is randomized to active treatment, but never takes a pill, a bad outcome in his case would be "blamed" on the control treatment, not on the study medicine he never took. Per-protocol provides a "best case" scenario evaluation of the study drug; one can think of per-protocol analysis as having increased sensitivity to the benefits of the study medication.

So, in a study like this, it is not enough to perform an intention-to-treat analysis, because all those crossovers might have obscured a significant signal of benefit. It is important to also perform a per-protocol analysis, to assure ourselves that, even in the best of circumstances, the medicine being studied is not beneficial, if even the per-protocol analysis cannot detect benefit.

If per-protocol analysis reveals benefit to the study medication, but intention-to-treat analysis does not show any benefit, then a new study should be conducted, which is better-designed and more carefully executed. In this case, a stronger statin, like atorvastatin, could be tested, and the patients and investigators could be double-blinded, and so on...

On 2017 Jul 05, David Keller commented:

This randomized trial of pravastatin was not blinded, so expectation effects may run rampant

Han and colleagues fail to mention in the above abstract that this randomized study of pravastatin for primary prevention of coronary heart disease and mortality was conducted open-label (unblinded). Blinding of subjects and investigators in clinical trials is required to control expectation effects, which can have an important influence in triggering cardiovascular events. The open-label design of this study is mentioned in the Methods section of the body of the paper, but there is no further discussion of the role of uncontrolled placebo, nocebo, Pygmalion and other expectation effects on the outcome of this study. Why was this study not double-blinded, and how dependable are open-label data for making important clinical decisions regarding statin use?

On 2017 Jul 05, David Keller commented:

This randomized trial of pravastatin was not blinded, so expectation effects may run rampant

Han and colleagues fail to mention in the above abstract that this randomized study of pravastatin for primary prevention of coronary heart disease and mortality was conducted open-label (unblinded). Blinding of subjects and investigators in clinical trials is required to control expectation effects, which can have an important influence in triggering cardiovascular events. The open-label design of this study is mentioned in the Methods section of the body of the paper, but there is no further discussion of the role of uncontrolled placebo, nocebo, Pygmalion and other expectation effects on the outcome of this study. Why was this study not double-blinded, and how dependable are open-label data for making important clinical decisions regarding statin use?

On 2017 Jul 05, David Keller commented:

Rationale for performing a per-protocol analysis of this study's data

The percentage of control patients who were switched to statin treatment by their personal physicians reached 29.0 % by the end of the study. Meanwhile, non-adherence to statin therapy grew to 22.2 % in the active treatment group.

Intention-to-treat (ITT) analysis assigns the events experienced by a patient (e.g. death, heart attack) to the group to which the patient was initially randomized, regardless of whether the patient actually took the study medicine (if he was randomized to take it), or whether he took the medicine off-protocol (despite being randomized to the control group). The net effect of these "crossover" events, from control to active treatment, or vice-versa, is to weaken the apparent benefit of the medication, as calculated using ITT. This is good, because as a sort of "worst-case scenario" evaluation, we are assured that, if our patients actually take the study medication, they should probably benefit at least as much as the patients did in the clinical trial.

However, in a case where we are evaluating whether there is really no benefit to a medication, we should also consider a best-case scenario evaluation, because if the medicine is not beneficial even when it is evaluated in a manner which is more highly sensitive for detecting benefit, we can be that much more certain that the study medication has no role in treating the study population.

Per-protocol analysis assigns the outcomes and events experienced by a patient based on his actual behavior during the study. If he crossed over from the control group to active treatment and then had a good outcome, that good outcome would be attributed by per-protocol analysis to the effects of active treatment, not to control treatment. Conversely, if a patient is randomized to active treatment, but never takes a pill, a bad outcome in his case would be "blamed" on the control treatment, not on the study medicine he never took. Per-protocol provides a "best case" scenario evaluation of the study drug; one can think of per-protocol analysis as having increased sensitivity to the benefits of the study medication.

So, in a study like this, it is not enough to perform an intention-to-treat analysis, because all those crossovers might have obscured a significant signal of benefit. It is important to also perform a per-protocol analysis, to assure ourselves that, even in the best of circumstances, the medicine being studied is not beneficial, if even the per-protocol analysis cannot detect benefit.

If per-protocol analysis reveals benefit to the study medication, but intention-to-treat analysis does not show any benefit, then a new study should be conducted, which is better-designed and more carefully executed. In this case, a stronger statin, like atorvastatin, could be tested, and the patients and investigators could be double-blinded, and so on...

On 2017 Jul 14, Daniel Weeks commented:

Concerns about the analyses presented in the Han et al (2017) paper have been described by Frank Harrell in his "Statistical Thinking" blog in the "Improper Subgrouping" section of the post entitled "Statistical Errors in the Medical Literature". He points out that this paper "makes the classic statistical error of attempting to learn about differences in treatment effectiveness by subgrouping rather than by correctly modeling interactions. They compounded the error by not adjusting for covariates when comparing treatments in the subgroups, and even worse, by subgrouping on a variable for which grouping is ill-defined and information-losing: age.". For further details and additional concerns, please see the blog post.