On 2016 May 03, Jason Doctor commented:

Dear Dr. Del Mar and Colleagues,

Thank you for your interest in our paper. We would like to respond to your questions and comments.

You ask, “Does this effect spill over to the other three quarters of ARIs? Or also to the other conditions for which antibiotics might be prescribed, including skin and urinary infections?” The implementation (or “triggering”) of our interventions was restricted to acute respiratory infections. We did not apply the study interventions in cases where co-morbid diagnoses for skin and urinary infections were present at the visit. We were unable to evaluate spill-over to these other diagnoses, but such spillover would be an interesting area for future research.

You also comment that our intervention could not evaluate total antibiotics dispensed and because of this may have underestimated the effects of the intervention for cases where ‘delayed prescribing’ was practiced. We actively attempted to discourage delayed prescribing with each of the three interventions by focusing on changing ordering behavior. Delayed prescribing is not a good treatment strategy because it sends conflicting messages to patients, forces patients to make a clinical decision, may result in patients consuming antibiotics unnecessarily, and may discourage follow-up visits for more serious medical conditions deserving careful evaluation (e.g., pneumonia).

You note that we found that diagnosis shifting was not evident, referring to our presentation of eTable 6. However, transforming the coefficient estimates in eTables 3, 4A and 4B into odds ratios, your group reports finding a significant effect on the trajectory of ‘antibiotic appropriate’ and ‘antibiotic inappropriate’ diagnoses over time. We note that eTables 3, 4A and 4b do not include antibiotic appropriate diagnoses of any kind, so evaluation of data from these eTables cannot measure diagnosis shifting. Only eTable 6, which reports our analysis of the proportion of all acute respiratory infections coded as antibiotic appropriate diagnoses over time, contains antibiotic appropriate diagnoses. As noted, we found no evidence of diagnosis shifting in the analyses reported in eTable 6.

As a final question, you ask why the control group’s data are decreasing in prescribing rate pre-randomization. You correctly point out that this cannot be due to the Hawthorne effect because it occurred prior to enrollment. You conjecture that this may be explained by diagnosis shifting of electronic health record coders. To address this, we make the following clarifying observations. First, the data presented in our graphs are from the statistical model and are not unadjusted raw rates over that time period. Unadjusted data were more variable during that period and while they showed an overall reduction, they did not show a strictly decreasing reduction month-to-month. Second, during the period of time before the intervention, there were numerous state and local efforts to reduce inappropriate prescribing. It is possible that the noisy downward trend was due to a greater awareness that brought about changing practice patterns over time. Third, as indicated in eTable 6, time was not a significant predictor of the proportion of all acute respiratory infections coded as antibiotic appropriate diagnoses. This means that the trend is unlikely due to any shifting of diagnoses due specifically to time. Whatever the reason for this trend, randomization and our primary analysis method insure that pre-intervention trajectories of any sort do not threaten the study’s statistical conclusions.

Authors: Daniella Meeker, PhD; Jeffrey A. Linder, MD, MPH; Craig R. Fox, PhD; Mark W. Friedberg, MD, MPP; Stephen D. Persell, MD, MPH; Noah J. Goldstein, PhD; Tara K. Knight, PhD; Joel W. Hay, PhD; Jason N. Doctor, PhD

Author Affiliations: Schaeffer Center for Health Policy and Economics, University of Southern California, Los Angeles (Meeker, Knight, Hay, Doctor); RAND Corporation, Santa Monica, California (Meeker); Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, Massachusetts (Linder, Friedberg); Anderson School of Management, University of California, Los Angeles (Fox, Goldstein); Department of Psychology, David Geffen School of Medicine at UCLA, Los Angeles (Fox, Goldstein); RAND Corporation, Boston, Massachusetts (Friedberg); Northwestern University, Chicago, Illinois (Persell).

On 2016 Mar 28, Chris Del Mar commented:

Reducing inappropriate antibioics for acute respiratory infections in primary care

We congratulate Meeker and colleagues on a very ambitious factorial trial to reduce antibiotics in primary care.1 The three interventions investigated appear to have had a small but important effect on the approximately one quarter of acute respiratory infections (ARIs) presenting to the primary care clinicians where antibiotics were judged inappropriate.

Does this effect spill over to the other three quarters of ARIs? Or also to the other conditions for which antibiotics might be prescribed, including skin and urinary infections? Parsimony for the indications studied might spill over into other clinical areas -- which would be important.

Sadly, this analysis could not measure changes in total antibiotics dispensed (rather than the surrogate outcome of those prescribed). The would be important because one important reduction strategy, ‘delayed prescribing’ (in which an antibiotic is prescribed but the patient advised to keep it ‘in case’, and not routinely have it dispensed2) might have been employed by some clinicians independently of the interventions being trialled, which might mean the effect is greater.

However there were two concerns raised at our Journal Club.

Might the observed effect be explained by Diagnosis Shifting, (in which high antibiotic prescribers disproportionately label a greater proportion of ARIs diagnoses as antibiotic-justifiable3)? The Authors declare diagnosis shifting was not evident, referring to eTable 6. However, transforming the coefficient estimates in eTables 3, 4A and 4B into odds ratios, we found a significant effect on the trajectory of ‘antibiotic appropriate’ and ‘antibiotic inappropriate’ diagnoses over time.

What explains the control group’s dramatic reduction of ‘antibiotic inappropriate’ prescriptions in the 18 months before the interventions commenced, which continued through the study randomization period? This was in the order of 10% in the pre-randomization period, and a further 10% in the control group during the randomization period – greater than for any of the interventions themselves, (taken from the slopes in Fig 2). This cannot be a Hawthorne effect because the data were collected for a time period before the clinicians were enrolled. It does not fit any nationwide trends. We speculate that it is explained by Diagnosis Shifting, perhaps by a misclassification by electronic health record coders. Until we understand this reduction, we can have no confidence in the smaller intervention effect.

Chris Del Mar MD professor of public health Paul Glasziou PhD professor of evidence based practice Elaine Beller MAppStat statistician On behalf of the Centre for Research in Evidence Based Practice Journal Club Bond University, Queensland 4229 Australia

1 Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: A randomized clinical trial. JAMA. 2016;315:562-70 2 Spurling GK, Del Mar CB, Dooley L, Foxlee R, Farley R. Delayed antibiotics for respiratory infections. Cochrane database of systematic reviews (Online). 2013;;CD004417:DOI: 10.1002/14651858.CD004417.pub3. 3 Howie JG, Richardson IM, Gill G, Durno D. Respiratory illness and antibiotic use in general practice. J R Coll Gen Pract. 1971;21:657-63.

On 2016 Mar 28, Chris Del Mar commented:

Reducing inappropriate antibioics for acute respiratory infections in primary care

We congratulate Meeker and colleagues on a very ambitious factorial trial to reduce antibiotics in primary care.1 The three interventions investigated appear to have had a small but important effect on the approximately one quarter of acute respiratory infections (ARIs) presenting to the primary care clinicians where antibiotics were judged inappropriate.

Does this effect spill over to the other three quarters of ARIs? Or also to the other conditions for which antibiotics might be prescribed, including skin and urinary infections? Parsimony for the indications studied might spill over into other clinical areas -- which would be important.

Sadly, this analysis could not measure changes in total antibiotics dispensed (rather than the surrogate outcome of those prescribed). The would be important because one important reduction strategy, ‘delayed prescribing’ (in which an antibiotic is prescribed but the patient advised to keep it ‘in case’, and not routinely have it dispensed2) might have been employed by some clinicians independently of the interventions being trialled, which might mean the effect is greater.

However there were two concerns raised at our Journal Club.

Might the observed effect be explained by Diagnosis Shifting, (in which high antibiotic prescribers disproportionately label a greater proportion of ARIs diagnoses as antibiotic-justifiable3)? The Authors declare diagnosis shifting was not evident, referring to eTable 6. However, transforming the coefficient estimates in eTables 3, 4A and 4B into odds ratios, we found a significant effect on the trajectory of ‘antibiotic appropriate’ and ‘antibiotic inappropriate’ diagnoses over time.

What explains the control group’s dramatic reduction of ‘antibiotic inappropriate’ prescriptions in the 18 months before the interventions commenced, which continued through the study randomization period? This was in the order of 10% in the pre-randomization period, and a further 10% in the control group during the randomization period – greater than for any of the interventions themselves, (taken from the slopes in Fig 2). This cannot be a Hawthorne effect because the data were collected for a time period before the clinicians were enrolled. It does not fit any nationwide trends. We speculate that it is explained by Diagnosis Shifting, perhaps by a misclassification by electronic health record coders. Until we understand this reduction, we can have no confidence in the smaller intervention effect.

Chris Del Mar MD professor of public health Paul Glasziou PhD professor of evidence based practice Elaine Beller MAppStat statistician On behalf of the Centre for Research in Evidence Based Practice Journal Club Bond University, Queensland 4229 Australia

1 Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: A randomized clinical trial. JAMA. 2016;315:562-70 2 Spurling GK, Del Mar CB, Dooley L, Foxlee R, Farley R. Delayed antibiotics for respiratory infections. Cochrane database of systematic reviews (Online). 2013;;CD004417:DOI: 10.1002/14651858.CD004417.pub3. 3 Howie JG, Richardson IM, Gill G, Durno D. Respiratory illness and antibiotic use in general practice. J R Coll Gen Pract. 1971;21:657-63.