Figure 3:

0.327 for antibiotic A and 0.326 for antibiotic B

(all patient attributes are observable)

First, we increase patient volume to 10 patients per timestep.

This creates both between-subpopulation differences (high-risk vs. low-risk mean parameter values) and within-subpopulation variation (σ=0.1 for all attributes)

mixed low-risk and high-risk heterogeneous populations as ground truth

Accurate: true risk levels correctly observed,

optimal policies should exploit.

where infection probability followed Gaussian distributions

accumulated

antibiograms

constant and homogeneous

Experiment Set 1

constant values and homogenous

.

differs in

patient information and visible AMR data, but do not update behavior through experience

.

variance penalty

Training Phase:

library of 9 heuristic workers

rule-based policies

high-risk

The community reward, on the other hand, is based on the observed AMR levels of all antibiotics

probability that a new infection at the current time is resistant to the given antibiotic.

eventually observed

λ∈[0,1], allowing the user to control the relative importance of individual versus community objectives. In all experiments reported in this manuscript, we fixed

long-horizon environment dynamics and policy architecture without explicit AMR reward shaping.

whether a patient is infected

probabilistic

internal latent ‘pressure’

At

The agent informs the environment that it decided to prescribe Antibiotic B for the current patient

prescribing decision

observations

Overall, our experiments demonstrate that hierarchical RL agents are capable of learning effective and farsighted antibiotic prescribing policies, and are robust against multiple types of information degradation

partial observability

however, the resistance profile of a potential infection is controlled by the array of AMR_LeakyBalloon models in the environment, and the resistance profile determines the likelihood that a given antibiotic treatment will be effective.

clinical benefit and failure multipliers

Figure 1:

For our RL agents, we used PPO (proximal policy optimization) implementations from the Python package stable-baselines3. The specific agent architectures used — flat memoryless, flat recurrent, hierarchical memoryless, and hierarchical recurrent — are described in Section 3.3.

A fixed prescribing rule (see Section 3.5), which emulates how real-world prescribers make antibiotic treatment decisions using observable patient and resistance information Reinforcement learning (RL) agents (see Section 3.3), which learn policies by interacting with the environment over many training episodes, using feedback from the reward signal to discover policies that maximize cumulative long-term reward. Unlike the fixed rule, RL agents are not pre-programmed with a specific decision logic; instead, they discover effective prescribing strategies through trial-and-error exploration.

(Lee 2026) and companion preprint (Lee and Blumberg 2026)

These experiments

By comparing the performance of the RL algorithms against the fixed prescribing rule baseline, we demonstrate that our simulator enables quantitative comparative policy analysis, allowing us to characterize the magnitude and nature of gains achievable by adaptive prescribing strategies relative to static prescribing rules.

We note here that this application of simulation and reinforcement learning to the area of antimicrobial resistance is relatively novel.

We then compared the performance of two types of prescribing algorithms in these scenarios: a ‘baseline’ fixed prescribing rule that emulates how real-world prescribers make decisions about antibiotic treatment, and reinforcement learning (RL) algorithms. Because the consequences of prescribing decisions unfold over time — treatment decisions made today can drive resistance that undermines antibiotic efficacy months later — antibiotic prescribing is fundamentally a sequential decision-making problem. Reinforcement learning is a subfield of machine learning focused on designing algorithms to optimize sequences of decisions in order to maximize cumulative long-term outcomes, making it a useful approach for this problem (see Section 3.1 for further details).

A detailed description of the simulator’s architecture and functionality is provided in a separate manuscript (Lee and Blumberg 2026).

Antibiotic Stewardship

Given these challenges, it is difficult to directly quantify the long-term effects of antibiotic stewardship program interventions using observational or interventional studies alone (Bertollo et al. 2018; Schweitzer et al. 2019). When real-world systems lack sufficient data to support direct modeling, simulation can offer a complementary framework in which researchers are able to directly and explicitly specify the underlying ground truth of key mechanisms, enabling controlled investigation of trade-offs that are difficult or impossible to observe in real-world settings.

A fundamental challenge to performing quantitative impact evaluation of these programs is the pervasive partial observability of key system components (Laxminarayan et al. 2013). Even in resource-rich settings where antibiotic prescription records are available, true selection pressure on pathogen populations is incompletely observed due to unmeasured sources of antibiotic exposure, including agricultural use and environmental contamination (Van Boeckel et al. 2015). Measurement of AMR itself presents an even greater challenge: while initiatives such as the WHO’s Global Antimicrobial Resistance and Use Surveillance System (GLASS) have sought to standardize surveillance, participation remains voluntary and coverage incomplete, and no universally adopted scalar metric of resistance at the community level yet exists (Organization 2022; Leth and Schultsz 2023).

to saturation

dring

hierarchical RL agents outperformed flat RL agents

evaluating the impact of stewardship strategie

benchmark prescribing policies discovered by reinforcement learning agents against a clinically-realistic fixed prescribing rule across four sets of experiments of increasing complexity, where each set of experiments varied in type and degree of observed information degradation.

Optimizing Prescribing Policies Under Antimicrobial Resistance Dynamics