This process is memory-bound, which means it is limited by how fast the hardware can stream model weights and cache data from memory. By default (i.e., with no optimization), this usually leads to low utilization, with a GPU generally running at 30-60% of its total capacity in terms of power draw and memory utilization

the lack of KV sharing across requests leads to redundant prefill computation and wasted memory.