• The Core Problem & Gap: Coding agents resend the full conversational context on every turn; during tight tool-execution loops, prompt prefix caching hits 90–99%, but cache retention plummets to ~18% after just an hour of human idle time.
• Two Distinct Modes of Cache Eviction:
  • Capacity Misses (Memory Full): Dominate inside rapid autonomous tool loops when GPU/host memory pools overflow under concurrent sessions; addressable via tiered memory offloading, cache eviction policies, and smarter request routing.
  • Retention Misses (Timeout Expirations): Occur during human-paced interruptions (reviewing diffs, eating lunch, answering prompts); require session-aware TTLs, cache pinning, client keepalives, or harness-level lifecycle hints.
• Benchmark Realism Gap: Standard evaluation suites (e.g., SWE-bench, TerminalBench) only simulate synthetic or non-human tool loops, needing merely ~5-minute cache retention to achieve near-optimal hit rates; in contrast, real human-in-the-loop workflows demand day-scale retention.
• Quantified Pause Dynamics (FreeInference & Production Data):
  • Tool calls experience a P90 gap of ~17.2 seconds with ~97% cache hits.
  • Subagent and "Ask User" queries experience ~6-minute P90 gaps (~91–92% cache hits).
  • Human idle intervals beyond 15–30 minutes drop hit rates to ~58%, falling to ~18% after 1 hour.
  • A 15-minute provider TTL forces ~60.9% additional prefill compute on human-driven sessions compared to unbounded cache retention.
• System-Level & Harness Inefficiencies:
  • Subagents frequently spawn without sharing parent prefix tokens, preventing KV reuse across threads.
  • Context compaction (condensing conversation history) and dynamic model-switching aggressively invalidate prefix matching.
  • Non-LLM execution (sandboxes, tool latency, user wait time) constitutes up to 72% of total session wall-clock time.

• The Role of an Inference Engine [00:02:16]:

  • Software layer operating between raw weights on disk and incoming HTTP user requests (e.g., vLLM, Ollama).
  • Performs five core responsibilities: loading model weights, maintaining context state, scheduling requests, executing matrix arithmetic, and serving an HTTP API.
  • Sits above compute kernels (low-level math operations) and below cluster orchestrators/control planes and API gateways.

• GPU Memory Dynamics (Weights vs. KV Cache) [00:05:16]:

  • Model Weights: Static memory cost determined by parameter count and precision (e.g., an 8B parameter model at 16-bit/2-byte precision occupies ~16 GB VRAM); remains fixed regardless of concurrent concurrency.
  • KV Cache: Dynamic working memory holding intermediate attention states across token generations; scales linearly with conversation depth and concurrent active users, directly competing with weights for remaining VRAM.

• Batching Strategies & Memory Management [00:07:39]:

  • Continuous Batching: Replaces naive static batching by evicting completed requests immediately and slotting new requests into the running batch, avoiding GPU idle cycles caused by differing generation lengths.
  • PagedAttention / Paging: Allocates KV cache in small, non-contiguous fixed-size blocks on demand rather than reserving maximum possible context windows per request, eliminating severe internal memory fragmentation.

• Weight Quantization [00:09:47]:

  • Reduces precision per parameter (e.g., from FP16 to INT8/INT4), slashing fixed model footprint.
  • The primary benefit is freeing VRAM for a larger KV cache pool, directly scaling concurrent user throughput on cost-effective GPUs.

• Prefix Caching [00:10:58]:

  • Caches identical token sequences across requests (e.g., static system instructions, tool definitions, recurring multi-turn histories).
  • Bypasses redundant prompt reprocessing, providing massive speed and efficiency gains for agentic workflows.

• Prefill vs. Decode & Speculative Decoding [00:12:28]:

  • Prefill Phase: Parallel ingestion of prompt tokens; strictly compute-bound as the GPU operates at peak arithmetic saturation.
  • Decode Phase: Autoregressive, strictly sequential token generation; heavily memory-bandwidth bound due to fetching full weights per generated token.
  • Speculative Decoding: Employs an ultra-fast draft model to predict token sequences validated by the target model in a single parallel pass, reducing end-to-end decode latency for individual users.

• Orchestration, Routing, and Model Scaling [00:14:46]:

  • Control Plane & Gateways: Standard round-robin load balancing fails for LLMs; gateways require cache-aware routing to send requests to replicas with pre-warmed KV caches.
  • Model Sharding: Distributes massive parameter architectures across multiple GPUs via Tensor Parallelism (intra-layer splitting) or Pipeline Parallelism (inter-layer pipelining).
  • LLM Autoscaling: Relies on inference-specific metrics (queue depth, KV cache utilization) rather than CPU/GPU utilization; hindered by substantial cold-start penalties when pulling images and warming weights.
  • Prefill/Decode Disaggregation: Decouples the prefill (compute-intensive) and decode (bandwidth-intensive) workloads across specialized hardware topologies for optimal hardware utilization.

This precisely quantifies why longer context windows are expensive beyond just model size: KV cache grows quadratically with context, and current GPU memory can't keep pace. Each eviction or recomputation directly inflates cost-per-token — making KV cache the hidden tax on long-context AI workloads.