Instead, with IndexShare, the KV cache of $h_5$ includes only $kv_{1:4}$, all from the hidden states of the target model. For training, we reuse both kv cache and topk indices of the first mtp step.
大多数人认为在多步预测解码中，每一步都应该独立计算KV缓存以保持信息完整性，但作者认为通过共享索引可以消除训练-推理差异，提高接受率。这一反直觉的观点挑战了模型推理的最佳实践，表明在某些情况下，限制信息流动可能反而提高模型性能。
a lightweight surrogate trained on them can absorb a significant portion of future traffic at near-zero marginal inference cost
大多数人认为模型替换会带来明显的质量下降或需要持续监督。但作者提出轻量级代理模型可以'吸收大量未来流量'且'边际推理成本接近零'，这种近乎零成本的替代方式颠覆了传统模型替换的质量-成本权衡观念。
TPU 8i is designed with more memory bandwidth to serve the most latency-sensitive inference workloads, which is critical because interactions between agents at scale magnify even small inefficiencies.
通常认为内存带宽是通用硬件的需求，但作者提出TPU 8i针对低延迟推理进行了优化，这与通用硬件设计追求平衡的常规做法不同。