A 606 MiB model at ~49 tokens/s consumes ~30 GB/s of memory bandwidth, close to the c6i.2xlarge's DRAM limit. No amount of SIMD tricks will help when the CPU is stalled waiting for model weights to arrive from DRAM.
这一数据揭示了现代CPU推理的关键瓶颈：内存带宽限制。代理最初尝试的SIMD微优化无法突破这一根本限制，这表明理解硬件特性和系统瓶颈对于有效优化至关重要。这一发现挑战了传统上认为计算是主要瓶颈的观念，强调了内存效率在AI推理中的核心地位。