Z.ai further reduced the LLM’s hardware footprint using a method called linear attention. Usually, doubling the size of a prompt quadruples the amount of memory that a model’s attention mechanism consumes.
本周唯一一条正面证据：线性注意力确实把 KV 内存从平方降到线性。但它只作用于长上下文的 KV 缓存，权重本身照样要进 HBM。所以"架构创新会削弱 HBM 依赖"依然没被证实，最多算延缓。
linear attention mech-anism
Softmax Function - converts vector of predictions into probabilities for each class. https://www.geeksforgeeks.org/deep-learning/the-role-of-softmax-in-neural-networks-detailed-explanation-and-applications/
Linear Attention = approximation of softmax (by using linear dot product of kernel feature maps to convert each step into addition for the update equations). https://linear-transformers.com/ https://haileyschoelkopf.github.io/blog/2024/linear-attn/
Also note that flash attention (a newer, better in terms of memory approach), is briefly mentioned at the end of this paper, and discussed more in the sequel paper.