Scaling Embeddings Outperforms Scaling Experts in Language Models
大多数人认为在MoE模型中增加专家数量是提升性能的最佳策略，但这篇论文提出扩展嵌入维度比扩展专家数量更有效。这一观点与主流MoE扩展思路相悖，暗示了模型设计的根本性转变。
13 Matching Annotations
- Last 7 days
-
magazine.sebastianraschka.com magazine.sebastianraschka.com
-
- Apr 2026
-
developer.nvidia.com developer.nvidia.com
-
For higher-interactivity scenarios, execution time for MoE models is bound by expert weight load time. By splitting, or sharding, the experts across multiple GPUs across NVL72 nodes, this bottleneck is reduced, improving end-to-end performance.
大多数人认为MoE模型的主要瓶颈在于计算能力，但作者指出专家权重加载时间是真正的瓶颈，并提出通过跨GPU分片专家权重来解决问题，这挑战了AI模型优化的传统认知，暗示了I/O可能比计算更重要。
-
- Jan 2024
-
github.com github.com
-
[Yomitan - The Moe Way]
site:: [[GitHub]] user:: themoeway url:: https://github.com/themoeway/yomitan accessed:: 2024-01-04
-
- Oct 2019
-
pieche.pixnet.net pieche.pixnet.net
-
gkingdom.pixnet.net gkingdom.pixnet.net
-
yanning0903.pixnet.net yanning0903.pixnet.net
-
熱騰騰
Tags
Annotators
URL
-
-
wennylee.pixnet.net wennylee.pixnet.net
-
halle0310.pixnet.net halle0310.pixnet.net
-
xiangyang17.pixnet.net xiangyang17.pixnet.net
-
uscake.pixnet.net uscake.pixnet.net
- Jun 2019
-
-
It is backed by a reserve of assets designed to give it intrinsic value;
Libra is a basket of things which we can question their intrinsic values also. Libra will re-trigger "Intrinsic Value of MoE" discussions.
-
- Jul 2016
-
medium.com medium.com
-
Massive Open Online Courses (MOOCs) have been the subject of much hyperbole in the educational/eLearning world for a few years now, under the guise of spreading university-quality education to the masses for free (the hyperbole is dwindling down, but not completely).
Cue Rolin Moe, who has investigated the MOOC hype so thoroughly. We may still follow a Gartner Cycle (Merton did warn us about self-fulfilling prophesies). But much of those phases have been documented.
-