Then Opus 4.7 shipped & the smarter model became much more expensive. The cause : a new tokenizer
大多数人认为AI模型变贵主要是因为能力提升，但作者揭示了一个反直觉的原因：更精确的分词器(tokenizer)导致需要处理更多token，从而使更智能的模型反而变得更贵。这挑战了'能力提升导致成本上升'的简单归因。
3 Matching Annotations
- Apr 2026
-
www.tomtunguz.com www.tomtunguz.com
-
-
www.claudecodecamp.com www.claudecodecamp.com
-
Code is hit harder than unique prose (1.29–1.39x vs 1.20x). Code has more repeated high-frequency strings — keywords, imports, identifiers — exactly the patterns a Byte-Pair Encoding trained on code would collapse into long merges.
这一发现挑战了我们对代码token化的常识认知。通常我们认为代码有更多重复模式应该更高效token化，但事实相反。这表明代码的语义复杂性超越了简单的重复模式，需要更细粒度的处理。这一反直觉结论对代码生成和代码理解模型的优化方向提出了新思考。
-
- Aug 2021
-
numinous.productions numinous.productions
-
Tools for thought are (mostly) public goods, and as a result are undersupplied:
-