This suggests that the models can identify which dataset an audio sample comes from and select the spelling convention that benchmark expects, even though both forms sound identical.
最关键的一层不是模型记住了答案，而是它先认出「我正在被哪个基准测试」，再切换成那套书写规范。这意味着污染检测不能只查文本重合，还得防声学指纹；同域但训练截止之后新采的音频一喂进去，这种行为大多消失，正好反证了它的存在。
On LibriSpeech, some of the strongest benchmark-performing models reproduced masked numbers in roughly 30–40% of examples, even though the number itself had been removed.
把数字从音频里静音掉，模型照样把原文的数字填回来，这是本文最干净的证伪设计——正确答案在声学上根本不存在，答对只可能来自记忆。这种「构造一个不可能答对的题」的思路，比事后统计污染率有力得多，值得搬到其他模态的评测里。
The models with the lowest WER—and therefore the strongest reported benchmark performance—are also the most likely to reproduce these errors.
这条相关性把「刷榜」从直觉变成了可量化指标：榜首模型恰恰最爱照抄参考文本里的错误，说明低 WER 里有一部分是背题得分。以后看 ASR 排行榜，第一名和第五名的差距可能不是听得更准，而是更认得出这是哪个数据集。
Trials measure efficacy, but the world cares about effectiveness. Today, benchmarks lack even a good measure of efficacy and are far away from effectiveness.
把临床试验里 efficacy（理想条件有效）和 effectiveness（真实世界有用）的区分搬到评测上，是本文最可外推的一层。当前榜单连第一层都没做扎实就在谈落地，等于跳过了医学花几十年才走完的路。注意作者是 Protege 的合作方，指出问题的同时也在卖解法。
The model saw the same case and the same 19 labels; only their positions changed. Yet models frequently changed answers!
只换选项顺序、病例和标签一字不动，模型答案就变，说明分数里混着大量与医学无关的形式噪声。这条可以直接迁移到任何榜单：看到排名差几个百分点，先问是不是换个 prompt、换个选项顺序就翻盘，别急着当能力差距读。
patient characteristics, comorbidities, facilities, and year only explain 3.4% of the variation in the choice to perform partial or full
这是全文最硬的一组数字：加进主刀医生身份，解释力从 3.4% 跳到 14.8%，也就是七成以上的差异来自「谁开的刀」。它把「基准答案」这件事拆穿了——医疗标注很多时候记录的是某个医生当天的偏好，模型答得跟它不一样，未必是错，只是不合口味。
Naively, you might pick a single set of benchmarks to measure all historical models, but this is a mistake.
方法论上最值得借鉴的一句：把 LLM 史切成三个时代、各用当代基准分别打分，避开了饱和基准把差距压平的陷阱。代价是三段分数彼此不可比，所以"追赶期减半"是趋势描述，不是能往外推的定律。
Z.ai compared GLM-5.3-Flash against Claude Opus 4.8, GPT-5.6 Terra and Gemini 3.7 Flash. The former model achieved the highest score on GDPval-AA v2, an evaluation that measures LLMs’ ability to perform knowledge work.
对照组混搭了各家不同档位（Flash 级与 Terra 级），是同价位比而非同能力比，榜首要打折看。真正可信的信号是十倍成本下降这个自比数据，而不是跨厂商名次。这与本期 Jalapeño 挑对照组的毛病同源。
An orchard in week one is not the same orchard in week twelve — season, weather, and pest population all shift. Comparing scores across time is therefore hard, sometimes impossible.
读物理评测的排行榜时该先警惕这一条。非平稳性意味着跨时间的分数不可比，再加上前一个参赛者留下的物理痕迹（sequential contamination）和网络延迟差异，「谁更强」的结论可能主要由排期运气决定。这也解释了为什么传统 benchmark 那套排行榜语义没法原样搬进现实世界。
which uses billing data from 70,000 Ramp credit card using companies to estimate model adoption.
用企业信用卡账单反推模型采用率，比厂商自报口径更难粉饰，这是难得的第三方数据。但要记住它测的是"谁在刷卡"而不是"谁在跑推理"：样本偏向美国中小企业，大厂的私有合约和直签 API 完全不在里面，用它做份额结论会系统性低估头部客户。
On Terminal-Bench 2.1 (81.0) it lands within a few points of Claude Opus 4.8 (85.0) — while staying ahead of Gemini 3.1 Pro.
大多数人认为开源模型与顶级闭源模型之间存在巨大差距，但作者认为GLM-5.2在终端基准测试中已经接近Claude Opus 4.8的性能，甚至超过了Gemini 3.1 Pro。这一观点挑战了AI领域'闭源模型遥遥领先'的行业共识，表明开源模型在特定编码任务上已经能够与顶级商业模型竞争。
The three metrics where we find acceleration are concentrated in programming and mathematics. These are areas that labs have explicitly targeted for improvement, and they share an important property: correctness is easy to verify automatically.
主流观点可能认为AI能力在各个领域的提升是均衡的，但作者指出加速现象主要集中在编程和数学领域，因为这些领域的正确性容易自动验证。这暗示AI进步可能不是普遍性的，而是集中在特定可量化的领域。
While model capabilities have improved dramatically for use cases like codegen and mathematical reasoning, they still lag behind on the data side (as evidenced through SQL benchmarks like Spider 2.0 and Bird Bench).
这一观点提供了令人惊讶的事实：尽管模型在代码生成和数学推理方面取得了显著进步，但在数据处理方面仍然落后。这挑战了模型能力全面提升的假设，暗示了数据推理可能需要特殊的处理方法。
While model capabilities have improved dramatically for use cases like codegen and mathematical reasoning, they still lag behind on the data side (as evidenced through SQL benchmarks like Spider 2.0 and Bird Bench).
令人惊讶的是：尽管AI模型在代码生成和数学推理方面取得了巨大进步，但在数据处理方面仍然落后。Spider 2.0和Bird Bench等基准测试显示，AI在SQL查询等基础数据任务上表现不佳，这表明当前AI技术存在明显的应用局限性。
Gemma 4 E4B matches or exceeds GPT-4o across multiple benchmarks including MATH, GSM8K, GPQA Diamond & HumanEval.
令人惊讶的是：Google的Gemma 4 E4B作为免费模型竟然在多个基准测试中超越了或匹敌了GPT-4o这一业界领先的商业模型。这表明开源和免费AI模型的质量已经达到了商业级别，打破了AI领域由少数大公司垄断的格局。
or instance, datasets such as AAAR-1.0[ 61], ScienceAgentBench [11 ], and TaskBench [ 83 ] provide struc-tured, expert-labeled benchmarks for assessing research reasoning,scientific workflows, and multi-tool planning. Others, such as Flow-Bench [96 ], ToolBench [38 ], and API-Bank [ 47 ], focus on tool useand function-calling across large API repositories. These bench-marks typically include not only the gold tool sequences but alsoexpected parameter structures, enabling fine-grained evaluation.In parallel, datasets like AssistantBench [ 109], AppWorld [91 ],and WebArena [ 126] simulate more open-ended and interactiveagent behaviors in web and application environments. They empha-size dynamic decision-making, long-horizon planning, and user-agent interactions. Several benchmarks also support safety androbustness testing—for example, AgentHarm [5 ] assesses poten-tially harmful behaviors, while AgentDojo [ 17 ] evaluates resilienceagainst prompt injection attacks. Leaderboards such as the Berke-ley Function-Calling Leaderboard (BFCL) [ 100] and Holistic AgentLeaderboard [ 88 ] consolidate these evaluations by
bench marking
While the initial results fall short, the AI field has a history of blowing through challenging benchmarks. Now that the APEX-Agents test is public, it’s an open challenge for AI labs that believe they can do better — something Foody fully expects in the months to come.
expectation that models will get trained against the tests they currently fail.
LLM benchmarks are essential for tracking progress and ensuring safety in AI, but most benchmarks don't measure what matters.
Paper concludes most benchmarks used for LLMs to establish progress are mistargeted / leave out aspects that matter.
PDF paper Saved Measuring what Matters: Construct Validity in Large Language Model Benchmarks in Zotero Paper for NeurIPS 2025 conference https://neurips.cc/Conferences/2025
let’s do some measurements using Audit tool of Google Chrome
the benchmarks that Rich chose weren't even remotely good ones. They had obvious flaws that even the authors acknowledge and Svelte's implementation actually cheats what it was testing.
compared the speed of DeviceDetector with the two most popular user agent parsers in the Ruby community, Browser and UserAgent.