Tests reject correct solutions: We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions
大多数人认为代码测试是客观公正的，能够准确评估模型的真实能力。但作者发现，近60%的测试案例存在缺陷，会拒绝功能上正确的解决方案。这一发现挑战了AI评估领域的共识，表明我们广泛使用的基准测试可能存在系统性问题，无法准确反映模型的实际编程能力。
The most famous chart in AI might be obsolete soon.
副标题本身就是一个令人震惊的声明：最著名的 AI 进展图表即将过时——不是因为 AI 停止进步，而恰恰是因为进步太快。这创造了一个奇异的悖论：评测工具的失效速度与被评测对象的进步速度正相关。我们对 AI 能力的理解，正在以比 AI 自身进步更慢的速度迭代——「评测滞后」将成为未来数年 AI 治理和决策的核心挑战。
Chetty, R., Friedman, J. N., Hendren, N., Stepner, M., & Team, T. O. I. (2020). How Did COVID-19 and Stabilization Policies Affect Spending and Employment? A New Real-Time Economic Tracker Based on Private Sector Data (Working Paper No. 27431; Working Paper Series). National Bureau of Economic Research. https://doi.org/10.3386/w27431