The model saw the same case and the same 19 labels; only their positions changed. Yet models frequently changed answers!
只换选项顺序、病例和标签一字不动，模型答案就变，说明分数里混着大量与医学无关的形式噪声。这条可以直接迁移到任何榜单：看到排名差几个百分点，先问是不是换个 prompt、换个选项顺序就翻盘，别急着当能力差距读。
6 Matching Annotations
- Aug 2026
-
-
- May 2026
-
-
Greater precision and control
该表述可能带有偏见，需要了解“Greater precision and control”是如何实现的，以及用户对此的评价。
-
- Apr 2026
-
-
We also found evidence that models that have seen the problems during training are more likely to succeed, because they have additional information needed to pass the underspecified tests.
大多数人认为AI模型的性能提升主要源于算法和架构的改进。但作者发现，模型在SWE-bench上的成功更多取决于它们是否在训练中见过这些问题，而非真正的编程能力提升。这一观点与行业普遍认为的'模型进步'叙事相悖，暗示当前AI发展评估可能存在严重偏差。
-
- Aug 2021
-
-
Sulik, J., & McKay, R. (2021). Studying science denial with a complex problem-solving task [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/huxm7
-
- Apr 2021
-
psyarxiv.com psyarxiv.com
-
Pleskac, T. J., Kyung, E., Chapman, G. B., & Urminsky, O. (2021, April 23). Single- or double-blind review? A field study of system preference, reliability, bias, and validity. https://doi.org/10.31234/osf.io/q2tkw
-
- Apr 2020
-
psyarxiv.com psyarxiv.com
-
Kamorowski, J., de Ruiter, C., Schreuder, M., Ask, K., & Jelicic, M. (2020, April 16). Forensic Mental Health Practitioners’ Use of Structured Risk Assessment Instruments, Views About Bias in Risk Evaluations, and Strategies to Counteract It. https://doi.org/10.31234/osf.io/te5c2
-