We also found evidence that models that have seen the problems during training are more likely to succeed, because they have additional information needed to pass the underspecified tests.
大多数人认为AI模型的性能提升主要源于算法和架构的改进。但作者发现，模型在SWE-bench上的成功更多取决于它们是否在训练中见过这些问题，而非真正的编程能力提升。这一观点与行业普遍认为的'模型进步'叙事相悖，暗示当前AI发展评估可能存在严重偏差。
Sulik, J., & McKay, R. (2021). Studying science denial with a complex problem-solving task [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/huxm7
Pleskac, T. J., Kyung, E., Chapman, G. B., & Urminsky, O. (2021, April 23). Single- or double-blind review? A field study of system preference, reliability, bias, and validity. https://doi.org/10.31234/osf.io/q2tkw
Kamorowski, J., de Ruiter, C., Schreuder, M., Ask, K., & Jelicic, M. (2020, April 16). Forensic Mental Health Practitioners’ Use of Structured Risk Assessment Instruments, Views About Bias in Risk Evaluations, and Strategies to Counteract It. https://doi.org/10.31234/osf.io/te5c2