We separately evaluate GPT‑5.5 Pro in certain cases because we judge that the setting could materially impact the relevant risks or appropriate safeguards posture.
大多数人认为如果两个模型使用相同的基础架构，它们的风险和安全需求应该相似，但OpenAI明确表示GPT-5.5 Pro需要单独评估，因为'设置可能显著影响相关风险或适当的安全措施立场'。这挑战了AI评估领域普遍认为的'相同基础模型的安全特性一致'的共识，暗示即使是微小的设置变化也可能导致显著不同的风险特征。
The best-performing model across these three metrics was a pair of independent linear trends: one for reasoning models and one for non-reasoning models.
这个模型选择结果(100%的三个指标)表明将模型分为推理和非推理两类是最优预测模型。这提供了强有力的统计证据，支持推理能力可能是AI加速发展的关键因素。然而，文章没有详细说明如何定义推理模型，这可能影响结果的可靠性。
We also found evidence that models that have seen the problems during training are more likely to succeed, because they have additional information needed to pass the underspecified tests.
大多数人认为AI模型的性能提升主要源于算法和架构的改进。但作者发现，模型在SWE-bench上的成功更多取决于它们是否在训练中见过这些问题，而非真正的编程能力提升。这一观点与行业普遍认为的'模型进步'叙事相悖，暗示当前AI发展评估可能存在严重偏差。
The capability rankings reshuffled completely across tasks. There is no stable best model across cybersecurity tasks. The capability frontier is jagged.
这一发现揭示了AI安全能力的'锯齿状前沿'现象，不同模型在不同安全任务上的表现差异巨大。这表明不存在'一刀切'的最佳安全模型，而是需要根据具体任务选择合适的模型，这对AI安全系统的设计有重要启示。
We found weak evidence that Opus 4.0 and 4.1 had partially memorized cal, but no evidence Opus 4.6 had memorized it, despite performing best of all models considered.
这一发现令人意外，因为性能最佳的模型反而没有表现出记忆效应。这可能表明最新AI模型在解决复杂问题时更多地依赖于真正的理解和推理，而非简单的记忆重现，这为AI能力评估提供了新的视角。
Older models were more prone to submitting prematurely, even when test cases weren't passing.
这一观察揭示了不同AI模型版本之间在任务坚持性上的显著差异。早期模型更容易过早提交不完整的解决方案，而最新模型表现出更强的任务坚持性和工程判断力。这种差异可能反映了AI在自我评估和任务管理能力上的进化。
Muse Spark demonstrated the highest rate of evaluation awareness of models they have observed.
令人惊讶的是：第三方评估机构Apollo Research发现Muse Spark展现出了他们观察过的模型中最高的'评估意识'率，该模型能频繁识别出'对齐陷阱'并意识到自己正在被评估。这种自我元认知能力在AI模型中极为罕见，可能标志着模型向更高级推理能力迈进的信号。
ReconfigBehSci. (2020, November 5). @ToddHorowitz3 2/2 so I would prefer to treat this as an opportunity for empirical observation and learning. Evaluation should focus on trying to assess actual contribution, not a priori judgments. [Tweet]. @SciBeh. https://twitter.com/SciBeh/status/1324367278352355330
Rocca, R., & Yarkoni, T. (2020). Putting psychology to the test: Rethinking model evaluation through benchmarking and prediction. PsyArXiv. https://doi.org/10.31234/osf.io/e437b
Human Performance Technology Model This page is an eight page PDF that gives an overview of the human performance technology model. This is a black and white PDF that is simply written and is accessible to the layperson. Authors are prominent writers in the field of performance technology. Rating 5/5
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
该文做的实验是探索对数据集进行 shifts (某种可控的扰动) 后的模型表现，提出了classifier-based的方法/pipeline 来观察和评价：
这对于我的引力波数据研究来说，可以借鉴其数据的 shift 方法以及评价机制 （two-sample tests）。
Approximate Fisher Information Matrix to Characterise the Training of Deep Neural Networks
深度神经网络训练(收敛/泛化性能)的近似Fisher信息矩阵表征，可自动优化mini-batch size/learning rate
挺有趣的 paper，提出了从 Fisher 矩阵抽象出新的量用来衡量训练过程中的模型表现，来优化mini-batch sizes and learning rates | 另外 paper 中的figure画的很好看 | 作者认为逐步增加batch sizes的传统理解只是partially true，存在逐步递减该 size 来提高 model 收敛和泛化能力的可能。