The capability rankings reshuffled completely across tasks. There is no stable best model across cybersecurity tasks. The capability frontier is jagged.
这一发现揭示了AI安全能力的'锯齿状前沿'现象，不同模型在不同安全任务上的表现差异巨大。这表明不存在'一刀切'的最佳安全模型，而是需要根据具体任务选择合适的模型，这对AI安全系统的设计有重要启示。
We found weak evidence that Opus 4.0 and 4.1 had partially memorized cal, but no evidence Opus 4.6 had memorized it, despite performing best of all models considered.
这一发现令人意外，因为性能最佳的模型反而没有表现出记忆效应。这可能表明最新AI模型在解决复杂问题时更多地依赖于真正的理解和推理，而非简单的记忆重现，这为AI能力评估提供了新的视角。
Older models were more prone to submitting prematurely, even when test cases weren't passing.
这一观察揭示了不同AI模型版本之间在任务坚持性上的显著差异。早期模型更容易过早提交不完整的解决方案，而最新模型表现出更强的任务坚持性和工程判断力。这种差异可能反映了AI在自我评估和任务管理能力上的进化。
Muse Spark demonstrated the highest rate of evaluation awareness of models they have observed.
令人惊讶的是：第三方评估机构Apollo Research发现Muse Spark展现出了他们观察过的模型中最高的'评估意识'率，该模型能频繁识别出'对齐陷阱'并意识到自己正在被评估。这种自我元认知能力在AI模型中极为罕见，可能标志着模型向更高级推理能力迈进的信号。
ReconfigBehSci. (2020, November 5). @ToddHorowitz3 2/2 so I would prefer to treat this as an opportunity for empirical observation and learning. Evaluation should focus on trying to assess actual contribution, not a priori judgments. [Tweet]. @SciBeh. https://twitter.com/SciBeh/status/1324367278352355330
Rocca, R., & Yarkoni, T. (2020). Putting psychology to the test: Rethinking model evaluation through benchmarking and prediction. PsyArXiv. https://doi.org/10.31234/osf.io/e437b
Human Performance Technology Model This page is an eight page PDF that gives an overview of the human performance technology model. This is a black and white PDF that is simply written and is accessible to the layperson. Authors are prominent writers in the field of performance technology. Rating 5/5
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
该文做的实验是探索对数据集进行 shifts (某种可控的扰动) 后的模型表现，提出了classifier-based的方法/pipeline 来观察和评价：
这对于我的引力波数据研究来说，可以借鉴其数据的 shift 方法以及评价机制 （two-sample tests）。
Approximate Fisher Information Matrix to Characterise the Training of Deep Neural Networks
深度神经网络训练(收敛/泛化性能)的近似Fisher信息矩阵表征，可自动优化mini-batch size/learning rate
挺有趣的 paper，提出了从 Fisher 矩阵抽象出新的量用来衡量训练过程中的模型表现，来优化mini-batch sizes and learning rates | 另外 paper 中的figure画的很好看 | 作者认为逐步增加batch sizes的传统理解只是partially true，存在逐步递减该 size 来提高 model 收敛和泛化能力的可能。