occasionally even identifying the benchmark

meaning safety benchmarks may not reflect real-world behavior

Models sometimes recognize they're being evaluated, occasionally even identifying the benchmark.

In contrast, NLA explanations indicate evaluation awareness on less than 1% of real claude.ai usage that opted in for training.

We plan to release new evaluations every 1–2 months.

Whatever is precise enough to benchmark is also precise enough to optimize for.

But experts have warned that these problems are an imperfect benchmark of artificial intelligence's mathematical prowess. They range dramatically in both significance and difficulty, and many AI solutions have turned out to be less original than they appeared.

Help lay the game and environment foundations for ARC-AGI-4 and ARC-AGI-5

The central question is not whether AI can imitate human conversation, but whether it can participate in the production of publishable scientific knowledge at a level comparable to a recognized human contributor.

The results demonstrate consistent improvements over strong baselines, supporting the effectiveness of agent resource management and closed loop self evolution.

We have also provided access to GPT-5.4-Cyber to the U.S. Center for AI Standards and Innovation (CAISI) and the UK AI Security Institute (UK AISI) so that they can conduct evaluations focused on the model's cyber capabilities and safeguards.

未来的评估体系，必须同时考虑：成功率、成本、延迟。这有点类似于对于云计算的考核标准，而不是传统软件。

The model frequently identified scenarios as 'alignment traps' and reasoned that it should behave honestly because it was being evaluated.

Muse Spark demonstrated the highest rate of evaluation awareness of models they have observed.

We found weak evidence that Opus 4.0 and 4.1 had partially memorized cal, but no evidence Opus 4.6 had memorized it, despite performing best of all models considered.

Older models were more prone to submitting prematurely, even when test cases weren't passing.

The standard AI judges use to define "safe" are measured wrong. They punish action. They ignore inaction.

The top names you should know as a baseline, adjusted for 'what people are actually recommending'

Out of all generated PoCs, 759 triggered crashes across 60 projects, and manual inspection confirmed 17 cases of incomplete patches spanning 15 projects

We built an automated scanning agent that systematically audited eight among the most prominent AI agent benchmarks — SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench — and discovered that every single one can be exploited to achieve near-perfect scores without solving a single task.

Add benchmark framework and release submission overview - Add benchmark runner with onlineMind2Web benchmark support - Add agent client abstraction for codex/claude backends - Add CLI entry point for running benchmarks (pnpm benchmark)

This benchmark is a six-part semantic scoring test that assesses any model's effectiveness at relevant calibration tasks. QCalEval measures a model's ability to interpret experimental results, classify outcomes, evaluate their significance, assess fit quality and key features, and generate actionable next-step recommendations.

Some recent models that don't currently have time horizons: Gemini 3.1 Pro, GPT-5.2-Codex, Grok 4.1

We evaluate on ODinW-13 following consistent protocols from prior work. ODinW (Open Detection in the Wild) measures open vocabulary dense detection, testing fine-grained visual reasoning.

Agentic AI is increasingly judged not by fluent output alone but by whether it can act, remember, and verify under partial observability, delay, and strategic observation.

Agentic AI is increasingly judged not by fluent output alone but by whether it can act, remember, and verify under partial observability, delay, and strategic observation.

Consequently, they cannot verify if tools were actually invoked, applied correctly, or used efficiently.

However, existing evaluations fall short: they lack flexible tool integration, test visual and search tools separately, and evaluate primarily by final answers.