Top models scored around 40%.

The next generation of benchmarks needs to be harder, more realistic, and less gameable

MMLU, GSM8K, and HumanEval are now saturated

SWE-Bench Verified score of 81.8 compared to Opus 4.6 (80.8) and Deepseek 4.0 Pro (80.0).

Research result of 83 and a production model, third-party verified score of 65.9, SubQ 1M-Preview compares favorably with other SOTA models like Claude Opus 4.7 (32.2), GPT 5.5 (74), and Gemini 3.1 Pro (26.3).

SubQ 1M-Preview scores 95% accuracy, compared to 94.8% for Claude Opus 4.6

The benchmark tasks were meticulously constructed to be realistic, involving the hard work of hundreds of experts and likely millions of dollars — placing it among the most expensive economics papers of all time.

ARC-AGI-3 was officially released this week. All frontier models score below 0.5%

DeepSeek-V4-Pro-Max beats GPT-5.2 and Gemini 3.0-Pro on standard reasoning benchmarks and lands just behind GPT-5.4 and Gemini 3.1-Pro

🔹 **Enhanced Agentic Capabilities:** Open-source SOTA in Agentic Coding benchmarks.

GPQAD | 94.4 | 90.9 | 92.7 | 92.4 | **95.1** | LCBv6 | 90.3 | 92.1 | 92.4 | 90.4 | **93.2** | SWEPro | 48.4 | 51.2 | _53.4_ | 51.3 | **54.2**

This means that improvements on SWE-bench Verified no longer reflect meaningful improvements in models' real-world software development abilities. Instead, they increasingly reflect how much the model was exposed to the benchmark at training time.

Tests reject correct solutions: We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions

With only 27B parameters, it outperforms the Qwen3.5-397B-A17B (397B total / 17B active) on every major coding benchmark

Multiple community tests show llama.cpp running 1.8x faster than Ollama on the same hardware with the same model, 161 tokens per second versus 89.

On the SWE-Pro benchmark, M2.7 scores 56.22%, nearly matching Opus's best level.

On GDPval-AA, M2.7 achieves an ELO score of 1495, the highest among open-source models.

On GDPval-AA, M2.7 achieves an ELO score of 1495, the highest among open-source models.

The 66.6% medal rate on MLE Bench Lite, achieved autonomously over 24 hour windows, tells you something real about how this model behaves when you give it a hard problem and step back.

We improved Claude's RCA accuracy by 12pp on OpenRCA

A conftest.py file with 10 lines of Python 'resolves' every instance on SWE-bench Verified.

This benchmark is a six-part semantic scoring test that assesses any model's effectiveness at relevant calibration tasks. QCalEval measures a model's ability to interpret experimental results, classify outcomes, evaluate their significance, assess fit quality and key features, and generate actionable next-step recommendations.

The model reportedly scored 93.9% on SWE-bench Verified and 77.8% on SWE-bench Pro, but its strongest signal came from real-world results, including uncovering a 27-year-old flaw in OpenBSD, a 16-year-old vulnerability in FFmpeg, and autonomously chaining Linux kernel exploits without human input.

We evaluate 452 tasks from the public APEX-Agents dataset spanning investment banking, management consulting, and corporate law

APEX-Agents requires agents to navigate realistic work environments with files and tools.

Gemini 3 Flash achieves the highest score of 24.0%

GPT-5.4 (xhigh) scores the highest on APEX-Agents-AA Pass@1 with a score of 33.3%, followed by Claude Opus 4.6 (Adaptive Reasoning, Max Effort) with a score of 33.0%, and Gemini 3.1 Pro Preview with a score of 32.0%

we may see a growing divergence between the capabilities we can measure and the capabilities we actually care about.

The most famous chart in AI might be obsolete soon.

it's impossible to get a score much higher than 93% without cheating because around 6.5% of MMLU questions contain errors.

If we took one task out of our task suite or added another task to our task suite, potentially instead of measuring this Claude Opus 4.6 time horizon of, I think, 14 and a half hours, we'd be measuring it at something like eight or 20 hours.

reasoning models tend to produce much shorter reasoning traces (up to 50%) for the same problem under different context conditions compared to the traces produced when the problem is presented in isolation.

we found that AI agent performance drops substantially when scoring AI performance holistically rather than algorithmically.

Because these benchmarks are human-authored, they can only test for risks we have already conceptualized and learned to measure.

the trained 4B model exceeding GPT-4.1 (49.4 percent) and GPT-4o (42.8 percent) despite being 50 times smaller

current systems remain highly vulnerable

NVIDIA yields unmatched inference throughput across the broadest range of workloads, from massive LLMs to advanced vision language models, to generative recommender systems and more, on industry-standard benchmarks.

NVIDIA was the first and only platform to submit DeepSeek-R1 results on MLPerf Inference when the benchmark debuted last year.

n July reasoning models from both OpenAI and Google Gemini achieved gold medal performance in the International Math Olympiad, a prestigious mathematical competition held annually (bar 1980) since 1959. This was notable because the IMO poses challenges that are designed specifically for that competition. There’s no chance any of these were already in the training data! It’s also notable because neither of the models had access to tools—their solutions were generated purely from their internal knowledge and token-based reasoning capabilities.

the inference efficiency improved by nearly three orders of magnitude or 1,000x in less than 2 years

there is essentially this Benchmark 00:09:58 called the math benchmark a set of difficult mathematic problems from a high school math competitions and when the Benchmark was released in 2021 gpt3 only got 5%

Beyond just audio recordings so for that reason two of our senior 00:15:02 researchers Benjamin Hoffman and Maddie cusumano have also developed a biologer benchmark data set and so a biologer is an animal born tag like the one in the image on the right here 00:15:14 and these produce very valuable data because they can inform us about animal ecophysiology and allow us to improve conservation by monitoring animal movements and behaviors with very high 00:15:27 resolution

beans and 00:13:54 this is a benchmark of animal sounds and it's a collection of audio recordings from more than 250 species and this large aggregate data set is a way to 00:14:07 test tools for classification and detection and these are outstanding problems in bioacoustics that we desperately need solutions to

this is AVS the very first Foundation model for animal communication

java循环长度的相同、循环体代码相同的两次for循环的执行时间相差了100倍?

targeting one of three TSH ranges (0.34 to 2.50, 2.51 to 5.60, or 5.61 to 12.0 mU/L)