We built an automated scanning agent that systematically audited eight among the most prominent AI agent benchmarks — SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench — and discovered that every single one can be exploited to achieve near-perfect scores without solving a single task.
令人惊讶的是：研究人员构建的自动化扫描工具发现，所有八个主流AI代理基准测试都存在漏洞，无需解决任何任务就能获得接近完美的分数。这表明整个AI评估领域存在系统性问题，几乎所有当前使用的基准测试都不可靠。