9 Matching Annotations
  1. Last 7 days
    1. One thing worth mentioning is that these request distributions will look different depending on which harness is being used, since different amounts/types of context are injected

      harness 影响的不止跑分,还一路传到硬件账单:注入多少上下文、怎么切子代理,直接决定 KV cache 复用率和推理成本结构。也要看到代价——这份数据集全部是 Claude Code 轨迹,代表性靠“世界上很大一部分 agent 编码流量走 Claude Code”这个假设撑着,harness 生态一变就得重采。

    1. Measured against Anthropic’s Claude Opus 4.8 and OpenAI’s GPT-5.5 — both much larger, frontier-scale systems — Faraday runs on a comparatively tiny model called Qwen 3.6 that has just 27 billion parameters.

      参数量对比是本文最抓眼球、也最容易误读的一句。被比较的不是两个模型,而是两套完整系统:Faraday 是 27B 权重加上专门的任务训练、工具与流程,对面是通用前沿模型跑同一批题。真正的结论应该是“为这个任务专门设计的系统赢了通用系统”,不是“小模型赢了大模型”。

    1. A model that absorbs permissions into itself has dissolved permissions. Absorption doesn’t end the harness. Absorption inverts the harness.

      全文最反直觉的一段,也是对“harness 是护城河”的反向预言:凡是能被权重吸收的 harness 能力迟早被吸收,Claude Code 删掉 80% 系统提示就是佐证。推论很实在——靠 harness 工程建立的优势,寿命按模型迭代周期计;能沉淀下来的是权限、身份、信任这些朝向人的接口。

    2. ran the same model over the same 106 tasks in different harnesses, and scores ranged from 52.4 to 76.2: a 23.8-point spread with zero change to the model.

      这是“harness 决定落地”目前最接近受控证明的一条:同一模型、同一批任务,只换 harness 就有 23.8 分差。对照 NVIDIA AVO 那种同时换后端、观测格式和记忆的比法,高下立判。仍要打折的是:106 个任务样本不大,而“一半的 agent 是 harness”是修辞,不是把分差当成贡献比例的依据。

    1. shows that system design – not model capability alone – can unlock frontier-level long-horizon performance

      方向性判断我同意:真正的运行单元是模型×harness×工具环境×上下文策略,不是裸模型。但“系统设计能解锁”和“这次提升由系统设计造成”是两回事,后者需要固定模型、只换 harness 的对照实验。目前公开材料只支持前者。

    2. Nvidia's AVO agent system lifted Claude Opus 5 from a 30% baseline to 100% on the ARC-AGI-3 reasoning benchmark, across all 183 levels.

      把 30% 和 100% 相减当成 harness 的贡献,是这轮报道里最常见的读法错误。两个数字来自不同评测设置:一边是 ARC Prize 跑裸模型,一边是 NVIDIA 自建观测接口、记忆与监督器的完整系统。NVIDIA 原文明说过这不是受控消融,本文没有转述这句限定。

    1. In the AVO configuration, the LLM operated in a text-only modality: each observation was supplied as an exact 64 x 64 text grid, with no images or image tokens sent to the model.

      与 VISTA 对照时的关键混杂项:对方主配置喂 512×512 渲染图,AVO 直接喂 64×64 文本网格。感知通道都换了,动作效率自然不可比。要警惕把“观测表示的胜利”读成“智能体架构的胜利”——真正的运行单元是模型×harness×工具环境×上下文策略。

    2. This should not be interpreted as a controlled ablation: the two systems differ in agent backend, observation representation, memory, context management, and other implementation details.

      全文最该被引用的一句:NVIDIA 自己先声明这不是受控消融。也就是说 12% 的动作数优势里,后端模型、观测表示、记忆、上下文管理四个变量同时在变,无法归因给任何单一设计。读跑分新闻时,这类作者自认的免责声明信息量往往大于标题数字。

    1. An OpenAI-backed study found that in June, 98% of OpenAI employees were using Codex, but just 17% of organizational subscribers and less than 1% of individual subscribers were using the agentic coding tool. That difference between near total adoption inside the company and negligible adoption outside it is the challenge and opportunity for the company.

      AI Buzzwords EP.100故事线B把这条数据当作"harness决定Agent能不能被普通人用起来"这个判断迄今最有力的一手商业证据——内部98%说明模型能力本身没问题,外部17%/1%的巨大落差说明卡住普通用户的是产品/harness没做到位,不是模型不够聪明。同一篇报道里还有一个容易被忽略的细节:OpenAI的非工程团队一开始用Codex时,工具还在"敌视"他们——反复追问代码相关问题、提示"你这里有个空diff",直到公司在2月到现在这段时间里把它做得更通用。这说明"让Agent普及"本身也是一个需要持续打磨的产品工程问题,不是模型发布后自动会发生的事。