Harness + closed-loop working view
Holding model and tools fixed, a mechanical working-view rule lifts tight-window SWE-bench F2PF from 28% to 49%. HarnessDev: model-built harnesses still lag human refs on code and search. Anatomy of 11 production harnesses: none import a general-purpose agent framework; none retrieve code with embeddings. HEART: 84% task completion vs 22% frontier average by retrieving Tool Primitives instead of dumping schemas.
Evidence & practices
Practice still treats Agent = Model + Harness, and now treats the working view, typed working-memory objects, the AOT optimizer, recoverability, gated self-evolution, and the production runtime's actual source as part of the solver. Harness Engineering (arXiv 2609.00006; HTML 2026-09-02; July 2026 corpus, second edition of an April study) is a source-code anatomy of eleven production coding harnesses — Claude Code, Codex CLI, Gemini CLI, Mistral Vibe, OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, OpenClaw — plus Omnigent (Databricks), the first meta-harness analyzed as a contrast point (23 canonical adapters plus aliases). It maps seven canonical subsystems and catalogs 13 observations and 29 design patterns. Two absences survive a threefold corpus expansion: across roughly four million lines of Python, TypeScript, and Rust, no agent runtime imports a general-purpose agentic framework (LangChain, LangGraph, AutoGen, or a dozen others; Gemini CLI uses neither of Google's own), and none retrieves code with vector embeddings — the field runs on hand-rolled async loops and deterministic retrieval (ripgrep, tree-sitter, glob, auto-discovered Markdown context files). SKILL.md skills lead MCP in adoption (9/11 vs 8/11); ACP ships in six systems with a third role, harness hosting (OpenHands running Claude Code, Codex, or Gemini CLI as interchangeable backends). A controlled longitudinal re-pin of the original eight systems shows convergence becoming imitation (Codex adopts Claude Code's hook vocabulary verbatim and ships an importer for its sessions and settings; OpenHands reads Claude Code's plugin format) and behavioral policy migrating from prompt prose to configuration. Thesis: in the first half of 2026 the coding harness completed a turn from tool to platform. Same Model, Different Harness (arXiv 2608.26218, 2026-08-26) still holds model, tasks, tools, and window fixed and changes only harness configuration: tight-window SWE-bench Verified F2PF 28%→49%, complete solutions 43→72; the frozen treatment transfers across models. HEART (arXiv 2609.01736, 2026-09-01) wraps 25,519 ToolFace functions as Tool Primitives with a Planner/Router/Verifier: 84% task completion vs 22% average of three frontier commercial models and up to 85% lower API cost. HarnessDev (arXiv 2609.01437, 2026-09-01) makes the eval unit runnable infrastructure: GPT-5 Terminal-Bench 2.1 is 35.2% Terminus 2 vs 49.6% Codex CLI; creation lags human refs on code/search; evolution is unstable and executor-dependent. HarnessEvolve (arXiv 2609.00829, 2026-09-01) gates AOT edits on quality+performance: CloudCoreNetwork-QA 43.4%→86.9% vs GEPA 65.3% (+21.6 pp). Environment Evolution (arXiv 2609.04128, 2026-09-03) is a training-env loop, not a production self-rewrite: off-policy difficulty ramps via a loop-engineered multi-agent harness lift Qwen3.6-27B and Qwen3.6-35B-A3B by 14.4 and 18.0 percentage points on Terminal-Bench 2.1. Keep it in the eval/search lane. Measure Before You Manage (arXiv 2608.31057): tool outputs are 55.5% of volume vs 40.2% retention-weighted cost. EvoUndo: 197 capability-improving self-edits fail recoverability; conventional repair 0/197. AutoSaddler still +9.0 GAIA2 / +9.6 SWE-Bench Pro / +10.0 Terminal-Bench 2.0. StarHarness 20–35 pp after 4–12 accepted AOT changes. HarnessLens 7.6–13.6% held-out gain. JIT-Agent: DeepSeek-V4-Flash under JIT harnesses beats GPT-5.6 on DeepSearchQA (+9.1). Decoding AI still: microcompaction ~60% window; full compaction ~80%; progressive skill disclosure.
Why it matters A weak harness makes frontier models expensive and unreliable; a strong one lets cheaper or open models close the gap. Scoring every harness candidate on a fixed suite hides regressions; feeding the full transcript under a tight window wastes the next search, edit, or test. Compressing all working-memory objects with one policy over-evicts artifacts and under-evicts tool dumps. A capability-positive self-edit that cannot be reversed is a production incident, not an improvement. HarnessDev shows creator self-eval is not held-out proof, and a harness that only works under its author-executor is not a portable org standard. The anatomy study shows production runtimes are hand-rolled platforms with deterministic retrieval — importing LangChain as the coding-agent loop is not how the field actually ships. Neither AOT, JIT, nor a closed-loop view licenses unconstrained self-modifying production harnesses. Evolving the training environment is not the same as evolving the production CLI.
Economic lever Measured AOT harness evolution, a closed-loop working view, typed working-memory accounting, bounded JIT synthesis, recoverability plus leak/bloat/forgetting gates, and behavior-aware verification compress $/task versus model upgrades; an ungoverned self-rewriting harness or a framework-shaped runtime that production CLIs do not use turns seat spend into high-variance token burn and unauditable regressions.
Watch for Whether vendor CLIs expose a closed-loop working view, typed working-memory telemetry, and a harness-evolution loop with attributable verification, undo, and leak/bloat gates; whether HarnessDev-style creation/evolution scores replace self-eval for internal harness selection; whether JIT generators ship as eval helpers rather than production self-rewrites; whether meta-harnesses (Omnigent-class) become the default way enterprises arbitrage vendor CLIs; whether Tool Primitive / ToolFace retrieval replaces full schema injection for large internal catalogs; whether Environment Evolution stays in training and does not leak into production hook/self-edit paths.
- Treat the vendor CLI as a starter harness, then add org-specific verification, audit, deny rules, and a working-view policy
- Compare harnesses on the same task and model before attributing gains to the model; score the model–harness pair as the solver
- Keep a complete run record but do not feed the full transcript as the working view once the window is under pressure
- Account working memory by object type (instruction, artifact, tool output, agent state); do not treat a token budget as delivered context
- Admit a self-edit only if an independent recoverability check can restore a typed prior state, a quality gate rejects leak/bloat, and a held-out executor eval does not regress; freeze JIT generator output behind the same sandbox, allowlist, and eval gate as human harness diffs; do not wrap production coding agents in a general-purpose agent framework the production corpus does not import
- [preprint] Environment Evolution for Terminal Agents (arXiv:2609.04128) — 2026-09-03. Off-policy env difficulty ramps; Qwen3.6-27B / 35B-A3B +14.4 / +18.0 pp on Terminal-Bench 2.1. Training-env, not production CLI rewrite.
- [preprint] Harness Engineering: Anatomy of Eleven Systems (arXiv:2609.00006) — HTML 2026-09-02 (July 2026 corpus, 2nd edition). 11 harnesses + Omnigent. ~4M LOC: no general-purpose agent framework; no embedding retrieval. SKILL.md 9/11 vs MCP 8/11; ACP in 6 with harness hosting. Codex copies Claude Code hooks; OpenHands reads Claude Code plugins.
- [preprint] HEART / Tool Primitives (arXiv:2609.01736) — 2026-09-01. ToolFace 25,519 functions; Planner/Router/Verifier. 84% task completion vs 22% avg of three frontier commercial models; API cost −up to 85%.
- [preprint] HarnessEvolve (arXiv:2609.00829) — 2026-09-01. Quality+performance gates; CloudCoreNetwork-QA 43.4%→86.9% vs GEPA 65.3% (+21.6 pp); w/o reference trajectories 57.8%.
- [preprint] HarnessDev (arXiv:2609.01437) — 2026-09-01. Eval unit = runnable infrastructure. GPT-5 Terminal-Bench 2.1: 35.2% Terminus 2 vs 49.6% Codex CLI. Creation lags human refs on code/search; evolution unstable, executor-dependent; Opus 4.8 held-out +4.44 pp.
- [preprint] Same Model, Different Harness (arXiv:2608.26218) — 2026-08-26. Tight-window SWE-bench Verified: F2PF 28%→49%, complete solutions 43→72; frozen treatment transfers across models.
- [preprint] Measure Before You Manage (arXiv:2608.31057) — 2026-08-31. 55 coding trajectories; tool outputs 55.5% volume vs 40.2% retention-weighted cost; four-level memory eval.
- [preprint] EvoUndo (arXiv:2608.28363) — 2026-08-28. 197 capability-improving self-edits fail recoverability; conventional repair 0/197; extended recovery language 191/197 oracle.
- [preprint] AutoSaddler (arXiv:2608.23041) — 2026-08-24. Trace-driven AOT patches with validation selection: +9.0 GAIA2, +9.6 SWE-Bench Pro, +10.0 Terminal-Bench 2.0.
- [preprint] StarHarness (arXiv:2608.24804) — 2026-08-25. 20–35pp after 4–12 accepted AOT changes; transfer across GPT and Qwen.
- [preprint] HarnessLens (arXiv:2608.27311) — 2026-08-27. Behavior-aware verification: 7.6–13.6% held-out gain at lower eval budget.
- [preprint] JIT-Agent (arXiv:2608.25593) — 2026-08-26. DeepSeek-V4-Flash under JIT harnesses beats GPT-5.6 on DeepSearchQA (+9.1).
- [preprint] openJiuwen (arXiv:2608.27969) — 2026-08-28. 82.6% SWE-bench Verified, 87.19% Terminal-Bench 2.1 vs selected official-leaderboard point estimates (+3.4 / +3.39 pp).
- [preprint] Architectural convergence in three LLM agent harnesses (arXiv:2608.23953) — 2026-08-25. Five-element middle form; external verifiability still missing across deepagents, pi, dsh.
- [vendor] Context engineering for coding agents (Decoding AI) — 2026-08-25. Microcompaction ~60% window; full compaction ~80%; progressive skill disclosure.