Harness + closed-loop working view
Convergence audit (arXiv 2609.17394): within-model scaffold ranges reach 29.8 pp vs 8.8 pp top-30 spread; 0/29 adjacent top pairs separable. Harness Design (arXiv 2609.20804): 176 settings; rule-based elision before summarization; recoverable elision unused. Harness or Model? (2609.11987): no average native-harness advantage. Co-evolution still: expert-trajectory imitation regresses 4–30 pp. PRISM still 14.2/14.9/10.1 pp held-out.
Evidence & practices
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries (arXiv 2609.17394, 2026-09-15) audits 254 submissions across four splits and shows the harness/scaffold is often the dominant observable variance driver over model weights. On SWE-bench Verified, the leading two entries each resolve 396 of 500 instances, and the top ten share 285 successes and 51 failures, leaving only 164 instances that distinguish outcomes (effective size fraction 0.33; for the top two, effective size drops to 36 instances, or 0.07). Frontier solution sets show median nesting of 0.935 against a score-implied baseline of 0.774. Crucially, observed within-model scaffold ranges reach 29.8 percentage points (e.g. Claude 3.5 Sonnet spanning 168 to 317 resolved instances), dwarfing the 8.8 pp spread across the entire top thirty of Verified. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05. Retiring universally solved instances adds zero statistical power because paired tests already ignore agreements. This audit directly reinforces An Empirical Study of Harness Design for Coding Agents (arXiv 2609.20804, 2026-09-17): a lightweight harness holding the execution loop fixed while varying planning, action space, and context management across 176 matched settings, four models, SWE-Bench Verified, and Terminal-Bench 2.1. Context management becomes valuable as the budget tightens, with most benefit from preventing overflow failures. Staging rule-based elision before LLM summarization is the strongest efficiency strategy; making elided content recoverable adds machinery models rarely use and yields no accuracy gain. Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models. Bash-capable models operate effectively with a bash-only interface at substantially lower cost. Harness or Model? (arXiv 2609.11987, telemetry-corrected 2026-09-08) remains the pairing rule: on a private contamination-controlled suite, vendor-native pairing has no average solve-rate advantage (Opus 4.8 48.8% native vs 50.0% neutral; GPT-5.5 55.6% vs 54.4%). PRISM (Beyond Prompts, arXiv 2609.05736v2, EMNLP 2026) is the budgeted-search rule for frozen models: edit prompts and tool-boundary middleware under a budget, reporting RelLift95 alongside mean lift. Multi-Harness RL (arXiv 2609.04518): eval harness variation accounts for 4.3× gain vs training recipe 1.16×. Co-evolution (arXiv 2609.09134): imitating expert trajectories under an evolved harness regresses 4–30 pp; on-policy turn correction preserves fit.
Why it matters A weak harness makes frontier models expensive and unreliable; a strong one lets cheaper or open models close the gap. Scoring every harness candidate on a fixed suite hides regressions; feeding the full transcript under a tight window wastes the next search, edit, or test. Mean held-out lift without RelLift95 will promote a brittle PRISM/GEPA winner. Compressing all working-memory objects with one policy over-evicts artifacts and under-evicts tool dumps. A capability-positive self-edit that cannot be reversed is a production incident, not an improvement. HarnessDev shows creator self-eval is not held-out proof. The anatomy study shows production runtimes are hand-rolled platforms with deterministic retrieval — importing LangChain as the coding-agent loop is not how the field actually ships. EvoHarnessBench shows adding plugins/skills/MCP can regress old work without a weight change. Multi-Harness RL shows the eval harness you score on can dwarf the training recipe — freeze both before attributing a model upgrade. Co-evolution shows a second attribution trap: imitating an expert under a harness evolved for a weaker planner can destroy the fit you just paid to evolve. Neither AOT, JIT, nor a closed-loop view licenses unconstrained self-modifying production harnesses.
Economic lever Measured AOT harness evolution, budgeted tool-boundary search with RelLift95, model- and budget-aware context/planning/action-space choices, a closed-loop working view, catalog-growth retention checks, typed working-memory accounting, bounded JIT synthesis, recoverability plus leak/bloat/forgetting gates, on-policy turn correction instead of expert-trajectory imitation, and behavior-aware verification compress $/task versus model upgrades; an ungoverned self-rewriting harness, a native-pairing bet that loses on the actual workload mix, expert-SFT that breaks model–harness fit, a mean-lift winner that fails the lower tail, an expanding catalog without old-task holdouts, or a framework-shaped runtime that production CLIs do not use turns seat spend into high-variance token burn and unauditable regressions.
Watch for whether Harness Design-style component ablations replace monolithic harness bake-offs; whether org evals freeze workload mix before attributing a native-harness win; Whether vendor CLIs expose a closed-loop working view, typed working-memory telemetry, and a harness-evolution loop with attributable verification, undo, and leak/bloat gates; whether RelLift95-style selection reliability lands next to mean harness-eval scores; whether HarnessDev-style creation/evolution scores replace self-eval; whether EvoHarnessBench-style retention is an internal gate as catalogs grow; whether Harbor-Index becomes a compact hard holdout without replacing hermetic oracles; whether co-evolution pipelines default to on-policy turn correction rather than expert-trajectory SFT; whether JIT generators stay eval helpers; whether meta-harnesses (Omnigent-class) become the default way enterprises arbitrage vendor CLIs; whether Tool Primitive / ToolFace retrieval replaces full schema injection; whether Environment Evolution stays in training.
- Stage rule-based elision before LLM summarization; do not build recoverable-elision machinery the model never uses
- Choose vendor-native vs portable harness on workload mix and billed cost, not on an assumed native capability premium
- Treat the vendor CLI as a starter harness, then add org-specific verification, audit, deny rules, and a working-view policy
- Compare harnesses on the same task and model before attributing gains to the model; score the model–harness pair as the solver; freeze the eval harness when comparing training recipes
- When searching harness edits, constrain middleware to tool-boundary intercepts and report RelLift95 of the selected harness, not only mean held-out lift
- Keep a complete run record but do not feed the full transcript as the working view once the window is under pressure
- Account working memory by object type (instruction, artifact, tool output, agent state); do not treat a token budget as delivered context
- When co-evolving weights and harness, correct the failing turn on-policy; do not imitate expert full trajectories under a harness evolved for the weaker model's planning style
- Re-run a frozen old-task holdout whenever the org plugin/skill/MCP catalog grows; admit a self-edit only if an independent recoverability check can restore a typed prior state, a quality gate rejects leak/bloat, and a held-out executor eval does not regress; freeze JIT generator output behind the same sandbox, allowlist, and eval gate as human harness diffs; do not wrap production coding agents in a general-purpose agent framework the production corpus does not import
- [preprint] Coding Agents Have Converged (arXiv:2609.17394) — 2026-09-15. 254 submissions audited; within-model scaffold ranges reach 29.8 pp vs 8.8 pp top-30 spread; 0/29 adjacent Verified pairs separable (McNemar alpha=0.05); frontier nesting 0.935; neff drops to 36 instances for top two.
- [preprint] An Empirical Study of Harness Design for Coding Agents (arXiv:2609.20804) — 2026-09-17. 176 matched settings; rule-based elision before LLM summarization; recoverable elision unused; planning as cost saver for strong models.
- [preprint] Harness or Model? Isolating the Harness Effect (arXiv:2609.11987) — Telemetry-corrected 2026-09-08. No average native advantage; Opus repo −9.0 pp / contest +23.7 pp; neutral 1.2–1.6× cost per solved.
- [preprint] Beyond Prompts / PRISM (arXiv:2609.05736v2) — v2 2026-09-09. EMNLP 2026. Held-out lifts 14.2 / 14.9 / 10.1 pp on BFCL / tau2-Retail / tau2-Telecom; RelLift95 of selected harness.
- [preprint] Co-Evolving Harnesses and Models (arXiv:2609.09134) — 2026-09-08. Expert-trajectory imitation under an evolved harness regresses 4–30 pp on all 7 enterprise tasks (Qwen3-Coder, Gemma 4). On-policy failing-turn correction preserves native planning style.
- [preprint] Harbor Adapters and Harbor-Index (arXiv:2609.04298) — 2026-09-03. 80+ adapters; 8 models × Terminus-2 + native harness on 54 benches. Harbor-Index 82 tasks; max 28.0% (GPT-5.5+Codex).
- [preprint] What Does Multi-Harness RL Learn? (arXiv:2609.04518) — 2026-09-03. Eval harness 2.14%→9.27% (4.3×) vs training recipe 1.16×; Cross−Within +0.25 pp on held-out harness.
- [preprint] EvoHarnessBench (arXiv:2609.04280) — 2026-09-03. 17 streams / 802 tasks / 520 tools / 42 skills / 62 agents. Harness-induced forgetting; retention vs adaptation tradeoff.
- [preprint] Environment Evolution for Terminal Agents (arXiv:2609.04128) — 2026-09-03. Off-policy env difficulty ramps; Qwen3.6-27B / 35B-A3B +14.4 / +18.0 pp on Terminal-Bench 2.1. Training-env, not production CLI rewrite.
- [preprint] Harness Engineering: Anatomy of Eleven Systems (arXiv:2609.00006) — HTML 2026-09-02 (July 2026 corpus, 2nd edition). 11 harnesses + Omnigent. ~4M LOC: no general-purpose agent framework; no embedding retrieval. SKILL.md 9/11 vs MCP 8/11; ACP in 6 with harness hosting. Codex copies Claude Code hooks; OpenHands reads Claude Code plugins.
- [preprint] HEART / Tool Primitives (arXiv:2609.01736) — 2026-09-01. ToolFace 25,519 functions; Planner/Router/Verifier. 84% task completion vs 22% avg of three frontier commercial models; API cost −up to 85%.
- [preprint] HarnessEvolve (arXiv:2609.00829) — 2026-09-01. Quality+performance gates; CloudCoreNetwork-QA 43.4%→86.9% vs GEPA 65.3% (+21.6 pp); w/o reference trajectories 57.8%.
- [preprint] HarnessDev (arXiv:2609.01437) — 2026-09-01. Eval unit = runnable infrastructure. GPT-5 Terminal-Bench 2.1: 35.2% Terminus 2 vs 49.6% Codex CLI. Creation lags human refs on code/search; evolution unstable, executor-dependent; Opus 4.8 held-out +4.44 pp.
- [preprint] Same Model, Different Harness (arXiv:2608.26218) — 2026-08-26. Tight-window SWE-bench Verified: F2PF 28%→49%, complete solutions 43→72; frozen treatment transfers across models.
- [preprint] Measure Before You Manage (arXiv:2608.31057) — 2026-08-31. 55 coding trajectories; tool outputs 55.5% volume vs 40.2% retention-weighted cost; four-level memory eval.
- [preprint] EvoUndo (arXiv:2608.28363) — 2026-08-28. 197 capability-improving self-edits fail recoverability; conventional repair 0/197; extended recovery language 191/197 oracle.
- [preprint] AutoSaddler (arXiv:2608.23041) — 2026-08-24. Trace-driven AOT patches with validation selection: +9.0 GAIA2, +9.6 SWE-Bench Pro, +10.0 Terminal-Bench 2.0.
- [preprint] StarHarness (arXiv:2608.24804) — 2026-08-25. 20–35pp after 4–12 accepted AOT changes; transfer across GPT and Qwen.
- [preprint] HarnessLens (arXiv:2608.27311) — 2026-08-27. Behavior-aware verification: 7.6–13.6% held-out gain at lower eval budget.
- [preprint] JIT-Agent (arXiv:2608.25593) — 2026-08-26. DeepSeek-V4-Flash under JIT harnesses beats GPT-5.6 on DeepSearchQA (+9.1).
- [preprint] openJiuwen (arXiv:2608.27969) — 2026-08-28. 82.6% SWE-bench Verified, 87.19% Terminal-Bench 2.1 vs selected official-leaderboard point estimates (+3.4 / +3.39 pp).
- [preprint] Architectural convergence in three LLM agent harnesses (arXiv:2608.23953) — 2026-08-25. Five-element middle form; external verifiability still missing across deepagents, pi, dsh.
- [vendor] Context engineering for coding agents (Decoding AI) — 2026-08-25. Microcompaction ~60% window; full compaction ~80%; progressive skill disclosure.
- [preprint] SWE-Bench Pro Verified (arXiv:2609.08149) — 2026-09-08. Anti-hacking leakage elimination + task inconsistency refinement; reveals raw benchmark overestimation of software engineering capability.