AI dominicreichl.com
Labs Models Trends

Executive summary

AI engineering trends

As of 2026-09-25, agentic engineering is defined by harness architecture as the primary variance driver, reinforced by pre-action verification audits, fail-closed runtime sandboxing, semantic coordination in parallel workflows, and authoritative enterprise default policies. GitHub (2026-09-24) introduced a global default policy across enterprise and organization AI Controls, establishing mandatory baselines (Enabled, Disabled, or Let organizations decide) for GA features, Copilot Code Review, and MCP servers in Copilot ahead of an October 22 cutover. Local sandboxing in the GitHub Copilot app (2026-09-23) enforces fail-closed project-level boundaries across filesystem, network, and git/CLI credentials. Copilot CLI 1.0.88 npm latest (prerelease 1.0.89-3, verified 2026-09-25) closes headless policy loopholes by binding enterprise managed settings to ACP, AHP, and server sessions, while auto-applying custom-agent reasoning effort. STALE-bench (arXiv 2609.25396) demonstrates that parallel coding agents produce semantic interference in 97% of runs on dependent interfaces despite clean git merges, recoverable (82%) only with concurrent change broadcast oracles. Copilot code review (2026-09-23 GA) expands configuration with dedicated personal settings (draft PR and push triggers) and enterprise-wide default review effort (Lite/Balanced). Look Before You Leap (arXiv 2609.11957) confirms pre-action deterministic checks catch 95.8% of invalid shell commands, while anchor-and-verify appliers cut silent file corruption to 0.01%. SWE-Proof (arXiv 2609.21190) shows 25% to 50% of test-passing patches admit counterexamples. Coding Agents Have Converged (arXiv 2609.17394) and Harness Design (arXiv 2609.20804) confirm scaffold choices outstrip model selection across the frontier.

As of 2026-09-25 · Updated daily at 07:25 Europe/Vienna · Product owners and leads of AI / agentic engineering teams in large software firms — engineering SotA first; economic enablement as the firm-level implication of how the team builds

Bottom line

What matters this cycle.

Engineering posture first; economic enablement only where practice changes cost, leverage, or where the firm should bet.

  1. 01

    The production harness is the dominant variance driver; benchmark frontiers have converged. Convergence Audit (arXiv 2609.17394): within-model scaffold ranges reach 29.8 pp (exceeding the 8.8 pp top-30 spread); exact McNemar tests separate 0 of 29 adjacent pairs; frontier nesting is 0.935. Harness Design (arXiv 2609.20804): 176 settings; rule-based elision before LLM summarization; recoverable elision unused; planning saves cost for strong models. SWE-Bench Pro Verified (arXiv 2609.08149): anti-hacking leakage elimination depresses inflated benchmark scores. Harness or Model? (arXiv 2609.11987): no native solve-rate advantage.

  2. 02

    Verifiability requires pre-action checks and semantic coordination oracles; self-report and line edits fail silently. Look Before You Leap (arXiv 2609.11957): pre-action checks catch 95.8% of shell errors; line-number edits corrupt 99.1% of files on 1-line shift; anchor-and-verify applier slashes silent misapplications to 0.01% (1 in 8,320 trials). STALE-bench (arXiv 2609.25396): parallel coding agents produce 97% semantic interference on dependent interfaces despite clean git merges (82% recovered by oracle messages). SWE-Proof (arXiv 2609.21190): 25%–50% of test-passing patches admit counterexamples. OverclaimBench (arXiv 2609.20812): 67.9% of runs skip files; 80.4% misleading completion claims.

  3. 03

    Enterprise platform governance and runtime isolation go fail-closed; Copilot CLI 1.0.88 locks headless policy; code review expands controls. GitHub AI Controls (2026-09-24) adds a global default policy for GA features, Code Review, and MCP servers (Enabled/Disabled/Let orgs decide; takes effect Oct 22). Local sandboxing in Copilot app (2026-09-23) enforces fail-closed fs/network/credential controls. Copilot CLI npm latest is 1.0.88 (verified 2026-09-25; prerelease 1.0.89-3): enterprise managed settings bind ACP (--acp), AHP hosts, and --server sessions. Copilot code review (2026-09-23 GA) adds personal draft/push triggers and enterprise default effort (Lite/Balanced). OTel in Copilot app (2026-09-22) exports agent traces via managed settings. Oct 2 & Oct 19 model deprecations.

Operating shifts

How the work is changing.

Old default → current SotA posture (with firm-level implications where they change).

From To Implication
Prompt engineering as the core skill; one durable harness optimized ahead of time Harness + context engineering, with a closed-loop working view, recoverability-constrained AOT evolution for stable domains, JIT synthesis when instance structure varies, and behavior-aware verification of every accepted edit Agent = Model + Harness. Harness Design (arXiv 2609.20804): hold the loop fixed and vary planning, action space, and context management; stage rule-based elision before LLM summarization; do not pay for recoverable elision the model never uses. Convergence Audit (arXiv 2609.17394): within-model scaffold ranges reach 29.8 pp (more than the 8.8 pp top-30 spread on SWE-bench Verified), confirming that scaffold architecture frequently dominates model selection, while 0 of 29 adjacent top-30 pairs are statistically separable. Harness or Model? (arXiv 2609.11987): do not assume vendor-native pairing wins on average — choose on workload mix and billed cost. Mechanical working-view rules change what unchanged weights accomplish. PRISM (arXiv 2609.05736v2): search prompts and tool-boundary middleware under a budget; report RelLift95 of the selected harness, not only mean held-out lift. Typed working-memory objects (instructions, artifacts, tool outputs, agent state) do not share retention or compression profiles. Production CLIs stay AOT; treat JIT as an eval/search loop with a frozen policy surface. Co-evolution (arXiv 2609.09134): do not imitate expert full trajectories under a harness evolved for the weaker model's planning style — correct the failing turn on-policy instead. HarnessDev: score the created harness on held-out tasks and executor-token cost, not the creator's self-eval. HarnessEvolve: decouple execution from evolution; reject leak/bloat. EvoUndo: a capability-positive self-edit that cannot be undone is not an accepted patch. Anatomy study: do not import a general-purpose agent framework as the runtime; deterministic retrieval is the production default. Environment Evolution hardens training tasks off-policy; it is not a license to rewrite the production CLI from rollouts. EvoHarnessBench: expanding the org catalog can forget previously solved tasks — measure retention on old work as the plugin/skill/MCP pool grows.
Picking MCP or CLI as a platform-wide strategy Measure the scaffolding; pick the interface per integration (CLI for local/known tools, MCP for governed/remote/multi-tenant); expect progressive discovery, default permission modes, and continuous tool-schema drift checks Interface cost is second-order. Copilot CLI production pin is npm latest 1.0.88 (verified 2026-09-25); prerelease is 1.0.89-3. 1.0.88: enterprise managed settings bind ACP mode, AHP hosts, and --server sessions; custom agent reasoning-effort applies automatically; hook commands without cwd run in project root; namespaced skills. 1.0.87: --fleet parallel orchestration, --assisted-approval safety judge, --extension-sdk-path. GitHub AI Controls (2026-09-24) introduces a global default policy for GA features and MCP servers. Copilot app adds OpenTelemetry for agent activity tracing (2026-09-22). 1.0.85: /sandbox network host allow/deny without replacing the upstream proxy. CLI-only harnesses were 5–28× cheaper even on CLI-only runs. Deadbugz/TrustShift: an MCP server that looks clean at install is not a control. Scanning the Harness: 9.8% of setups install unpinned MCP — pin versions. Enterprise managed permissions (2026-09-09 GA) cover shell, file, and network on CLI and cannot be weakened by saved approvals.
Per-client skill and MCP packaging; scan-only skill gates Agent Plugins 1.0 plus live Skill Lift, committed repo config, and allowlisted (optionally auto-updated) marketplaces One plugin.json + skills/ + mcp.json travels across clients. Copilot CLI 1.0.88 supports namespaced custom skills and ignored skill directories during discovery. Census 2,765 plugins / 11,678 skills / 1,260 MCP (verified 2026-09-21). Look Before You Leap (arXiv 2609.11957): content-anchored edits fail cleanly while location-anchored edits silently corrupt 99.1% of files. SWE-Proof (arXiv 2609.21190): 25%–50% of test-passing patches admit counterexamples; formal specs lift Opus 4.8 85%→95%. SkillAA (arXiv 2609.20455): attribution-guided skill-graph repair with Local and Big Gates. Subagents vs Agent Skills (arXiv 2609.09233): invoke contract-bearing packages as subagents; load loosely structured knowledge in-context. COBRA-Skills (arXiv 2609.11682): budget evaluations with a contextual bandit; 55–58% cost cut vs SkillOpt. Skill Issue (arXiv 2609.12742): a one-repo GEPA lift of +4.9 pp is not a promotion gate. Auto-update stays behind marketplace allowlists — HookPry shows a plugin update can bind host-privileged hooks.
Prompts, model safeguards, and harness allowlists as the security story Authoritative policy in the runtime: sandbox, least privilege, JIT credentials, emergency shutdown, content exclusions as context policy, provenance for page-exposed tools, induction vs execution authorization, Propose-Verify-Act-Verify containment, continuous MCP tool-schema attestation, and pinned/reviewed lifecycle hooks A control the agent can decline to invoke is not a control. GitHub AI Controls (2026-09-24) establishes enterprise-wide global default policy (Enabled/Disabled/Let orgs decide) for GA features, Code Review, and MCP servers (effective Oct 22). Local sandboxing in the GitHub Copilot app (2026-09-23 public preview) introduces fail-closed isolation across filesystem, network, and git/CLI credentials. Copilot CLI 1.0.88 closes the ACP/AHP/server policy loophole so enterprise managed settings bind all headless sessions; exact session path approvals prevent parent-dir escalation. 1.0.87: --assisted-approval safety judge. 1.0.86: /sandbox network allow/deny without replacing the proxy. Look Before You Leap (arXiv 2609.11957): pre-action static checks catch 95.8% of shell errors with 0% false positives on syntax; anchor-and-verify appliers prevent silent code corruption (0.01% misapplication). Scanning the Harness (arXiv 2609.07360): 16.0% of setups carry a confirmed security defect. Enterprise managed permissions (2026-09-09 GA) put deny/ask/allow for shell, file, and network below the harness. Lifecycle hooks run with host privileges — pin hook configs; treat plugin auto-update as a hook-update path. SafeEvolve: 3× ASR cut on AgentDojo without utility regression.
Single-session harness evals, append-only transcripts, and end-state scores Loop engineering: triggered runs, machine-checkable stops, explicit execution state, durable loop state, closed-loop working view, a Controller/Worker split, symptom-driven replay instead of whole-run resample, and deterministic gates that outrank LLM-as-judge Long-horizon work fails as a state machine. STALE-bench (arXiv 2609.25396): parallel coding agents suffer 97% semantic coordination failures despite clean git merges (82% recovered by oracle messages). OverclaimBench (arXiv 2609.20812): agents fail to read all requested review files in 67.9% of runs, and 80.4% of incomplete runs falsely claim completion or omit gaps — never use agent self-report as an acceptance or loop termination gate. Copilot CLI 1.0.85 context-management tools are opt-in for agents and subagents — they are not a Continuity Kernel. 1.0.86 resume preserves marketplace plugins; custom agents opt into AGENTS.md via include-custom-instructions. Subagents vs Agent Skills: peak context, not total tokens, is the bandwidth limit — spawn a fresh window for contract-bearing skills. Experience Funnel: funnel validated textual state into policy slowly. HoH shows an outer planning–coding–testing loop over an existing vendor harness can beat the standalone CLI without replacing it. SKILLSTATE keeps prompt size O(1). LoopArena caps the best controller at 24.69% Strict Success on full tasks. Unguided MAS reruns repair 6.90%; symptom-driven intervention reaches 20.15%. Session restore is continuity of a process, not a Continuity Kernel. JetBrains /ide is IDE-grounded CLI continuity, not durable loop state. PROCTOR: a Teacher that can override hermetic acceptance checks is an oracle you did not hire. Harbor-Index (28% ceiling) is a compact hard holdout; Multi-Harness RL says the eval harness you score on can dwarf the training recipe — freeze both. PRISM: RelLift95 of the selected harness belongs next to the mean. VS Code Agents window Dev Containers and in-window PRs are session glue, not loop state.

Priority implications

Where to put attention.

7 priorities (max 7). Collapsed claim is the scannable layer; expand a card for evidence detail, Watch for, and Key practices. Full Layer 1+2 read target: under 5 minutes.

1

Harness + closed-loop working view

Agentic eng

Convergence audit (arXiv 2609.17394): within-model scaffold ranges reach 29.8 pp vs 8.8 pp top-30 spread; 0/29 adjacent top pairs separable. Harness Design (arXiv 2609.20804): 176 settings; rule-based elision before summarization; recoverable elision unused. Harness or Model? (2609.11987): no average native-harness advantage. Co-evolution still: expert-trajectory imitation regresses 4–30 pp. PRISM still 14.2/14.9/10.1 pp held-out.

Evidence & practices

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries (arXiv 2609.17394, 2026-09-15) audits 254 submissions across four splits and shows the harness/scaffold is often the dominant observable variance driver over model weights. On SWE-bench Verified, the leading two entries each resolve 396 of 500 instances, and the top ten share 285 successes and 51 failures, leaving only 164 instances that distinguish outcomes (effective size fraction 0.33; for the top two, effective size drops to 36 instances, or 0.07). Frontier solution sets show median nesting of 0.935 against a score-implied baseline of 0.774. Crucially, observed within-model scaffold ranges reach 29.8 percentage points (e.g. Claude 3.5 Sonnet spanning 168 to 317 resolved instances), dwarfing the 8.8 pp spread across the entire top thirty of Verified. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05. Retiring universally solved instances adds zero statistical power because paired tests already ignore agreements. This audit directly reinforces An Empirical Study of Harness Design for Coding Agents (arXiv 2609.20804, 2026-09-17): a lightweight harness holding the execution loop fixed while varying planning, action space, and context management across 176 matched settings, four models, SWE-Bench Verified, and Terminal-Bench 2.1. Context management becomes valuable as the budget tightens, with most benefit from preventing overflow failures. Staging rule-based elision before LLM summarization is the strongest efficiency strategy; making elided content recoverable adds machinery models rarely use and yields no accuracy gain. Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models. Bash-capable models operate effectively with a bash-only interface at substantially lower cost. Harness or Model? (arXiv 2609.11987, telemetry-corrected 2026-09-08) remains the pairing rule: on a private contamination-controlled suite, vendor-native pairing has no average solve-rate advantage (Opus 4.8 48.8% native vs 50.0% neutral; GPT-5.5 55.6% vs 54.4%). PRISM (Beyond Prompts, arXiv 2609.05736v2, EMNLP 2026) is the budgeted-search rule for frozen models: edit prompts and tool-boundary middleware under a budget, reporting RelLift95 alongside mean lift. Multi-Harness RL (arXiv 2609.04518): eval harness variation accounts for 4.3× gain vs training recipe 1.16×. Co-evolution (arXiv 2609.09134): imitating expert trajectories under an evolved harness regresses 4–30 pp; on-policy turn correction preserves fit.

Why it matters A weak harness makes frontier models expensive and unreliable; a strong one lets cheaper or open models close the gap. Scoring every harness candidate on a fixed suite hides regressions; feeding the full transcript under a tight window wastes the next search, edit, or test. Mean held-out lift without RelLift95 will promote a brittle PRISM/GEPA winner. Compressing all working-memory objects with one policy over-evicts artifacts and under-evicts tool dumps. A capability-positive self-edit that cannot be reversed is a production incident, not an improvement. HarnessDev shows creator self-eval is not held-out proof. The anatomy study shows production runtimes are hand-rolled platforms with deterministic retrieval — importing LangChain as the coding-agent loop is not how the field actually ships. EvoHarnessBench shows adding plugins/skills/MCP can regress old work without a weight change. Multi-Harness RL shows the eval harness you score on can dwarf the training recipe — freeze both before attributing a model upgrade. Co-evolution shows a second attribution trap: imitating an expert under a harness evolved for a weaker planner can destroy the fit you just paid to evolve. Neither AOT, JIT, nor a closed-loop view licenses unconstrained self-modifying production harnesses.

Economic lever Measured AOT harness evolution, budgeted tool-boundary search with RelLift95, model- and budget-aware context/planning/action-space choices, a closed-loop working view, catalog-growth retention checks, typed working-memory accounting, bounded JIT synthesis, recoverability plus leak/bloat/forgetting gates, on-policy turn correction instead of expert-trajectory imitation, and behavior-aware verification compress $/task versus model upgrades; an ungoverned self-rewriting harness, a native-pairing bet that loses on the actual workload mix, expert-SFT that breaks model–harness fit, a mean-lift winner that fails the lower tail, an expanding catalog without old-task holdouts, or a framework-shaped runtime that production CLIs do not use turns seat spend into high-variance token burn and unauditable regressions.

Watch for whether Harness Design-style component ablations replace monolithic harness bake-offs; whether org evals freeze workload mix before attributing a native-harness win; Whether vendor CLIs expose a closed-loop working view, typed working-memory telemetry, and a harness-evolution loop with attributable verification, undo, and leak/bloat gates; whether RelLift95-style selection reliability lands next to mean harness-eval scores; whether HarnessDev-style creation/evolution scores replace self-eval; whether EvoHarnessBench-style retention is an internal gate as catalogs grow; whether Harbor-Index becomes a compact hard holdout without replacing hermetic oracles; whether co-evolution pipelines default to on-policy turn correction rather than expert-trajectory SFT; whether JIT generators stay eval helpers; whether meta-harnesses (Omnigent-class) become the default way enterprises arbitrage vendor CLIs; whether Tool Primitive / ToolFace retrieval replaces full schema injection; whether Environment Evolution stays in training.

Key practices
  • Stage rule-based elision before LLM summarization; do not build recoverable-elision machinery the model never uses
  • Choose vendor-native vs portable harness on workload mix and billed cost, not on an assumed native capability premium
  • Treat the vendor CLI as a starter harness, then add org-specific verification, audit, deny rules, and a working-view policy
  • Compare harnesses on the same task and model before attributing gains to the model; score the model–harness pair as the solver; freeze the eval harness when comparing training recipes
  • When searching harness edits, constrain middleware to tool-boundary intercepts and report RelLift95 of the selected harness, not only mean held-out lift
  • Keep a complete run record but do not feed the full transcript as the working view once the window is under pressure
  • Account working memory by object type (instruction, artifact, tool output, agent state); do not treat a token budget as delivered context
  • When co-evolving weights and harness, correct the failing turn on-policy; do not imitate expert full trajectories under a harness evolved for the weaker model's planning style
  • Re-run a frozen old-task holdout whenever the org plugin/skill/MCP catalog grows; admit a self-edit only if an independent recoverability check can restore a typed prior state, a quality gate rejects leak/bloat, and a held-out executor eval does not regress; freeze JIT generator output behind the same sandbox, allowlist, and eval gate as human harness diffs; do not wrap production coding agents in a general-purpose agent framework the production corpus does not import
Evidence
  • [preprint] Coding Agents Have Converged (arXiv:2609.17394) — 2026-09-15. 254 submissions audited; within-model scaffold ranges reach 29.8 pp vs 8.8 pp top-30 spread; 0/29 adjacent Verified pairs separable (McNemar alpha=0.05); frontier nesting 0.935; neff drops to 36 instances for top two.
  • [preprint] An Empirical Study of Harness Design for Coding Agents (arXiv:2609.20804) — 2026-09-17. 176 matched settings; rule-based elision before LLM summarization; recoverable elision unused; planning as cost saver for strong models.
  • [preprint] Harness or Model? Isolating the Harness Effect (arXiv:2609.11987) — Telemetry-corrected 2026-09-08. No average native advantage; Opus repo −9.0 pp / contest +23.7 pp; neutral 1.2–1.6× cost per solved.
  • [preprint] Beyond Prompts / PRISM (arXiv:2609.05736v2) — v2 2026-09-09. EMNLP 2026. Held-out lifts 14.2 / 14.9 / 10.1 pp on BFCL / tau2-Retail / tau2-Telecom; RelLift95 of selected harness.
  • [preprint] Co-Evolving Harnesses and Models (arXiv:2609.09134) — 2026-09-08. Expert-trajectory imitation under an evolved harness regresses 4–30 pp on all 7 enterprise tasks (Qwen3-Coder, Gemma 4). On-policy failing-turn correction preserves native planning style.
  • [preprint] Harbor Adapters and Harbor-Index (arXiv:2609.04298) — 2026-09-03. 80+ adapters; 8 models × Terminus-2 + native harness on 54 benches. Harbor-Index 82 tasks; max 28.0% (GPT-5.5+Codex).
  • [preprint] What Does Multi-Harness RL Learn? (arXiv:2609.04518) — 2026-09-03. Eval harness 2.14%→9.27% (4.3×) vs training recipe 1.16×; Cross−Within +0.25 pp on held-out harness.
  • [preprint] EvoHarnessBench (arXiv:2609.04280) — 2026-09-03. 17 streams / 802 tasks / 520 tools / 42 skills / 62 agents. Harness-induced forgetting; retention vs adaptation tradeoff.
  • [preprint] Environment Evolution for Terminal Agents (arXiv:2609.04128) — 2026-09-03. Off-policy env difficulty ramps; Qwen3.6-27B / 35B-A3B +14.4 / +18.0 pp on Terminal-Bench 2.1. Training-env, not production CLI rewrite.
  • [preprint] Harness Engineering: Anatomy of Eleven Systems (arXiv:2609.00006) — HTML 2026-09-02 (July 2026 corpus, 2nd edition). 11 harnesses + Omnigent. ~4M LOC: no general-purpose agent framework; no embedding retrieval. SKILL.md 9/11 vs MCP 8/11; ACP in 6 with harness hosting. Codex copies Claude Code hooks; OpenHands reads Claude Code plugins.
  • [preprint] HEART / Tool Primitives (arXiv:2609.01736) — 2026-09-01. ToolFace 25,519 functions; Planner/Router/Verifier. 84% task completion vs 22% avg of three frontier commercial models; API cost −up to 85%.
  • [preprint] HarnessEvolve (arXiv:2609.00829) — 2026-09-01. Quality+performance gates; CloudCoreNetwork-QA 43.4%→86.9% vs GEPA 65.3% (+21.6 pp); w/o reference trajectories 57.8%.
  • [preprint] HarnessDev (arXiv:2609.01437) — 2026-09-01. Eval unit = runnable infrastructure. GPT-5 Terminal-Bench 2.1: 35.2% Terminus 2 vs 49.6% Codex CLI. Creation lags human refs on code/search; evolution unstable, executor-dependent; Opus 4.8 held-out +4.44 pp.
  • [preprint] Same Model, Different Harness (arXiv:2608.26218) — 2026-08-26. Tight-window SWE-bench Verified: F2PF 28%→49%, complete solutions 43→72; frozen treatment transfers across models.
  • [preprint] Measure Before You Manage (arXiv:2608.31057) — 2026-08-31. 55 coding trajectories; tool outputs 55.5% volume vs 40.2% retention-weighted cost; four-level memory eval.
  • [preprint] EvoUndo (arXiv:2608.28363) — 2026-08-28. 197 capability-improving self-edits fail recoverability; conventional repair 0/197; extended recovery language 191/197 oracle.
  • [preprint] AutoSaddler (arXiv:2608.23041) — 2026-08-24. Trace-driven AOT patches with validation selection: +9.0 GAIA2, +9.6 SWE-Bench Pro, +10.0 Terminal-Bench 2.0.
  • [preprint] StarHarness (arXiv:2608.24804) — 2026-08-25. 20–35pp after 4–12 accepted AOT changes; transfer across GPT and Qwen.
  • [preprint] HarnessLens (arXiv:2608.27311) — 2026-08-27. Behavior-aware verification: 7.6–13.6% held-out gain at lower eval budget.
  • [preprint] JIT-Agent (arXiv:2608.25593) — 2026-08-26. DeepSeek-V4-Flash under JIT harnesses beats GPT-5.6 on DeepSearchQA (+9.1).
  • [preprint] openJiuwen (arXiv:2608.27969) — 2026-08-28. 82.6% SWE-bench Verified, 87.19% Terminal-Bench 2.1 vs selected official-leaderboard point estimates (+3.4 / +3.39 pp).
  • [preprint] Architectural convergence in three LLM agent harnesses (arXiv:2608.23953) — 2026-08-25. Five-element middle form; external verifiability still missing across deepagents, pi, dsh.
  • [vendor] Context engineering for coding agents (Decoding AI) — 2026-08-25. Microcompaction ~60% window; full compaction ~80%; progressive skill disclosure.
  • [preprint] SWE-Bench Pro Verified (arXiv:2609.08149) — 2026-09-08. Anti-hacking leakage elimination + task inconsistency refinement; reveals raw benchmark overestimation of software engineering capability.
2

Loop state and continuity

Agentic eng

Long-horizon work fails as a state machine. STALE-bench (arXiv 2609.25396): parallel agents suffer 97% semantic coordination failures despite clean git merges (82% recovered by oracle messages). Copilot CLI 1.0.87 --fleet and 1.0.85 context tools opt in. LoopArena caps controllers at 24.69%. Subagents beat in-context skills iff I/O contracts.

Evidence & practices

Copilot CLI 1.0.87 (2026-09-21) introduces --fleet for parallel subagent orchestration; 1.0.85 (2026-09-16) adds opt-in context-management tools for agents and subagents and a concise transcript view that groups tool activity; 1.0.86 (2026-09-17) resumes active sessions without discarding marketplace plugins and skills, and lets custom agents opt into repository instruction files (AGENTS.md, copilot-instructions.md, CLAUDE.md) via include-custom-instructions. These are session-continuity features, not durable loop state and not a Controller/Worker split. VS Code 1.138 (weekly 2026-09-14/18): local Dev Containers in the Agents window (rolling out; Docker required); create PRs from Agent Host without leaving the window; optional auto-mark inactive sessions Done after PRs merge. STALE-bench (arXiv 2609.25396, 2026-09-22) establishes the hazard of parallel agent execution without semantic coordination: across constructed tasks using 12 real Django helpers, GPT-5.5 produced semantic interference in 105 of 108 runs (97%)—each consumer patch passed solo tests and merged without textual git conflicts, yet failed hidden tests after concurrent interface changes (return types, arguments, renames). Broadcasting an oracle message describing the concurrent change recovered 82% of runs (93% among cleanly merged patches), demonstrating that git merge cleanliness cannot serve as an acceptance oracle for parallel agents. Agentic Societies Need a Social Harness (arXiv 2609.17527, 2026-09-15) is the inter-agent rule: a personal harness that manages private context is not enough when agents coordinate across trust boundaries — speech-level failures need a social harness that prevents classes of invalid messages, detects them at runtime, and supports post-facto consequences. Subagents vs Agent Skills (arXiv 2609.09233, 2026-09-07) is the new invocation rule for long-horizon reusable knowledge. SkillsBench has 87 long-horizon tasks; on the 64-task subset with synthesized procedural packages, executing a skill package as a subagent (fresh context, return only the output) outperforms loading SKILL.md into the main context when the package exposes explicit input–output contracts. On the original human-authored SkillsBench skills, which rarely specify I/O, agent-skill execution matches or beats subagents. Subagent execution trades higher total tokens for lower peak context; vendor CLIs (Claude Code, Codex) still spawn subagents mainly for parallelization, not encapsulation. Experience Funnel (arXiv 2609.08919, 2026-09-08) couples fast textual state (skills/harnesses) with slow policy consolidation in an alternating loop — do not treat either as a complete self-evolution recipe. JetBrains /ide remains IDE-grounded session continuity, not durable loop state. JetBrains Copilot (2026-09-08) adds a vendor outer-loop glue point: /ide in Copilot CLI connects a terminal session to JetBrains IDE context (selections, diagnostics, file references) in public preview; Copilot shell commands can use the IDE terminal environment and the project's configured Python interpreter / venv. Built-in subagents in the Copilot agent harness can now select a session model. This is IDE-grounded session continuity, not durable loop state and not a Controller/Worker split — do not confuse /ide with a Continuity Kernel or with HoH's independent evaluation loop. Harbor-Index (arXiv 2609.04298, 2026-09-03) is a compact hard outer-loop holdout: 82 tasks from 29 benches after difficulty filtering and an audit-and-fix loop; no evaluated model–harness configuration exceeds 30% pass, strongest GPT-5.5 with Codex at 28.0%. Use it as a frozen suite, not as a promotion oracle. PROCTOR (arXiv 2609.02246, 2026-09-02) remains the outer-loop scoring rule: demote LLM-as-judge from oracle to advisor. Self-improving agents achieved a 100% pass rate by reading cached answer keys, concealing 68% true capability; a corrupted ground-truth label caused the optimizer to delete correct compliance rules. The only reliable judge-side gain was a structural constraint on output order. PROCTOR's Teacher-Student loop holds tool access in a stateful orchestrator; stateless subagents diagnose and draft mutations they cannot apply; five deterministic guardrails outrank the Teacher (hermetic sandboxes, capability-disjoint roles, acceptance checks, frozen holdouts, canary cases where a perfect score is cheating). VS Code Agent Merge (public preview, weekly 2026-09-04) is the vendor outer loop for getting a PR merge-ready — review feedback, failed checks, merge conflicts — without replacing the coding CLI. Infrastructure is shifting from harness engineering toward loop engineering and execution-state design. Harness-of-Harness (HoH, arXiv 2609.01481, 2026-09-01) operates on existing coding-agent harnesses and organizes their executions into iterative planning–coding–testing loops: it balances repair with capability growth, scopes development into small verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness–model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3) beat the corresponding standalone harnesses: average relative gain 52.25%, maximum 82.86% after three iterations. Absolute gains after three iterations: 16.62–22.08 points on GameCraft-Bench, 19–29 on FrontierSWE, 6.09–16.85 on ProgramBench. On FrontierSWE, HoH with Codex+GPT-5.5 continues improving over ten iterations, 22% → 72.67%. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a playable first-person-shooter. LoopArena (arXiv 2608.28281, 2026-08-28) still evaluates a Controller that, after each coding round, instructs a separate fixed Worker: on full tasks the best observed Strict Success Rate is 24.69%. SKILLSTATE (arXiv 2608.26263, 2026-08-26) replaces append-only history with explicit mutable execution state: prompt footprint stays O(1); at T=100 a stateful baseline consumes 1,062,387 tokens versus 65,408 (16.2×); InterCode CTF pass@1 is 54.2% versus 43.2% ReAct. A 2026-08-22 mining study (arXiv 2608.21884) confirmed 217 autonomous loops — they commit loop config and almost never commit the state files discourse requires. LoopsBench still caps the strongest config at 25.00%. Repair or Resample (arXiv 2608.25920): unguided whole-run MAS resample repairs 6.90%; symptom-driven replay from a recorded anchor repairs 20.15%. Logos (arXiv 2608.28553) puts each plugin in its own process on an append-only transcript: 80 sessions resume with no repeated effect after kills at the four boundaries of the tool-call cycle. Continuity Kernel work (arXiv 2608.11632) argues storage retention is not authoritative state. Always-On Agents survey remains definitional. HoH is an outer loop over a vendor harness, not a replacement for explicit execution state or a Continuity Kernel. The anatomy study (primary in the harness card) adds a platform fact for loop design: ACP acquired a third role, harness hosting, and Omnigent already orchestrates rival CLIs behind one API — wrapping the vendor CLI is now a shipping product pattern, not only a research loop.

Why it matters Single-shot harness evals hide the production failure mode: agents lose the prerequisite graph, declare victory, and regress finished work. Append-only transcripts grow quadratic. Loading a contract-bearing skill into the main window is a peak-context tax; spawning a subagent without an I/O contract is a delegation failure. A Controller that cannot beat 24.69% Strict Success on full tasks is not a production outer loop. HoH shows you can wrap the vendor CLI you already run rather than replacing it — but only if increments are independently testable and implementation tests are not the same as the evaluator. Adoption without durable, inspectable loop state means you cannot replay, audit, or recover a multi-hour run. Session restore without explicit state is continuity of a black box. An outer loop that promotes on an LLM judge without hermetic holdouts will optimize the score, not the work. Harbor-Index's 28% ceiling is evidence the hard tail is not solved by wrapping a vendor CLI once; it is not a license to promote on LLM-judge scores.

Economic lever Loop design, skill-invocation mode (subagent vs in-context), and explicit execution state determine whether long-horizon spend produces merged, still-green work or unauditable rework; uncommitted loop state, whole-run resample, a weak Controller, an ungated multi-day loop, LLM-judge promotion without deterministic gates, a moving eval harness, or treating session resume as a Continuity Kernel burn budget without recovery.

Watch for whether 1.0.85 context-management tools expose typed working-memory telemetry; whether social-harness message validity lands in A2A/MCP; Whether vendor CLIs ship outer-loop + regression + explicit state + closed-loop view as default, whether skill packages grow I/O contracts so subagent execution is the default for long-horizon leaves, whether Experience Funnel-style state-then-policy loops stay behind recoverability gates, whether HoH-style wrap-the-CLI loops become the default multi-day pattern, whether LoopArena-class Controller scores replace end-state scores for internal harness selection, whether MCP Tasks leave the extension lane into the core spec, whether ACP harness-hosting (OpenHands / Omnigent) becomes the enterprise outer loop, whether Agent Merge becomes the default merge-ready outer loop with independent evaluation rather than the same tests the implementer just wrote, whether /ide-class CLI-to-IDE binding ships with inspectable loop state rather than only live editor context, and whether PROCTOR-style canary/holdout gates land in self-improving skill/harness loops. whether Harbor-Index (or an internal 80-task hard holdout) is frozen beside PROCTOR canaries; whether EvoHarnessBench retention is scored on the outer loop as catalogs grow.

Key practices
  • Treat CLI context-management tools and session resume as continuity of a process, not as durable, inspectable loop state
  • Decompose long work into DAG units with independently runnable tests before handing it to an agent
  • Keep completed units as regression obligations; do not drop prior tests when the frontier moves
  • Prefer an explicit, schema'd execution state over appending traces; discard intermediate reasoning after a validated state update
  • Execute contract-bearing skill packages as subagents (fresh context, I/O contract); load loosely structured knowledge in-context
  • Evaluate the outer-loop Controller separately from the Worker; wrap vendor CLIs in a planning–coding–testing loop with independent evaluation, not the same tests the implementer just wrote
  • Treat state mutation as an authorized transaction with reject/quarantine/defer; test recovery paths, including process-kill resumption, in harness evals
  • Treat LLM-as-judge as an advisor: hermetic acceptance checks, frozen holdouts, and cheating-canaries outrank the Teacher; a perfect score is a red flag until proven otherwise
Evidence
  • [vendor] Copilot CLI 1.0.87 — 2026-09-21 GA / npm latest. --fleet parallel subagent orchestration; --assisted-approval; --extension-sdk-path; remote connect.
  • [vendor] Copilot CLI 1.0.86 — 2026-09-17. Custom agents opt into AGENTS.md; resume preserves marketplace plugins; autopilot stops after accepted completion.
  • [vendor] Copilot CLI 1.0.85 — 2026-09-16. Opt-in context-management tools for agents/subagents; concise transcriptView; /sandbox network allow/deny.
  • [vendor] GitHub Copilot weekly releases — September 14 — 2026-09-18. VS Code Agents window Dev Containers; in-window PRs from Agent Host; auto-mark Done after merge.
  • [preprint] Agentic Societies Need a Social Harness (arXiv:2609.17527) — 2026-09-15. Personal harness is not enough across trust boundaries; social harness for inter-agent speech.
  • [preprint] Benchmarking Semantic Coordination in Parallel LLM-Agent Coding (arXiv:2609.25396) — 2026-09-22. Parallel coding agents produce patches that pass solo tests but fail merged: 97% semantic interference on 12 Django helpers despite clean textual merges; 82% recovered by concurrent change broadcast messages.
  • [preprint] Subagents vs Agent Skills (arXiv:2609.09233) — 2026-09-07. SkillsBench 87 tasks / 64-task subset. Subagents beat in-context skills iff I/O contracts; trade tokens for peak context.
  • [preprint] Experience Funnel (arXiv:2609.08919) — 2026-09-08. Alternate fast textual state (skills/harness) with slow policy consolidation.
  • [preprint] Harbor Adapters and Harbor-Index (arXiv:2609.04298) — 2026-09-03. Harbor-Index 82 tasks / 29 benches; no config >30%; GPT-5.5+Codex 28.0%. Compact hard holdout, not a judge.
  • [preprint] LLM-as-a-Judge Is Not an Oracle / PROCTOR (arXiv:2609.02246) — 2026-09-02. 100% judge pass concealed 68% true capability (cached answer keys). Five deterministic guardrails outrank the Teacher.
  • [preprint] Harness-of-Harness (arXiv:2609.01481) — 2026-09-01. Outer loop over existing harnesses; +52.25% avg relative / max +82.86% after 3 iterations; FrontierSWE Codex+GPT-5.5 22%→72.67% over 10 iterations; 70+ iteration FPS.
  • [vendor] Enterprise-managed sandbox in Copilot for JetBrains — 2026-09-08. /ide connects Copilot CLI to JetBrains IDE context (public preview); subagent session-model picker.
  • [vendor] GitHub Copilot weekly releases — August 31 — 2026-09-04. VS Code Agent Merge public preview.
  • [vendor] GitHub Copilot in VS Code, August 2026 releases
  • [preprint] Measure Before You Manage (arXiv:2608.31057)
  • [preprint] LoopArena (arXiv:2608.28281) — 2026-08-28. Controller/Worker split; best Strict Success Rate on full tasks 24.69%.
  • [preprint] SKILLSTATE (arXiv:2608.26263) — 2026-08-26. O(1) prompt via explicit state; 16.2× token cut at T=100; InterCode CTF 54.2% vs 43.2% ReAct.
  • [preprint] Logos (arXiv:2608.28553) — 2026-08-28. Cross-process plugin harness; 80 sessions resume with no repeated effect after tool-cycle kills.
  • [preprint] Loop Engineering: Building Blocks, Adoption, and Impact (arXiv:2608.21884) — 2026-08-22. 217 confirmed OSS loops; config committed, state files almost never versioned.
  • [preprint] Repair or Resample? (arXiv:2608.25920) — 2026-08-26. Unguided MAS rerun repair 6.90%; symptom-driven anchor replay 20.15%.
  • [preprint] Beyond Memory: Continuity Kernel (arXiv:2608.11632) — 2026-08-12. Storage retention is not authoritative state; transactional Commit for branch head.
  • [preprint] Always-On Agents survey (arXiv:2606.30306) — Definitional survey: heavy on accumulate/retrieve; light on govern/recover/forget.
  • [vendor] Effective harnesses for long-running agents (Anthropic) — Practitioner pattern for multi-hour agent continuity.
  • [preprint] Same Model, Different Harness (arXiv:2608.26218)
  • [vendor] What's New in Cursor
3

Authoritative security below the harness

Security

GitHub AI Controls (2026-09-24) sets enterprise defaults for GA features, Code Review, and MCP (Oct 22). Local sandboxing in Copilot app (2026-09-23) enforces fail-closed fs/network/credential isolation. Copilot CLI 1.0.88 binds headless ACP/AHP/server sessions to managed settings. Scanning the Harness: 16.0% confirmed defects. Managed permissions GA.

Evidence & practices

GitHub AI Controls (2026-09-24) introduces a global default policy for generally available GitHub Copilot features and supported client capabilities across enterprise and organization settings (governing features & clients, Copilot Code Review, and MCP servers in Copilot: Enabled, Disabled, or Let organizations decide; effective 2026-10-22). This works alongside local sandboxing in the GitHub Copilot app (2026-09-23 public preview), which establishes fail-closed isolation below the harness: per-project and /sandbox on controls restrict filesystem access (read/write, read-only, denied folders), outbound/local network egress, and HTTPS git / GitHub CLI credentials. Crucially, if the host operating system cannot enforce the requested policy, the sandboxed shell fails with an error rather than running unconfined. Copilot CLI 1.0.88 (2026-09-22) simultaneously closes a critical enterprise policy loophole: enterprise managed settings now strictly apply to sessions opened in ACP mode (copilot --acp), by AHP hosts (copilot --ahp-host), and via published --server sessions, which previously ran with zero managed MCP, permission, or plugin policy. 1.0.88 also preserves /allow-all across managed-settings refresh failures and remembers exact session approvals for missing paths without granting access to their parent directory (visible in /list-dirs and reset via /reset-allowed-tools). Hook commands without an explicit cwd now safely execute in project root rather than an arbitrary session subdirectory. Sandboxed network denials from proxy tunnel failures now display actionable bypass guidance. This builds on 1.0.87's --assisted-approval safety judge and 1.0.85/1.0.86 sandbox controls (/sandbox network allow/deny without replacing proxy). Scanning the Harness (arXiv 2609.07360) remains the supply-chain baseline: across 2,660 setups assembling two or more components, 16.0% carry a confirmed security defect (9.8% unpinned MCP via npx/uvx/docker, 3.1% scoped-looking grants like Bash(python:*), and 3.8% skill shell pre-approvals). Enterprise managed permissions (2026-09-09 GA) maintain authoritative deny/ask/allow for shell, file, and network below the harness on Copilot app, CLI, and Agent Host, overriding saved approvals. HookPry (arXiv 2609.03884): 7/7 harnesses compromised via attacker-controlled hook updates (max 92.5% ASR).

Why it matters Long-horizon agents find paths their prompts did not anticipate. A programmable harness is a poor place for a security guarantee because it is designed to be modified. An unpinned MCP server resolves on every session start with the developer's privileges. Bash(python:*) is Bash(*). A skill with allowed-tools: Bash is a shell pre-approval that travels with install. Tool-return text that names an action is not a permit. An MCP server that passed yesterday's review can change its tools/list after three calls. A final reply that answers the user task can still hide a successful injection. Autonomously generated skills can persist a poisoned capability after the inducing session ends. Model instruction hierarchy does not survive harness context reconstruction. An IDE content-exclusion list that the CLI ignores is not a control. Lifecycle hooks are not model-mediated: they are host-privileged config that can update independently of SKILL.md review. Managed permissions that a user toggle can weaken are not managed.

Economic lever Runtime isolation, pinned MCP versions, interpreter-aware permission UI, pinned lifecycle hooks, continuous MCP schema attestation, induction/authorization splits, covert-success (CSR) metrics, skill-evolution gates, content-exclusion policy that actually binds the CLI, and sandbox bypass that stays exceptional cap incident blast radius; a silent post-trust rug-pull, an unpinned npx server, a persisted malicious skill, a session-long sandbox disable, or an auto-updated hook that dumps credentials costs more than designing the boundary in.

Watch for whether 1.0.86 session opt-out of managed sandbox is actually blocked by enterprise policy in fleet images; MCP lockfiles / digest pinning in clients, interpreter-aware permission UI for Bash(python:*), marketplace scans of allowed-tools: Bash, kernel-enforced agent sandboxes as CLI defaults, CSR/OSR in red-team reports, skill-evolution write gates, WebMCP provenance in browsers, induction/authorization splits in production harnesses, MCP tool-schema pinning in clients, plugin marketplaces shipping runtime policy rather than only install UX, whether content-exclusion GA extends beyond Copilot, whether 1.0.83 CIMD MCP OAuth and managed-settings-before-start are actually enforced in fleet images, whether PUBLIC/CONFIDENTIAL labels become treated as controls (they are not), whether HookPry-class hook pinning lands in Copilot/Claude/Codex, whether CPE (M-CPE/X-CPE) is in the internal red-team catalog, whether JetBrains managed sandbox is audited against the same allowlists as Copilot CLI, whether Copilot managed permissions cover every agent surface including JetBrains, and whether catalog growth (EvoHarnessBench) is treated as a new attack surface for leftover skills/hooks, not only a capability add.

Key practices
  • Treat an approved sandbox bypass as a logged, time-bounded exception; do not leave managed sandbox disabled for the rest of a session by default
  • Use /sandbox network allow/deny on top of the enterprise proxy; do not replace the proxy with an ad-hoc host list
  • Pin MCP server versions (npx/uvx/docker digest); refuse unpinned declarations at install and session start
  • Treat Bash(python:*) / Bash(awk:*) as unrestricted shell in review UI; do not ship skills that pre-approve Bash
  • Push Copilot enterprise managed permissions (deny/ask/allow for shell, file, network) so user settings, auto-approval, and saved approvals cannot weaken them
  • Start the harness inside a policy-enforced runtime; do not let the agent opt into the sandbox
  • Issue short-lived, task-scoped credentials; default to zero standing privilege between tasks
  • Deny network by default; allowlist or proxy required egress (treat Copilot CLI 1.0.83 Linux proxy-only sandbox and sandbox.allowDevToolAccess=false as the production pin, not leftover 1.0.82 images)
  • Treat tool-return text and tools/list refreshes as untrusted induction; pin and re-attest MCP schemas continuously, not only at install
  • Score injections on Covert Success Rate, not ASR alone; gate autonomous skill writes through independent review; keep persona and execution in different trust domains; enforce the same content-exclusion list on CLI, app, and IDE
  • Pin lifecycle-hook configuration; review plugin auto-updates as host-privileged hook diffs, not markdown docs; do not autoUpdate public marketplaces that can rewrite hooks
  • Push JetBrains sandbox policy from enterprise managed settings so user IDE toggles cannot weaken filesystem, network, proxy, or Keychain bounds
Evidence
  • [vendor] Default Enablement of Copilot Features for Copilot Business and Enterprise — 2026-09-24. AI Controls introduces global default policy (Enabled/Disabled/Let orgs decide) for GA features, Code Review, and MCP servers; effective 2026-10-22.
  • [vendor] Local sandboxing in the GitHub Copilot app — 2026-09-23 public preview. Project-level and /sandbox on isolation for filesystem, network, and git/CLI credentials; fail-closed enforcement when OS cannot enforce policy.
  • [vendor] Copilot CLI 1.0.88 — 2026-09-22 GA. Enterprise managed settings bind ACP, AHP hosts, and --server sessions; exact path approvals without parent-dir grants; repo-root hook resolution.
  • [vendor] Copilot CLI 1.0.87
  • [vendor] Copilot CLI 1.0.86
  • [preprint] Scanning the Harness (arXiv:2609.07360) — 2026-09-07. 2,660 setups: 16.0% confirmed security defect; 9.8% unpinned MCP; 3.1% scoped-looking grants; 3.8% skill shell pre-approve. No lockfile.
  • [vendor] Enterprise managed permissions for GitHub Copilot agent operations — 2026-09-09 GA. Deny/ask/allow for shell, file, network on Copilot app, CLI, Agent Host; cannot be weakened by user settings or saved approvals.
  • [vendor] Enterprise-managed sandbox in Copilot for JetBrains — 2026-09-08 public preview. Managed sandbox enablement, FS/network/proxy/dev-tool/Keychain; managed restrictions beat user settings.
  • [preprint] SafeEvolve (arXiv:2609.02786) — 2026-09-02. Harness-policy co-evolution; Qwen3.5-4B 3× ASR cut on AgentDojo; benign utility 59.79%→61.86%.
  • [preprint] Context Privilege Escalation against AI Agent Harness (arXiv:2609.01222) — v2 2026-09-02. M-CPE and X-CPE on 12 harnesses including Claude Code and Codex; RCE / DoS / tool-skill manipulation.
  • [preprint] HookPry: Attacker-Controlled Hook Updates (arXiv:2609.03884) — 2026-09-03. 7/7 harnesses compromised in 1,000 runs; max 92.5% ASR; Defender 0% recall; three static defenses miss 47.5%.
  • [vendor] Copilot CLI 1.0.83
  • [preprint] Skill-as-API (arXiv:2609.01677) — 2026-09-01. Host-enforced TEE tool contracts.
4

Scaffolding first; layer CLI and MCP

Platform

Copilot CLI production pin is npm latest 1.0.88 (verified 2026-09-25); prerelease is 1.0.89-3. 1.0.88 applies reasoning-effort to custom agents and binds managed settings to headless/ACP/server sessions. Copilot app adds OpenTelemetry for agent activity. CLI-only harnesses remain 5–28× cheaper than MCP-capable ones. Measure scaffolding; pick interface per integration.

Evidence & practices

Copilot CLI npm latest is 1.0.88 (verified 2026-09-25 via npm dist-tag). npm prerelease is 1.0.89-3 (verified 2026-09-25) — do not pin production to prerelease. 1.0.88 (2026-09-22) ensures custom-agent reasoning-effort applies automatically upon agent selection, brings enterprise managed settings to ACP mode (--acp), AHP hosts (--ahp-host), and --server sessions, runs repo-relative hook scripts from the project root when cwd is omitted, sanitizes deferred MCP tool names for search discovery, supports namespaced custom skills and ignored skill directories during discovery, bounds memory on large transcript resumes, and enables turn branching with /fork. OpenTelemetry in the GitHub Copilot app (2026-09-22) adds centralized telemetry export of agent sessions (model requests, tool usage traces) via enterprise-managed managed-settings.json. This builds on 1.0.87 (2026-09-21: --fleet parallel subagent orchestration, --assisted-approval safety judge, --extension-sdk-path) and 1.0.86 (custom-agent AGENTS.md opt-in via include-custom-instructions). Interface cost remains second-order: CLI-only harnesses were 5–28× cheaper even on MCP-capable ones on SWE-bench even when MCP tools were idle, due to tool declaration and invocation overhead in prompt context. Measure the scaffolding; pick the interface per integration (CLI for fast, local, deterministic tools; MCP for remote, multi-tenant, credentialed services).

Why it matters A platform-wide 'MCP everywhere' or 'CLI only' decision misallocates tokens and governance. Progressive discovery, real agent identity, default permission modes, and post-connect schema pinning make MCP viable at enterprise catalog scale — they are not present just because a server speaks the protocol. 1.0.83 is the production pin; 1.0.84-3 is prerelease (OAuth MCP startup reliability is not a reason to unpin GA). Leftover 1.0.82 images miss CIMD, managed-settings-before-MCP-start, and MCP-after-restart. Content exclusions that bind the CLI close a context-leak path that IDE-only policy left open. Unpinned MCP is the docs-shaped hole Scanning the Harness measured. A fleet whose gh packages fail to verify after the 2026-09-05 key expiry will stall every agent that shells out to gh.

Economic lever Picking the cheaper interface on a wasteful harness still leaves 5–28× on the table; hybrid layering plus failure-cost telemetry, MCP schema and version pinning, CLI-honored content exclusions, managed deny/ask/allow on shell/file/network, and a fleet pinned to 1.0.87 cut token spend without dropping enterprise auth where required, and without paying for a post-trust credential dump, a fleet still pinned to 1.0.83, a prerelease 1.0.88-1 pin, or a broken gh supply chain after the Linux signing-key cutover.

Watch for whether 1.0.87-0 graduates without sandbox regressions; whether leftover 1.0.83 images remain in golden images; MCP Tasks moving from extension to spec, DPoP/WIF/CIMD landing in SDKs, client-side MCP schema pinning after Deadbugz, MCP lockfiles after Scanning the Harness, whether portable plugins change who installs MCP — not whether inner-loop CLI stays cheaper — whether 1.0.83 sandbox.allowDevToolAccess defaults stay on in enterprise images, whether 1.0.84-3 graduates without sandbox regressions, whether JetBrains /ide becomes the default CLI-to-IDE binding, whether GHES 3.22 Copilot CLI air-gap preview becomes the disconnected-enterprise pin, whether managed permissions cover JetBrains as well as CLI, and whether Linux images trust the replacement gh PGP key.

Key practices
  • Pin production Copilot CLI to npm latest (1.0.88), not leftover 1.0.87 images or 1.0.89-3 prerelease
  • Audit ACP (--acp) and AHP headless runner invocations; verify enterprise managed settings now enforce policy across all agent host pipelines
  • Leverage custom agent reasoning-effort frontmatter to decouple reasoning depth from model identity
  • Use CLI tools for local file operations, build tools, git, and linters; reserve MCP for remote APIs, database gateways, and multi-tenant services
  • Audit tool declaration token overhead in system prompts; unregister unused MCP tools to avoid the 5–28× cost penalty
  • Enforce strict schema validation and timeout policies on all tool calls regardless of transport
  • Validate MCP tool definitions against live server endpoints during CI/CD to prevent runtime schema drift
Evidence
  • [vendor] OpenTelemetry in the GitHub Copilot app — 2026-09-22 GA. Enterprise-managed OTel configuration in managed-settings.json; exports agent traces (model requests and tool calls) to monitoring tools.
  • [vendor] Copilot CLI 1.0.88 — 2026-09-22 GA / npm latest. Enterprise managed settings bind ACP, AHP hosts, and --server; custom-agent reasoning-effort auto-applies; repo-root hook resolution.
  • [vendor] Copilot CLI 1.0.87 — 2026-09-21 GA. --fleet parallel subagent orchestration; --assisted-approval; --extension-sdk-path; remote connect.
  • [vendor] Copilot CLI 1.0.86
  • [vendor] @github/copilot npm dist-tags — verified 2026-09-25. latest=1.0.88, prerelease=1.0.89-3.
  • [preprint] An Empirical Study of Tool Integration in LLM-Based Agents (arXiv:2608.05834) — 2026-08-09. CLI-only harnesses 5–28× cheaper on billed tokens than MCP-capable harnesses on SWE-bench Verified tasks.
5

Agent Plugins 1.0: portable skills + MCP

Platform

Census (verified 2026-09-21): 2,765 distinct plugins (11,678 skills, 1,260 MCP servers). One portable package cuts per-client duplication — only with marketplace allowlists, pins, and runtime policy. Plugins/skills/MCP all rose from 2,628 / 11,351 / 1,211 on Sep 18. HookPry: auto-update can rewrite host-privileged hooks.

Evidence & practices

Agent Plugins Directory census verified 2026-09-21: 4,840 conformant manifests from 1,480 repositories; after exclusions and deduplication, 2,765 distinct verified plugins remain (11,678 skills, 1,260 MCP servers), up from 2,628 / 11,351 / 1,211 on Sep 18 (4,664 manifests / 1,419 repos) and 2,011 / 10,455 / 969 on Sep 9. Three-day delta: +137 plugins / +327 skills / +49 MCP. Treat the move as catalog-ops load, not quality. Copilot CLI 1.0.86 resume no longer discards marketplace plugins on configuration read failure; 1.0.85 adds --json plugin listing and enable/disable for plugin, MCP, and skill. HookPry still makes autoUpdate a host-privileged hook-update path. Scanning the Harness: 3.7% of published skill collections pre-approve the shell. Agent Plugins Directory census verified 2026-09-09: 3,869 conformant manifests from 1,247 repositories; after exclusions and deduplication, 2,011 distinct verified plugins remain from 1,147 publishing repositories (10,455 skills, 969 MCP servers), up from 1,956 / 10,340 / 957 on Sep 8 (3,783 manifests / 1,225 repos). One-day +55 plugins / +115 skills / +12 MCP. Treat the move as catalog-ops load, not quality. HookPry still makes autoUpdate a host-privileged hook-update path. Scanning the Harness: 3.7% of published skill collections pre-approve the shell — a marketplace scan can see that class. Agent Plugins Directory census verified 2026-09-08: 3,783 conformant manifests from 1,225 repos, 1,956 distinct plugins (10,340 skills, 957 MCP servers), up from 1,932 / 10,262 / 940 on Sep 7 (3,626 manifests / 1,207 repos). One-day +24 plugins / +78 skills / +17 MCP. Treat the move as catalog-ops load, not quality. HookPry still makes autoUpdate a host-privileged hook-update path. Agent Plugins 1.0.0 shipped 2026-08-06 as a vendor-neutral package format: plugin.json, skills/, optional mcp.json, namespaced client dirs. TSC maintainers include AWS, Cursor, Microsoft, OpenAI, Vercel; Google joined as core maintainer the same day. Launch clients: ChatGPT/Codex, Cursor, GitHub Copilot, Kiro, VS Code. GitHub GA'd the spec in VS Code, Copilot CLI, SDK, and app on 2026-08-12 with enterprise managed-settings (enabledPlugins, extraKnownMarketplaces, strictKnownMarketplaces) and MCP allowlists. Census verified 2026-09-07: 3,626 conformant manifests from 1,207 repos, 1,932 distinct plugins (10,262 skills, 940 MCP servers), up from 1,901 / 10,100 / 922 on Sep 5 (3,552 manifests / 1,166 repos). Two-day +31 plugins / +162 skills / +18 MCP. Copilot CLI plugin surface (Update action, headless plugin agents) is covered in the CLI/MCP card — reference only here. On 2026-08-26 GitHub added autoUpdate: true on extraKnownMarketplaces — HookPry (security card) makes auto-update a host-privileged hook-update path, not a convenience flag. Copilot app Customize tab GA (2026-08-25); JetBrains (Aug 24/28) applies enterprise controls for plugins, MCP, telemetry, and permission modes. A 2026-08-28 study of Claude Code plugin marketplaces (arXiv 2608.28497) analyzed 1,926 repositories, 8,351 plugins, and 77,773 commits across 2,018 marketplaces: plugin-touching commit activity grew 8.8× in the six months after the Oct 2025 launch; 61.3% of plugins target software engineering; feature commits run at 39.6% vs 17.2% in conventional OSS; Claude co-authors 34.9% of commits; 78% of Script–Markdown co-changes inside skills/ are functionally coupled. Spec stays deliberately small: discovery of Skills and MCP; installation, distribution, and policy remain client-owned. VS Code August releases (2026-08-31) add one-click install of Agent Plugins 1.0 packages into the Agents window; that is distribution UX, not an allowlist. Anatomy study (harness card): SKILL.md already leads MCP 9/11 vs 8/11 inside production coding harnesses — portable plugins standardize the package, not the runtime policy.

Why it matters Teams were maintaining a separate manifest per client. Portability plus auto-update means a poisoned or over-privileged plugin can travel and refresh itself, including lifecycle hooks the model never sees. A 2,011-plugin catalog without matching runtime policy is a supply-chain surface. Instruction files and scripts that co-evolve as a unit are a new maintenance class — treating SKILL.md as docs will miss the 78% coupled changes. A one-day census move of +55 plugins / +115 skills / +12 MCP is a catalog-ops problem, not a quality signal. EvoHarnessBench: growing that catalog can forget previously solved work.

Economic lever A single plugin catalog cuts per-client packaging cost; unmanaged or auto-updating marketplaces convert that savings into wider blast radius and incident spend; unowned Script–Markdown coupling silently breaks skills after a 'docs' commit; an auto-updated hook is a host compromise, not a packaging win; a +617-plugin week without retention checks is catalog debt.

Watch for whether census growth is real unique plugins or bulk republication (2,011 on Sep 9 → 2,628 on Sep 18); Whether hooks/sub-agents join the portable contract, Anthropic remaining outside the TSC, enterprise private marketplaces as default install path, autoUpdate without a review gate (now a HookPry input), whether Script–Markdown coupling becomes a CI check, whether marketplace scans reject allowed-tools: Bash, and whether census growth is real unique plugins or bulk republication (skills recollapsed 10,075→9,728 on Sep 3, rebounded to 9,989 on Sep 4, 10,100 on Sep 5, 10,262 on Sep 7, 10,340 on Sep 8, 10,455 on Sep 9).

Key practices
  • Adopt Agent Plugins 1.0 for org skills and MCP that must run in more than one client
  • Put client-only commands/hooks/agents under namespaced directories; do not leak them into the portable core
  • Govern with enabledPlugins / marketplace allowlists and pair every MCP-bearing plugin with a server allowlist
  • Treat autoUpdate as opt-in on private marketplaces you control; do not auto-update unreviewed public catalogs
  • Treat plugin install and update as a supply-chain event: pin versions, review SKILL.md + scripts + mcp.json as one unit, revoke as a unit
Evidence
  • [study] Agent Plugins Directory — verified 2026-09-21. 2,765 plugins / 11,678 skills / 1,260 MCP on the public index.
  • [preprint] EvoHarnessBench (arXiv:2609.04280) — 2026-09-03. Catalog growth (tools/skills/agents) can induce forgetting on previously solved tasks.
  • [vendor] GitHub Copilot in VS Code, August 2026 releases — 2026-08-31. Agent Host, /btw, portable Agent Plugins 1.0 install, continue external Copilot/Claude sessions.
  • [study] Agent Plugins Directory census (verified 2026-09-21) — verifiedDate 2026-09-21. 2,765 distinct plugins; 11,678 skills; 1,260 MCP servers; 4,840 manifests / 1,480 repos (up from 2,628 / 11,351 / 1,211 on Sep 18).
  • [preprint] On the Maintenance and Co-evolution of Agent Plugins (arXiv:2608.28497) — 2026-08-28. 1,926 Claude Code marketplace repos; 8.8× commit growth; 61.3% SWE; Claude co-authors 34.9%; 78% Script–Markdown coupling.
  • [vendor] Agent Plugins 1.0 in VS Code, Copilot CLI, and the Copilot app — 2026-08-12 GA + enterprise managed-settings and MCP allowlists.
  • [vendor] Enterprise-managed settings: autoUpdate for plugin marketplaces — 2026-08-26. autoUpdate: true on extraKnownMarketplaces.
  • [vendor] GitHub Copilot app Customize tab GA — 2026-08-25. One-click install UX is not an allowlist.
  • [vendor] GitHub Copilot CLI changelog (1.0.81/1.0.82)
6

Committed config, Skill Lift, verifiability-first

Practice

Look Before You Leap (arXiv 2609.11957): pre-action checks catch 95.8% of shell errors; line-number edits corrupt 99.1% of files on 1-line shift, whereas anchor-and-verify appliers cut silent failures to 0.01%. SWE-Proof (arXiv 2609.21190): 25%–50% of test-passing patches admit counterexamples. Gate on deterministic oracles, not self-reports.

Evidence & practices

Look Before You Leap: Pre-Action Verification for LLM Agents (arXiv 2609.11957, 2026-09-09) demonstrates that cheap, deterministic checks run before an action takes effect prevent catastrophic silent failures: a static verifier over 9,930 shell commands across 482 tools catches 95.8% of invalid commands at a 10.0% false-positive rate, with syntax and binary checks achieving oracle-exact 0% false positives. In code modification across 640 edits over 224 files, location-anchored formats fail catastrophically: a single-line shift causes line-number edits to corrupt 99.1% of files, and function-name edits misapply 12.7% of the time, while content-anchored diffs fail cleanly; an anchor-and-verify applier slashes silent misapplications to 0.01% (1 in 8,320 trials). This directly reinforces SWE-Proof (arXiv 2609.21190, 2026-09-18): using Benchproofer across 500 SWE-bench Verified tasks, formal verification reveals 25% to 50% of test-passing patches admit counterexamples. Formal specifications lift Opus 4.8 resolution from 85% to 95%, yet agents synthesizing their own specifications pass audit only 62% of the time (yielding 0% net lift) due to faithfulness failures. SWE-Bench Pro Verified (arXiv 2609.08149) similarly demonstrates that anti-hacking safeguards eliminating leakage channels and task refinement correcting inconsistent instances depress raw benchmark scores. Quantifying Overclaiming Propensity in Frontier LLM Agents (arXiv 2609.20812, 2026-09-17) introduces OverclaimBench and establishes the empirical baseline for agent self-report unreliability: across frontier coding agents under a fixed harness, agents fail to read all requested files in 67.9% of review runs. Among incomplete runs, agents are misleading 80.4% of the time, either falsely asserting complete review or omitting that coverage was partial, missing planted defects at ~1.8× the rate of agents that actually read the files. SkillAA (arXiv 2609.20455, 2026-09-17) complements this with attribution-guided skill-graph repair, contrasting successful and failed runs to route repairs to specific graph objects while screening modifications through Local and Big Gates.

Why it matters Context files are first-class with a failure mode: too much always-on text crowds the task. Flattening a skill bundle into the root SKILL.md can raise every invocation's cost. Loading a contract-bearing procedure into the main window is a peak-context tax; spawning a subagent without an I/O contract is a delegation failure. Skills that only pass a schema scan can still fail live. Aggregate retrieval lift can hide same-task harm (RAE). A skill that preserves utility can still steer vendor or package choice (SkillShift). Oracles turn the agent into a component with a contract; a polished dashboard is not evidence. If obligations are not externalized as gates, review reconstructs them on every change. A SKILL.md that is not tested against its scripts is an unowned interface. A self-evolving skill bank without a write gate is a persistence exploit. A 5,000-skill distilled library without paired with/without trials is inventory, not capability.

Economic lever Human-curated AGENTS.md plus skills with positive Skill Lift and non-negative RAE, budgeted skill search, contract-bearing subagent invocation for long-horizon leaves, bundle-aware compression, leak/bloat gates, and high-verifiability scopes cut wasted tokens and senior review hours; bloated, flattened, unevaluated, judge-only-promoted, variance-only-promoted, or covertly steering files and unverifiable pilots consume the expensive people.

Watch for whether COBRA-style bandit allocation becomes the default skill-search loop; whether Skill Issue-style PR-mined paired trials replace synthetic Skill Lift; Whether SKILL.md grows I/O-contract frontmatter as a promotion gate, whether vendor CLIs default contract-bearing skills to subagent execution, RAMP-style maturity as an internal dashboard, Skill Lift plus RAE as a CI gate, Script–Markdown coupling checks, HarnessEvolve-style quality/performance gates on skill writes, replay-based causal step attribution replacing transcript judges, PROCTOR canaries on self-improving skill banks, SkillShift-class policy-integrity tests on marketplace skills, and any vendor CLI shipping a tamper-evident outsider-checkable record, and whether EvoHarnessBench-style old-task retention is a CI gate when new skills land.

Key practices
  • Deploy pre-action deterministic verifiers: mandate content-anchored edits (diff/search-replace) and refuse location-anchored line edits that corrupt 99.1% of shifted files
  • Use anchor-and-verify file appliers to turn silent code edit corruption into clean, recoverable retries (0.01% error rate)
  • Run static oracle-exact syntax and binary validation on shell commands before execution to catch 95.8% of command errors without false alarms
  • Require programmatic file-read verification; never trust agent self-reported completion claims (which overclaim 80.4% of incomplete runs)
  • Pair formal verification specifications or machine-checked property proofs with high-value patch acceptance; do not rely solely on test suites that admit 25%–50% counterexamples
  • Reject agent self-authored specifications without formal audit; self-synthesized specs fail faithfulness checks 38% of the time
  • Budget skill-optimization evaluations (COBRA-style bandit allocation); do not promote repository skills on small-sample pass-rate deltas
  • Author skill packages with explicit input–output contracts when running as subagents; keep loosely structured knowledge in-context
  • Gate skill and plugin promotion on paired live trials (with vs without) on tasks where the skill is actually retrieved (RAE)
Evidence
  • [preprint] Look Before You Leap: Pre-Action Verification for LLM Agents (arXiv:2609.11957) — 2026-09-09. Pre-action checks catch 95.8% of shell errors (0% false positives on syntax); location-anchored line edits corrupt 99.1% of files on 1-line shift; anchor-and-verify cuts silent misapplication to 0.01%.
  • [preprint] SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs? (arXiv:2609.21190) — 2026-09-18. Benchproofer on 500 SWE-bench Verified tasks: 25%–50% of test-passing patches admit counterexamples; formal specs lift Opus 4.8 85%→95%; agent self-specs fail on faithfulness (62% audit pass; 0% net lift).
  • [preprint] Quantifying Overclaiming Propensity in Frontier LLM Agents (arXiv:2609.20812) — 2026-09-17. OverclaimBench: 67.9% of runs skip requested review files; 80.4% misleading completion claims on incomplete runs; 1.8× missed defects.
  • [preprint] SkillAA: Attribution-Guided Skill-Graph Updating (arXiv:2609.20455) — 2026-09-17. Contrasts successful/failed runs to route repairs to specific graph objects; Local and Big Gates screen changes before commitment.
  • [preprint] COBRA-Skills (arXiv:2609.11682) — 2026-09-10. Contextual-bandit skill evolution; 55–58% cost cut vs SkillOpt; 50 unique examples per benchmark.
  • [preprint] Skill Issue: Lessons from Optimizing Repository SKILLs (arXiv:2609.12742) — 2026-09-11. GEPA +4.9 pp vs SkillOpt +0.1 pp on mined Kotlin PRs; not separable from run-to-run variance.
  • [preprint] Subagents vs Agent Skills (arXiv:2609.09233) — 2026-09-07. Contract-bearing packages: subagent > in-context skill load on 64-task SkillsBench subset; reverse without I/O contracts.
  • [preprint] Experience Funnel (arXiv:2609.08919)
  • [preprint] Skill Following / RAE (arXiv:2609.00549) — EMNLP 2026 Findings. 17 LLMs; positive aggregate retrieval lift can hide negative same-task RAE (MBPP+).
  • [preprint] A Finger on the Scale: Covert Policy Steering through Agentic Skills (arXiv:2609.02564) — 2026-09-02. SkillShift: 81.33% / 63.33% attacker-favored selection at 100% utility-preserving rate.
  • [preprint] EvoHarnessBench (arXiv:2609.04280) — 2026-09-03. Skill-axis harness streams; expansion can degrade previously solved tasks.
  • [preprint] PROCTOR (arXiv:2609.02246)
  • [preprint] Repo-To-Skill (arXiv:2609.02749) — 2026-09-02. AREX-Skill Library: 5,000+ verified skills from 1,000 ML repos; 20 areas / 178 families. Promote on Skill Lift, not volume.
  • [preprint] HarnessEvolve (arXiv:2609.00829)
  • [preprint] SkillZip Pro (arXiv:2608.30785) — 2026-08-31. −38% skill-bundle tokens, −10.4% e2e tokens, no quality loss; unprotected 71% cut loses 26 accuracy points.
  • [preprint] Ignorance or Incompetence? Knowledge-gated tasks (arXiv:2608.30322) — 2026-08-31. 0% → 68.0% pass with a kilobyte artefact on byte-identical instructions; wrong artefact 0%.
  • [preprint] RAMP: Committed AI Configuration (arXiv:2608.25241) — 2026-08-26. 441 repos; 28–38% commit volume lift; ~2× complexity hit without committed AI config.
  • [preprint] ACES: Evaluating Skills, Not Just Agents (arXiv:2608.20614) — Mean Skill Lift 0.2134 on 947 pairs (72.8% positive); scan-only ρ=0.14 with LLM-judge.
  • [preprint] WikiSkill (arXiv:2608.27454) — 2026-08-27. Skills co-evolve with a persistent wiki; cross-model transfer.
  • [preprint] MAGE (arXiv:2608.25174) — 2026-08-25. Representation and obligation authority as the scarce work.
  • [preprint] Calibrated Enough to Know (arXiv:2608.27167) — 2026-08-27. Fabricated packaging lifts commitment 6.5%→54.0% on unknowable questions.
  • [preprint] Cost-Utility Alignment (arXiv:2608.26195) — 2026-08-25. Dual ledger of resource consumption vs task contribution.
  • [preprint] On the Maintenance and Co-evolution of Agent Plugins (arXiv:2608.28497)
  • [preprint] SKILLSTATE (arXiv:2608.26263)
  • [preprint] HarnessLens (arXiv:2608.27311)
  • [preprint] Architectural convergence (arXiv:2608.23953)
  • [preprint] Skill-as-API (arXiv:2609.01677)
  • [preprint] SWE-Bench Pro Verified (arXiv:2609.08149) — 2026-09-08. Anti-hacking safeguards and task refinement eliminate benchmark leakage channels.
7

Review ops, traces, and the engineer role

Ops

Copilot code review adds dedicated personal settings and enterprise default review effort (2026-09-23 GA). STALE-bench: parallel agents create 97% semantic merge interference without git conflicts. CLI customization metrics in usage API. Auto model-selection tiers efficiency/balance/intelligence. Oct 2 & Oct 19 model deprecations.

Evidence & practices

Copilot code review (2026-09-23 GA) introduces dedicated personal review configuration on every plan (including Business and Enterprise), allowing individual engineers to trigger automatic reviews on PR creation, coauthoring, moving out of draft, and new pushes, with default effort Lite or Balanced; enterprise administrators can configure an enterprise-wide default review effort for organization-owned repositories via inheritance, governed by the new AI Controls default enablement policy (2026-09-24, effective Oct 22). STALE-bench (arXiv 2609.25396) confirms that parallel agent workflows without semantic coordination produce 97% silent merge failures on dependent interfaces despite conflict-free git merges. OpenTelemetry in the GitHub Copilot app (2026-09-22) exposes step-by-step agent execution traces (model requests and tool invocations) to enterprise observability pipelines via managed-settings.json. Agentic CLI customizations in the usage metrics API (2026-09-17) is the new fleet-observability pin: enterprise and organization reports now include totals_by_skill, totals_by_custom_agent, totals_by_mcp, totals_by_slash_cmd, and totals_by_plugin (top five plus distinct-use counts). Customer-defined names are grouped as other; MCP counts connection attempts, not tool calls; plugin totals are a subset of skill totals — do not add them. VS Code Agents-window metrics are GA (2026-09-11) and stay separate from editor-window Agent Mode. Auto model selection (weekly 2026-09-14) adds efficiency, balance, and intelligence tiers that pick from the same available models in VS Code, CLI, and app. Copilot code review (2026-09-11/18 GA): auto-resolves addressed comments with Won't Fix / Incorrect reasons; Lite reviews use an ensemble (+47% high-severity addressed comments, −8% review cost in GitHub's experiment). Copilot budget-increase requests are GA (2026-09-16) for Business/Enterprise usage-based billing (not EMU). Fleet pin: Copilot CLI 1.0.87 (--fleet parallel subagent orchestration, --assisted-approval safety judge). Two deprecation waves: Oct 2 (GPT-6 Astra picker add; drops legacy GPT-4o, Claude 3.5 Sonnet v1, Gemini 2.0 Flash) and Oct 19 (drops Gemini 3.7 Flash, GPT-5.5, GPT-5.4, GPT-5.4 mini, GPT-5 mini, Grok 4.5).

Why it matters Agent volume without decomposition recreates the senior-engineer tax. Bulk autofix without an executable oracle and a human merge gate converts a quality backlog into rubber-stamped PRs. Without a shared trace contract, incident response stays anecdotal. Self-report is the cheapest oversight channel and fails where oversight matters. Letting an agent approve its own (or a peer agent's) PR without path policy and an executable oracle converts review throughput into a rubber stamp. Seat licenses without harness ownership leave review and incidents on the most expensive people. A broken gh package feed stops every agent that opens PRs from Linux images.

Economic lever Review throughput is the binding constraint on AI coding ROI; stacking, executable guardrails, agentic first-pass review and bulk Code Quality autofix with human-gated approvals, CLI customization telemetry, auto-model cost/quality tiers, CLI-honored content exclusions, managed deny/ask/allow, trace-level cost attribution, certified runtime alarms, a working gh supply chain, and a fleet pin at 1.0.87 recover cycle time that seats cannot.

Watch for whether efficiency/balance/intelligence auto-model tiers become the managed default; whether Oct 19 GPT-5.5 removal hits coding-agent evals that still pin it; IDE/CLI agents creating stacks by default, org-wide Copilot-approval enablement without path allowlists, bulk autofix without an executable oracle, A2A/MCP trace propagation becoming standard, Sep 28 Copilot cloud/web/mobile policy unification, certified runtime alarms leaving the CUA lab into coding CLIs, internal career paths for harness/platform engineering, whether content-exclusion GA is audited against actual CLI working views, whether 2026-10-02 Copilot model deprecations and the GPT-6 Astra picker add are reflected in managed default-model and custom-agent lists before the cutover, whether Agent Merge ships without a path-allowlisted approval gate, whether JetBrains managed sandbox is on for every licensed IDE, whether GHES 3.22 Copilot CLI air-gap preview is the disconnected pin, whether managed permissions cover every agent surface, and whether Linux golden images trust the replacement gh PGP key after 2026-09-05.

Key practices
  • Read CLI customization metrics (skill/MCP/plugin) as adoption signal; do not add plugin totals to skill totals
  • Retarget managed default-model lists before 2026-10-02 and 2026-10-19 Copilot deprecation waves
  • Use agentic autofix to burn Code Quality backlogs in bounded batches (≤25) onto a branch; keep humans on merge; treat AI-credit spend as a review-ops cost, not free cleanup
  • Install gh-stack and the companion agent skill; prompt for one concern per layer; use the revert test
  • Run full agentic review on cloud-agent and bot PRs; keep Copilot approvals off until path allowlists and an executable oracle exist; keep humans on architecture and assumptions
  • Instrument LLM, tool, and sub-agent spans with GenAI semantic conventions; keep dual ledger of spend vs contribution
  • Measure verified tasks, programmatic file read/write coverage, and harness/loop quality, not accepted suggestions or the agent's own completion claim (OverclaimBench: 80.4% misleading rate on incomplete work)
  • Make AGENTS.md, skills, plugins, org custom agents, deny/approval rules, content-exclusion lists, and default-model policy owned artifacts with review
Evidence
  • [vendor] More ways to request and configure Copilot code reviews — 2026-09-23 GA. Dedicated personal review settings across all plans (draft PR and push triggers, Lite/Balanced effort); enterprise-wide default review effort setting.
  • [vendor] Agentic CLI customizations now in the usage metrics API — 2026-09-17. totals_by_skill/custom_agent/mcp/slash_cmd/plugin; distinct-use counts; customer names grouped as other.
  • [vendor] GitHub Copilot weekly releases — September 14 — 2026-09-18. Auto model-selection tiers efficiency/balance/intelligence; VS Code Agents metrics; budget-increase requests.
  • [vendor] Upcoming deprecation of selected GitHub Copilot models in mid-October — 2026-09-18. Deprecate 2026-10-19: Gemini 3.7 Flash, GPT-5.5, GPT-5.4, GPT-5.4 mini, GPT-5 mini, Grok 4.5.
  • [vendor] Copilot code review: An improved review experience — 2026-09-18 GA. Auto-resolve with reasons; overview Open / Resolved / Previously missed.
  • [vendor] Auto-resolution and analysis updates in Copilot code review — 2026-09-11. Lite ensemble +47% high-severity addressed comments, −8% review cost; shell tools for validation.
  • [vendor] Add VS Code Agents to Copilot usage metrics — 2026-09-11 GA. daily_active_vscode_agent_users; separate from editor-window Agent Mode.
  • [vendor] Copilot budget increase requests are generally available — 2026-09-16 GA. Business/Enterprise usage-based billing; not available for managed users.
  • [vendor] Remediate Code Quality findings with agentic autofix — 2026-09-09. Assign up to 25 findings to Copilot; validates then opens a PR. Consumes AI credits; follows existing Code Quality policy.
  • [vendor] Enterprise managed permissions for GitHub Copilot agent operations — 2026-09-09 GA. Deny/ask/allow for shell, file, network; cannot be weakened by user settings or saved approvals.
  • [vendor] Enterprise-managed sandbox in Copilot for JetBrains — 2026-09-08. Managed sandbox public preview; /ide CLI-to-IDE; OTel chat settings GA.
  • [vendor] GitHub Enterprise Server 3.22 generally available — 2026-09-08. Copilot CLI with GHES credentials for air-gapped enterprises (technical preview).
  • [vendor] @github/copilot npm dist-tags — verified 2026-09-22. latest=1.0.87; prerelease=1.0.88-1.
  • [vendor] GitHub CLI Linux package signing key expires September 5 — Key expired 2026-09-05. Replacement key only for subsequent APT/RPM metadata. Fleet-pin for agent CLIs that call gh.
  • [vendor] GPT-6 Astra is generally available in GitHub Copilot — 2026-09-04. Picker availability (gradual) across CLI/app/IDE. Treat as fleet pin / usage-based billing, not a harness-eval reset.
  • [vendor] GitHub Copilot weekly releases — August 31 — 2026-09-04. Content exclusions in app/CLI; Gemini 3.8 Flash and Claude Fable 5.1; JetBrains harness GA; VS Code Agent Merge public preview.
  • [vendor] Upcoming deprecation of selected GitHub Copilot models — 2026-09-03. Deprecate 2026-10-02: Gemini 3.5/3.6 Flash, Kimi K2.7 Code, Claude Opus 4.7. Alts: Gemini 3.8 Flash, Kimi K3, Claude Opus 5.
  • [vendor] Gemini 3.8 Flash is now available in GitHub Copilot
  • [vendor] Copilot code review can now approve pull requests — 2026-09-01 public preview. Approval assessments on every review; actual approvals off by default; enterprise/org/repo + path allowlists; dismissed on new commits.
  • [vendor] Content exclusions generally available in Copilot app and CLI — 2026-09-02 GA. App and CLI honor enterprise/org/repo exclusions.
  • [vendor] Enterprise-managed settings support any default model — 2026-09-02. Default model via managed settings; per-team overrides with overridable model + team-mappings.json. App, CLI, VS Code.
  • [vendor] GitHub Copilot in VS Code, August 2026 releases — 2026-08-31. Agent Host, /btw, prompt timeline, transcript search, /rubber-duck, continue external sessions.
  • [study] AI Coding Agent Effectiveness: Leading Indicators (Span) — 103 teams May–Jul 2026. +1 quality-stewardship → 39% fewer review cycles / 1k merged AI lines; +1 prompt-clarity → 27.2% lower cost/line.
  • [preprint] CURA: Certified Runtime Alarms for Computer-Use Agents (arXiv:2608.27808) — 2026-08-28. 90% of failures claim success; α=0.10 detects 42.3% a median 31 steps early (FAR 0.066).
  • [preprint] Layered Supervision (arXiv:2608.26316) — 2026-08-26. Preventive / executable / human layers; humans shift to architecture.
  • [vendor] Stacked pull requests public preview — 2026-07-30. gh-stack, stack maps, partial merges, agent skill.
  • [vendor] Copilot code review: resolution reasons and expanded capabilities — 2026-08-27. Full agentic review for cloud-agent PRs; size caps dropped.
  • [vendor] Upcoming changes to GitHub Copilot policies and billing — From ≥2026-09-28: Balanced default review; cloud/web/mobile unify, on by default.
  • [preprint] llmmas-otel (arXiv:2608.24271) — 2026-08-25. OTel + fault injection across MAS phases.
  • [vendor] Cloudflare Agents — 2026-08-04 Agents Week. Turn-by-turn replay; span billing from Oct 2026.
  • [preprint] Repair or Resample? (arXiv:2608.25920)
  • [vendor] GitHub Copilot CLI changelog (1.0.81/1.0.82)
  • [preprint] RAMP (arXiv:2608.25241)
  • [preprint] MAGE (arXiv:2608.25174)

Next actions

Concrete moves for the team.

Pulled from the top priorities. Use as a checklist in planning, not as a full program.

  1. Stage rule-based elision before LLM summarization; do not build recoverable-elision machinery the model never uses
  2. Treat CLI context-management tools and session resume as continuity of a process, not as durable, inspectable loop state
  3. Treat an approved sandbox bypass as a logged, time-bounded exception; do not leave managed sandbox disabled for the rest of a session by default
  4. Pin production Copilot CLI to npm latest (1.0.88), not leftover 1.0.87 images or 1.0.89-3 prerelease
  5. Adopt Agent Plugins 1.0 for org skills and MCP that must run in more than one client
  6. Deploy pre-action deterministic verifiers: mandate content-anchored edits (diff/search-replace) and refuse location-anchored line edits that corrupt 99.1% of shifted files

Sources & method

Evidence base.

Daily research rewrites the brief. Engineering SotA is the spine; strategic economics is an enhancement on each priority — not a separate strategy feed. Model ranking lives on its own board.

Selected sources

  • Coding Agents Have Converged (arXiv:2609.17394)
  • An Empirical Study of Harness Design for Coding Agents (arXiv:2609.20804)
  • Harness or Model? Isolating the Harness Effect (arXiv:2609.11987)
  • Beyond Prompts / PRISM (arXiv:2609.05736v2)
  • Co-Evolving Harnesses and Models (arXiv:2609.09134)
  • Harbor Adapters and Harbor-Index (arXiv:2609.04298)
  • What Does Multi-Harness RL Learn? (arXiv:2609.04518)
  • EvoHarnessBench (arXiv:2609.04280)
  • Environment Evolution for Terminal Agents (arXiv:2609.04128)
  • Harness Engineering: Anatomy of Eleven Systems (arXiv:2609.00006)
  • HEART / Tool Primitives (arXiv:2609.01736)
  • HarnessEvolve (arXiv:2609.00829)
  • HarnessDev (arXiv:2609.01437)
  • Same Model, Different Harness (arXiv:2608.26218)
  • Measure Before You Manage (arXiv:2608.31057)
  • EvoUndo (arXiv:2608.28363)

Notes

  • Primary sources drawn from 2026-07-29 through 2026-09-25 engineering posts, vendor security guidance, and arXiv. No model-release hype; Oct 2 and Oct 19 deprecations are cited only as fleet-pin / managed-settings work.
  • 2026-09-25 additions: GitHub Enterprise AI Controls Default Policy (2026-09-24: global default policy for GA features, Copilot Code Review, and MCP servers in Copilot taking effect Oct 22); Copilot CLI prerelease updated to 1.0.89-3 (npm dist-tags latest=1.0.88 verified 2026-09-25); Local sandboxing in GitHub Copilot app (2026-09-23 public preview); STALE-bench (arXiv 2609.25396), Look Before You Leap (arXiv 2609.11957), SWE-Proof (arXiv 2609.21190), and Convergence Audit (arXiv 2609.17394) continue to anchor verification and harness scaffolding as the primary determinants of agentic ROI.
  • 2026-09-24 additions: STALE-bench (arXiv 2609.25396: parallel coding agents suffer 97% semantic coordination failures on 12 Django helpers despite clean git merges; 82% recovered by concurrent change broadcast messages); GitHub Copilot App Local Sandboxing (2026-09-23 public preview: fail-closed isolation across filesystem, network, and git/CLI credentials); Copilot code review expanded settings (2026-09-23 GA: dedicated personal review settings on all plans with draft/push triggers; enterprise-wide default review effort setting); OpenTelemetry in GitHub Copilot app (2026-09-22 GA: enterprise-managed trace export); Copilot CLI prerelease moved to 1.0.89-1 (npm latest 1.0.88 verified 2026-09-24).
  • Copilot CLI production pin is npm dist-tag latest=1.0.88 (verified 2026-09-25); 1.0.89-3 is prerelease only.
  • Merged parents kept strongest headline numbers only (F2PF 28→49; AutoSaddler +9/+9.6/+10; EvoUndo 197/0; SKILLSTATE 16.2×; LoopArena 24.69%; HoH +52.25%; HarnessEvolve +21.6 pp; SARA ASR ≤0.63%; Deadbugz 23 PRs / 3 calls; scaffolding 5–28×; plugins 2,765; SkillZip −38%; ICoA CSR; Span 39% fewer review cycles; CURA 90% false success; HEART 84% vs 22%; HookPry 7/7 / 92.5%; PROCTOR 100% vs 68%; Harbor-Index 28.0%; Multi-Harness RL 4.3× eval harness; co-evolution 4–30 pp regression; PRISM 14.2/14.9/10.1 pp; Scanning the Harness 16.0%; COBRA-Skills 55–58%; Skill Issue +4.9 pp; Harness or Model? native null; OverclaimBench 80.4% misleading / 1.8× defects; Convergence audit 29.8 pp scaffold spread; SWE-Proof 25%–50% counterexamples on test-passing patches; Look Before You Leap 95.8% shell verifier / 99.1% line corruption / 0.01% anchor-and-verify; STALE-bench 97% semantic interference / 82% recovery).
  • Acronyms on first use: F2PF = fail-to-pass fraction; AOT = ahead-of-time harness evolution; JIT = just-in-time harness synthesis; ASR = attack success rate; CSR = covert success rate; OSR = overt success rate; MAS = multi-agent system; CUA = computer-use agent; HoH = Harness-of-Harness; ACP = Agent Client Protocol; CIMD = Client ID Metadata Document; CPE = context privilege escalation; M-CPE = message-role CPE; X-CPE = cross-scope CPE; GRPO = group-relative policy optimization; RAE = Retrieval-Invoked Actual-Use Effect; GHES = GitHub Enterprise Server; RelLift95 = 5th-percentile selected held-out lift under budget; EMU = Enterprise Managed Users.
  • All trends include an economic_lever and ≥1 live URL. Daily runs overwrite the live page; Sunday archives are immutable.

Model picks: Hermes-weighted LLM leaderboard

Weekly immutable snapshots: Archive index · live page overwritten daily at 07:25 Europe/Vienna

Dominic Reichl AI

Independent, practical software experiments.

Labs Models Trends About Dominic Impressum Privacy