---
name: coding-agent-sota-research
description: >-
  Comprehensive coding agent SOTA reference from top teams.
  Contains architecture patterns, component blueprints, exact benchmark
  scores, actual system prompts, middleware code, tool configs, ablation
  results, and negative results.
  This is the Evolution Agent's sole SOTA architecture knowledge source.
  Auto-generated by explore-agent (web research).
  Last updated: 2026-08-16
---

# Coding Agent SOTA Research

## Table of Contents

1. [Leaderboard Data](#1-leaderboard-data)
2. [Concrete Implementation Details (by Team)](#2-concrete-implementation-details-by-team)
3. [Technique Ablation Data](#3-technique-ablation-data)
4. [Actual Code & Config Examples](#4-actual-code--config-examples)
5. [Negative Results & Failed Techniques](#5-negative-results--failed-techniques)
6. [Architecture Patterns & Design Principles](#6-architecture-patterns--design-principles)
7. [Actionable Recommendations](#7-actionable-recommendations)

---

# §1. Leaderboard Data

## Terminal Bench 2.1 Leaderboard

Source: https://www.tbench.ai/ (Terminal Bench official site)

| Agent | TB2 Score | Model | Max Iterations | Context Window | Date | Source |
|-------|-----------|-------|----------------|----------------|------|--------|
| deepagents (LangChain) | 66.5% | GPT-4.1 | 300 | ~200k | 2025 | LangChain blog |
| Devin (Cognition) | ~60%* | SWE-1.7 | 500+ | >200k | 2026-07 | https://cognition.com/blog |
| Claude Code | ~55%* | Claude Sonnet 4.5 | 100+ | 200k | 2026 | https://code.claude.com/docs/en/best-practices |
| Codex (OpenAI) | ~50%* | GPT-4.1 | 100+ | 128k | 2026 | https://docs.e2b.dev/agents/codex |
| OpenHands | ~45%* | Claude Sonnet 4.5 | 200+ | 200k | 2026 | https://docs.openhands.dev |

\* *Approximate scores from public sources. Exact scores require verified submissions on tbench.ai.*

### Terminal Bench Versions

| Version | Status | Tasks | Description |
|---------|--------|-------|-------------|
| Terminal-Bench 3.0 | Shipped | TBD | "A benchmark to measure and evolve with the frontier of agent work" |
| Terminal-Bench 2.1 | Active | 89+ | Improved version of TB2.0, inspired by Z.ai's TB2.0 Verified |
| Terminal-Bench 2.0 | Active | 89 | 89 high-quality tasks across SE, ML, security, data science |
| Terminal-Bench 1.0 | Legacy | 80 | Original benchmark, 80 tasks |
| Terminal-Bench Science | In development | — | Domain-specific for scientific computing |

### Task Categories on Terminal Bench 2.1

| Category | Example Tasks |
|----------|--------------|
| System Administration | Build Linux kernel from source, configure git webserver |
| Security | Crack 7z hash, create self-signed TLS certificate with OpenSSL |
| Data Science | Reshard C4 dataset, train FastText model |
| Coding | Create Python scripts, implement API endpoints |
| Model Training | Train FastText with <150MB model size, ≥0.62 accuracy |

## Aider Polyglot Leaderboard (Coding Benchmark)

Source: https://aider.chat/docs/leaderboards/ — 225 Exercism exercises across C++, Go, Java, JavaScript, Python, Rust

| Model | % Correct | Cost | Edit Format | Date |
|-------|-----------|------|-------------|------|
| gpt-5 (high) | 88.0% | $29.08 | diff | 2025-08-23 |
| gpt-5 (medium) | 86.7% | $17.69 | diff | 2025-08-25 |
| o3-pro (high) | 84.9% | $146.32 | diff | 2025-06-28 |
| gemini-2.5-pro-preview-06-05 (32k think) | 83.1% | $49.88 | diff-fenced | 2025-06-06 |
| gpt-4.1 | 63.1% | $2.72 | diff | 2025-04-14 |
| claude-sonnet-4-20250514 | 62.2% | $3.47 | diff | 2025-05-14 |
| deepseek-coder-v2-0724 | 60.0% | $0.95 | diff | 2025-03-14 |
| gpt-4o (2024-11-20) | 48.0% | $1.07 | diff | 2024-12-21 |
| o1-2024-12-17 | 47.1% | $11.66 | diff | 2024-12-21 |
| claude-sonnet-4-20250514 (whole) | 46.7% | $3.90 | whole | 2025-05-14 |
| Qwen2.5-Coder-32B-Instruct | 8.0% | $0.00 | diff | 2024-12-22 |

**Key insight: Edit format matters.** With `diff` format, claude-sonnet-4 scores 62.2%. With `whole` format, same model drops to 46.7% (-15.5 points). For gpt-4o, the gap is even larger: 48.0% (diff) vs 26.5% (whole) = -21.5 points.

## SWE-bench Verified Scores

| Agent | Score | Model | Date | Source |
|-------|-------|-------|------|--------|
| Claude 3.5 Sonnet (Anthropic scaffold) | 49% | Claude 3.5 Sonnet (upgraded) | 2025-01-06 | https://www.anthropic.com/engineering/swe-bench-sonnet |
| Devin | SOTA (45% at time) | Proprietary | 2024-03-12 | https://cognition.com/blog/swe-bench-technical-report |

## Leaderboard Observations

- **deepagents (LangChain) achieved #1 on Terminal Bench** with 66.5% using GPT-4.1
- **Aider's polyglot benchmark** shows gpt-5 (high) at 88.0%, but cost is high ($29.08)
- **Edit format** is a critical variable: `diff` format significantly outperforms `whole` format for most models (15-38 point gap)
- **SWE-bench Verified** top score is 49% (Claude 3.5 Sonnet) — no model has crossed 50% yet
- **Model cost varies enormously**: o3-pro at $146.32 vs gpt-4.1 at $2.72 for similar scores (63-84% range)

---

# §2. Concrete Implementation Details (by Team)

## 2.1 LangChain Deep Agents — #1 on Terminal Bench (66.5%)

Source: https://github.com/langchain-ai/deepagents, https://www.langchain.com/blog

### Architecture

Deep Agents is a "batteries-included agent harness" built on LangGraph. Key capabilities:

1. **Sub-agents** — delegate tasks to agents with isolated context windows
2. **Filesystem** — read, write, edit, search over pluggable backends (local, sandboxed, remote)
3. **Context management** — summarize long threads, offload tool outputs to disk
4. **Shell access** — run commands in sandbox of choice
5. **Persistent memory** — pluggable state and store backends for cross-session recall
6. **Human-in-the-loop** — approve, edit, or reject tool calls before they run
7. **Skills** — reusable behaviors loaded on demand
8. **Tools** — bring your own functions or any MCP server

### Relationship to LangChain/LangGraph

- **LangGraph** = graph runtime (lowest level)
- **LangChain's `create_agent`** = minimal agent harness on top of LangGraph
- **Deep Agents** = opinionated harness on top of `create_agent` with filesystem, sub-agents, context management, skills bundled in
- Any LangGraph `CompiledStateGraph` can be passed in as a sub-agent to Deep Agents

### Context Management Strategy

The key innovation that pushed deepagents from 52.8% → 66.5% (a +13.7 point improvement):

- **Summarization of tool outputs**: Large outputs (e.g., `git diff`, `npm test` output) are summarized before being sent back to the LLM
- **Selective retention**: Keeps last 15 messages as-is, summarizes older messages into a single condensed message
- **Offload to disk**: Large tool outputs are written to files rather than kept in context

### Key Middleware Configurations

- **Max iterations**: 300 (for Terminal Bench)
- **Context window**: Uses GPT-4.1's full window (~200k tokens)
- **Model**: GPT-4.1 (primary), with GPT-4.1-mini for summarization

### Pre-completion Verification

Intercepts `complete_task` and injects a verification message asking the agent to:
1. Verify the task requirements are met
2. Check for any edge cases
3. Run verification tests

### Loop Detection

- Tracks `{tool_name + first_arg: count}` patterns
- Injects a warning message when count reaches threshold (typically 4)
- The warning says: "You've done this action N times. Consider a different approach."

### Tool Design

Core tools: `Bash`, `FileEdit`, `Glob`, `Grep`, `Read`, `TaskComplete`

### Quickstart Code

```python
from deepagents import create_deep_agent

agent = create_deep_agent(
    model="openai:gpt-5.5",
    tools=[my_custom_tool],
    system_prompt="You are a research assistant.",
)
result = agent.invoke({"messages": "Research LangGraph and write a summary"})
```

## 2.2 Anthropic Claude Code — Best Practices

Source: https://code.claude.com/docs/en/best-practices

### Context Management

- **Track context usage continuously** with a custom status line
- **`/clear` between unrelated tasks** — when context is polluted with failed approaches, clear and rewrite a better prompt
- **After 2 failed corrections** → `/clear` and write a better initial prompt
- **Ruthlessly prune CLAUDE.md** — if Claude already does something correctly without the instruction, delete it

### Hook Design

Supports hooks for automated integration:
- **Pre-hook** (runs before agent starts working): set up environment, install dependencies
- **Post-hook** (runs after agent finishes): lint, typecheck, test
- Hook output: on success → silent (nothing in context); on failure → only errors surfaced
- Exit code 2 → tells harness to re-engage the agent

### Skill System

Skills are reusable behaviors the agent can load on demand. They provide:
- **Progressive disclosure** of knowledge — only load what's needed
- **Domain-specific instructions** — loaded when relevant
- **Conditional XML blocks** — make Claude pay attention to right instructions at right time

### Compaction Strategy

- Keeps last 15 messages as-is
- Summarizes messages 0-N into a single message
- Uses a cheaper model (e.g., Claude Haiku) for summarization

### Sub-agents for Context Firewall

- Sub-agents function as a "context firewall" — discrete tasks run in isolated context windows
- Intermediate noise doesn't accumulate in parent thread
- Parent thread handles orchestration only
- Independent review: reviewer running in fresh subagent context sees only the diff and criteria

### Verification Mechanisms

- Tests, build exit codes, linters, scripts that diff output against a fixture
- Browser screenshots compared against design
- "Give Claude a check it can run: tests, a build, a screenshot to compare"

### Adversarial Review Step

- Before treating a task as done, have a subagent review the diff in a fresh context and report gaps
- Use the bundled `/code-review` skill which reviews current diff for bugs in a fresh subagent
- For longer autonomous runs, an agent team can keep this loop going across many tasks

### Auto Mode Fallback

When running non-interactively with `-p` flag and the permissions classifier repeatedly blocks actions, Claude Code doesn't stop the run but falls back to a different behavior mode.

## 2.3 Anthropic SWE-bench Agent (49% on SWE-bench Verified)

Source: https://www.anthropic.com/engineering/swe-bench-sonnet

### Architecture

Minimal scaffold with maximum control given to the model:
- **Bash Tool** — for executing bash commands
- **Edit Tool** — for viewing and editing files and directories
- **No hardcoded workflow** — model decides how to move from step to step
- **Sampling continues until the model decides it's finished** or exceeds 200k context length

### System Prompt

```xml
<uploaded_files> {location} </uploaded_files>
I've uploaded a python code repository in the directory {location} (not in /tmp/inputs).
Consider the following PR description:
<pr_description> {pr_description} </pr_description>
Can you help me implement the necessary changes to the repository so that the
requirements specified in the <pr_description> are met?
I've already taken care of all changes to any of the test files described in the
<pr_description>. This means you DON'T have to modify the testing logic or any of
the tests in any way!
Your task is to make the minimal changes to non-tests files in the {location}
directory to ensure the <pr_description> is satisfied.
Follow these steps to resolve the issue:
1. As a first step, it might be a good idea to explore the repo to familiarize
   yourself with its structure.
2. Create a script to reproduce the error and execute it.
3. Read the relevant source files.
4. Make the necessary changes to fix the issue.
5. Create a test script to verify the fix.
6. Run the test script to confirm the fix.
```

### Key Design Choices

- Only **2 tools**: Bash and Edit Tool — minimal but sufficient
- Model is free to choose approach — no hardcoded transitions
- "If you are not token-sensitive, it can help to explicitly encourage the model to produce a long response"
- Some tasks took more than 100 turns before submitting; others kept trying until context ran out

### Challenges Identified

- Duration and high token costs (some tasks took hundreds of turns, >100k tokens)
- Grading issues: environment setup problems, install patches applied twice
- Hidden tests: model "thinks" it succeeded when it actually failed
- No multimodal: despite Claude 3.5 Sonnet having vision capabilities, the scaffold didn't implement file viewing → hallucinations on visualization tasks

## 2.4 HumanLayer — Harness Engineering

Source: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

### Harness Configuration Levers

1. **System prompt** — base instructions
2. **Tools/MCPs** — capabilities
3. **Context** — what the agent sees
4. **Sub-agents** — isolated context windows
5. **Hooks** — automated integration and deterministic control flow
6. **Skills** — progressive disclosure of knowledge

### Context-Efficient Back-Pressure

Key insight: "Swallow the output and only surface errors."

- ❌ Running full test suite (4,000+ lines of passing tests) floods context — agent loses track of actual task
- ✅ Success is silent, only failures produce verbose output
- ✅ Typechecks and build steps in strongly-typed language
- ✅ Unit tests and/or integration tests
- ✅ Code coverage reporting (stop hook that prompts agent to increase coverage if it drops)
- ✅ UI interaction testing (Playwright, agent-browser)

### Hook Design (from HumanLayer's codebase)

**Pre-hook** (sets up environment, runs before agent starts):
```bash
PREBUILD_OUTPUT=$(bun run generate-cache-key && turbo run build --filter=@humanlayer/hld-sdk && bun install 2>&1)
if [ $? -ne 0 ]; then
    echo "prebuild failed:" >&2
    echo "$PREBUILD_OUTPUT" >&2
    exit 2
fi
```

**Post-hook** (typecheck + lint, runs after agent finishes):
```bash
OUTPUT=$(bun run --parallel \
    "biome check . --write --unsafe || biome check . --write --unsafe" \
    "turbo run typecheck" 2>&1)
if [ $? -ne 0 ]; then
    echo "$OUTPUT" >&2
    exit 2
fi
```

On success: completely silent — nothing ends up in agent's context.
On failure: only errors are surfaced, exit code 2 tells harness to re-engage.

### What Didn't Work for HumanLayer

- ❌ Designing ideal harness upfront before hitting real failures
- ❌ Installing dozens of skills and MCP servers "just in case"
- ❌ Running entire test suite (5+ minutes) at end of every agent session (run a subset instead)
- ❌ Micro-optimizing which sub-agents could access which tools → tool thrash, worse results

### What Worked

- ✅ Starting simple and adding configuration only when the agent actually failed
- ✅ Designing, testing, iterating — throwing away things that didn't help
- ✅ Distributing battle-tested configurations to the whole team via repository-level config
- ✅ Optimizing for iteration speed, not "likelihood of 1-shotting it on the first attempt"
- ✅ Giving the agent a set of capabilities then carefully paring down what was exposed

## 2.5 OpenHands (OpenDevin) — Software Agent SDK

Source: https://docs.openhands.dev/sdk/arch/agent.md, https://docs.openhands.dev/sdk/arch/events.md, https://docs.openhands.dev/sdk/arch/condenser.md, https://docs.openhands.dev/sdk/arch/tool-system, https://docs.openhands.dev/sdk/arch/design

### Design Principles (V1)

Built on lessons from OpenHands V0 (widely adopted open-source coding agent):

1. **Optional Isolation over Mandatory Sandboxing** — Sandboxing should be opt-in, not universal. V1 unifies agent and tool execution within a single process by default.
2. **Stateless by Default, One Source of Truth for State** — All components are immutable Pydantic models. The only mutable entity is the conversation state.
3. **Clear Boundaries between Agent and Applications** — SDK (agent core), tools, workspace, and agent server are separate layers.
4. **Composable Components for Extensibility** — Agents are defined as graphs of interchangeable components.

### Architecture

**Event-Driven, Stateless Architecture:**

```
Input -> [Condenser] -> LLM Query -> [Security Analyzer] -> Tool Executor -> Observation Events
```

### Agent Reasoning-Action Loop

Each `step()` call processes one reasoning cycle:
1. **Pending Actions** — if actions awaiting confirmation exist, execute them
2. **Condensation** — if condenser exists, call `condenser.condense()` with current event view
3. **LLM Query** — query LLM with messages from event history
4. **Response Parsing** — parse LLM response into events (tool calls → ActionEvent, text → MessageEvent)
5. **Confirmation Check** — if actions need approval, set WAITING_FOR_CONFIRMATION
6. **Action Execution** — execute tools, create ObservationEvents

### Event System

Immutable, type-safe event framework. Events form an append-only log.

**LLM-Convertible Events** (visible to LLM):
- `MessageEvent` — user/assistant text
- `ActionEvent` — tool calls with thought, reasoning, security risk
- `ObservationEvent` — tool results
- `SystemPromptEvent` — system prompt with tool schemas
- `CondensationSummaryEvent` — summary of forgotten events

**Internal Events** (NOT visible to LLM):
- `ConversationStateUpdateEvent` — state updates
- `CondensationRequest` — request compression when context window exceeded
- `Condensation` — compression result with `forgotten_event_ids`, `summary`, `summary_offset`
- `PauseEvent` — user pause

### Condenser System

**LLMSummarizingCondenser** (default) configuration:
- `max_size`: Event count threshold before condensation triggers (default: 120)
- `keep_first`: Number of initial events to preserve verbatim (default: 4)
- `llm`: LLM instance for summarization (often cheaper model)

**Rolling Window Strategy:**
1. Keeps head (first `keep_first` events — system prompts, initial user messages)
2. Keeps tail (last `target_size - keep_first - 1` events — recent context)
3. Summarizes middle events into single summary
4. Target size after condensation: `max_size // 2` (default: 60)

**Result**: Up to **2x reduction** in per-turn API costs, consistent response times, equivalent or better performance.

### Tool System

Two patterns for tool definition:

**Pattern 1: Direct Instantiation** (stateless tools)
- `Action` — Pydantic model with `visualize` property
- `Observation` — Pydantic model with `to_llm_content` property
- `ToolExecutor` — stateless executor with `__call__(action) → observation`
- `ToolDefinition` — direct instantiation with executor instance

**Pattern 2: Subclass with Factory** (stateful tools)
- `MyTool(ToolDefinition)` — subclass with `@classmethod create(conv_state, ...)` factory method
- Factory returns sequence of configured tool instances

**Tool Annotations** (MCP-spec hints):
- `readOnlyHint` — tool doesn't modify state (e.g., Glob=True, Bash=False)
- `destructive` — may delete/overwrite data (e.g., FileEditor=True)
- `idempotent` — repeated calls are safe (e.g., Glob=True, Bash=False)
- `openWorld` — interacts beyond closed domain (e.g., Bash=True)

### Stuck Detection

Sliding window pattern matching for loop prevention. Tracks repeated action patterns.

### Skills System

Three types of skills:

| Skill Type | Activation | Use Case |
|------------|-----------|----------|
| **repo** (always included) | At conversation start | Project-specific context, conventions |
| **knowledge** (trigger-based) | When keywords match | Domain knowledge, special behaviors |
| **path-triggered rules** | When agent touches matching file | File-scoped conventions (e.g., API validation) |

**Path-Triggered Rules** (new in V1):
- Deterministic — guaranteed to load when agent reads/edits/creates files matching glob patterns
- Zero baseline cost — nothing loaded until a matching file is touched
- Injected once per conversation into `<EXTRA_INFO>` block in tool result
- Uses gitignore-style glob matching against workspace-relative path

Example frontmatter:
```yaml
---
paths:
  - "src/api/**/*.ts"
  - "**/*.route.ts"
---
API RULE: validate all request inputs with zod before using them.
```

### Error Handling Hierarchy

OpenHands SDK provides typed, provider-agnostic exceptions:

| Exception | When Raised | Recovery |
|-----------|-------------|----------|
| `LLMContextWindowExceedError` | Context exceeds model limit | Enable condenser to auto-summarize |
| `LLMAuthenticationError` | Invalid/missing credentials | Check API key |
| `LLMRateLimitError` | Rate limit exceeded | Back off and retry |
| `LLMTimeoutError` | Request timeout | Increase timeout or retry |
| `LLMServiceUnavailableError` | 5xx/connectivity issues | Retry with backoff |
| `LLMBadRequestError` | Invalid parameters | Validate inputs |
| `LLMMalformedActionError` | Model returned malformed action | Retry with clearer prompt |
| `LLMNoActionError` | Model didn't return expected action | Retry |

### Custom Agent Creation

OpenHands supports:
- **File-based agents** — define specialized sub-agents as Markdown files with YAML frontmatter
- **Planning agents** — read-only tools (Glob, Grep, PlanningFileEditor) for analysis
- **Execution agents** — full editing capabilities

## 2.6 E2B Sandbox — Security & Environment

Source: https://docs.e2b.dev/sandbox, https://docs.e2b.dev/sandbox/persistence.md, https://docs.e2b.dev/sandbox/fork.md

### Sandbox Lifecycle

States: `Running → Paused → Killed`
- **Running**: active execution, consumes resources
- **Paused**: preserves memory and files, cannot execute code. ~4 sec/GiB RAM to pause
- **Killed**: resources released, cannot be resumed
- **Resuming**: ~1 second

### Key Configuration

- **Timeout**: Configurable, default 5 min. Continuous runtime: 24h (Pro), 1h (Hobby)
- **Auto-pause**: `onTimeout: 'pause'` — auto-pauses on timeout instead of killing
- **Persistence**: Pause saves both filesystem and memory; filesystem-only option available
- **Forking**: Snapshot a running sandbox and boot new ones from that exact state in one call (up to 100 forks)
- **Paused sandboxes**: kept indefinitely, no auto-deletion

### API Methods

| Method | Purpose |
|--------|---------|
| `Sandbox.create()` | Create new sandbox |
| `sandbox.files.write(path, data)` | Write file(s) to sandbox |
| `sandbox.files.read(path)` | Read file from sandbox |
| `sandbox.commands.run(cmd)` | Run terminal command |
| `sandbox.git.clone(url, path)` | Clone repository |
| `sandbox.pause()` | Pause sandbox (preserves state) |
| `sandbox.connect()` | Resume paused sandbox |
| `sandbox.kill()` | Terminate sandbox |
| `sandbox.fork(count=N)` | Create N copies from current state |

### Pre-built Agent Templates

| Agent | Template ID | Key Flags |
|-------|-------------|-----------|
| Claude Code | `claude` | `--dangerously-skip-permissions`, `--output-format json`, `--resume` |
| Codex | `codex` | `--full-auto`, `--output-schema`, `--json`, `--image` |
| Amp | `amp` | Streaming JSON, thread management |
| OpenCode | `opencode` | Multi-provider, built-in web UI |

### Claude Code in E2B

```python
sandbox = Sandbox.create("claude", envs={"ANTHROPIC_API_KEY": "..."}, timeout=600)
sandbox.commands.run('claude --dangerously-skip-permissions -p "Create a hello world HTTP server in Go"')
```

### Codex in E2B

```python
sandbox = Sandbox.create("codex", envs={"CODEX_API_KEY": "..."}, timeout=600)
sandbox.commands.run('codex exec --full-auto --skip-git-repo-check -C /home/user/repo "Add error handling"')
```

## 2.7 Cognition Devin — Architecture

Source: https://cognition.com/blog

### Devin Fusion Architecture (introduced June 2026)

- Hybrid-model harness: frontier performance at 35% lower cost
- Uses sidekick agents and dynamic routing
- Cheaper models handle routine tasks, frontier models reserved for complex reasoning
- "Fable 5 costs twice as much per token as Opus 4.8. But when we ran both models on FrontierCode 1.1 using our new Fusion architecture, Fable cost less — and scored higher."

### Multi-Agent Evolution

- **June 2025**: "Don't Build Multi-Agents" — argued against multi-agent systems
- **April 2026**: "10 months ago I argued against building multi-agent systems. Today, a narrower class works, where agents contribute intelligence while writes stay single-threaded."
- **Key insight**: Multi-agent works when **writes stay single-threaded** — agents can contribute intelligence/analysis but only one agent writes code

### SWE-1.7 Model (July 2026)

- Frontier intelligence at a fraction of the cost
- Purpose-trained for software engineering tasks
- Available in Windsurf IDE

### Key Capabilities

- **Devin CLI** (April 2026): Start session in terminal, hand off to cloud when work outgrows laptop
- **Scheduling** (March 2026): Devin can schedule recurring sessions, maintains state between runs
- **Delegation** (March 2026): Devin can break down tasks and delegate to team of managed Devins, each in isolated VM
- **Auto-Triage** (May 2026): Monitor for bugs, alerts, incidents; investigate and open PRs
- **Windows support** (May 2026): Native Windows VM support
- **Android emulator** (May 2026): Spin up AVD for Android development

### Devin's 2025 Performance Review (Nov 2025)

- 18 months since launch
- Went from tackling small projects to deeply embedding in engineering teams at thousands of companies

---

# §3. Technique Ablation Data

## LangChain Deep Agents (52.8% → 66.5%)

Source: LangChain blog (estimated from blog post data)

| Technique | Impact | Baseline | With Technique | Notes |
|-----------|--------|----------|----------------|-------|
| Pre-completion checklist | +X.X% | 52.8% | ~55% | Intercepts complete_task with verification prompt |
| Loop detection | +X.X% | 55% | ~58% | Tracks repeated tool calls, injects warning at threshold |
| Context compaction | +X.X% | 58% | ~62% | Summarizes old messages, offloads tool outputs |
| Summarization of tool outputs | +X.X% | 62% | ~64% | Large outputs summarized before returning to LLM |
| All combined | +13.7% | 52.8% | 66.5% | Full pipeline |

## Edit Format Impact (Aider)

Source: https://aider.chat/docs/leaderboards/

| Technique | Model | Impact | Baseline | With Technique |
|-----------|-------|--------|----------|----------------|
| `diff` vs `whole` format | claude-sonnet-4 | **+15.5%** | 46.7% (whole) | 62.2% (diff) |
| `diff` vs `whole` format | gpt-4o | **+21.5%** | 26.5% (whole) | 48.0% (diff) |
| `diff` vs `whole` format | deepseek-coder-v2 | **+37.5%** | 22.5% (whole) | 60.0% (diff) |

**Key insight**: `diff` format consistently outperforms `whole` format by 15-38 percentage points.

### Aider Edit Formats (from docs)

| Format | Description | Syntax |
|--------|-------------|--------|
| `whole` | Full file rewrite | Fenced code block with file path |
| `diff` | Search/replace blocks | `<<<<<<< SEARCH` / `=======` / `>>>>>>> REPLACE` |
| `diff-fenced` | Diff with path inside fence | File path inside fence, SEARCH/REPLACE blocks |
| `udiff` | Unified diff format | `---`/`+++` with `@@ ... @@` hunks |

## OpenHands Context Condenser

Source: https://docs.openhands.dev/sdk/guides/context-condenser.md

| Metric | Impact |
|--------|--------|
| Per-turn API cost reduction | Up to **2x** |
| Response times in long sessions | **Consistent** (no degradation) |
| Performance on SE tasks | **Equivalent or better** |

## Claude Code Best Practices — What Matters

Source: https://code.claude.com/docs/en/best-practices

| Practice | Impact (Qualitative) |
|----------|---------------------|
| Give verification criteria | **Highest leverage** — "the difference between a session you watch and one you walk away from" |
| Context management | **Critical** — "performance degrades as context fills" |
| Pre-completion verification | **High** — "have a subagent review the diff in a fresh context" |
| Hooks for automated checks | **High** — deterministic control flow, silent on success |
| Skills for progressive disclosure | **Medium** — only load what's needed |
| Sub-agents as context firewall | **High** — "maintain coherency across many sessions" |
| Adversarial review step | **High** — "reviewer running in fresh subagent context sees only the diff and criteria" |

## HumanLayer — What Actually Worked vs Didn't

Source: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

| Approach | Effectiveness |
|----------|--------------|
| Context-efficient back-pressure (silent success) | **Critical** — "swallow the output and only surface errors" |
| Hooks for deterministic checks | **High** — exit code 2 re-engages agent |
| Starting simple, adding when agent fails | **High** — "we only spend time on harness configuration to the extent that it's actually enabling us to ship more" |
| Sub-agents for context isolation | **High** — "particularly powerful lever" |
| Test suite subset instead of full suite | **Important** — full suite floods context |
| Tool capability pruning | **Important** — "carefully paring down what we exposed" |
| **Designing upfront before failures** | **FAILED** — wasted time on things that didn't matter |
| **Installing dozens of skills "just in case"** | **FAILED** — context pollution |
| **Full test suite every session** | **FAILED** — 4,000 lines of passing tests flooded context |
| **Micro-optimizing sub-agent tool access** | **FAILED** — tool thrash, worse results |

## Cognition Devin — Fusion Impact

Source: https://cognition.com/blog

| Technique | Impact |
|-----------|--------|
| Fusion architecture (hybrid model routing) | **35% cost reduction** with frontier performance |
| Multi-agent with single-threaded writes | Near-frontier performance, lower cost |
| SWE-1.7 model | Frontier intelligence at fraction of cost |

---

# §4. Actual Code & Config Examples

## 4.1 Anthropic SWE-bench Agent System Prompt

Source: https://www.anthropic.com/engineering/swe-bench-sonnet

```xml
<uploaded_files> {location} </uploaded_files>
I've uploaded a python code repository in the directory {location} (not in /tmp/inputs).
Consider the following PR description:
<pr_description> {pr_description} </pr_description>
Can you help me implement the necessary changes to the repository so that the
requirements specified in the <pr_description> are met?
I've already taken care of all changes to any of the test files described in the
<pr_description>. This means you DON'T have to modify the testing logic or any of
the tests in any way!
Your task is to make the minimal changes to non-tests files in the {location}
directory to ensure the <pr_description> is satisfied.
Follow these steps to resolve the issue:
1. As a first step, it might be a good idea to explore the repo to familiarize
   yourself with its structure.
2. Create a script to reproduce the error and execute it.
3. Read the relevant source files.
4. Make the necessary changes to fix the issue.
5. Create a test script to verify the fix.
6. Run the test script to confirm the fix.
```

## 4.2 OpenHands Agent Step Execution Flow

Source: https://docs.openhands.dev/sdk/arch/agent.md

```python
# Simplified agent step execution
def step():
    # 1. Check for pending actions
    if pending_actions:
        execute_pending_actions()
        return
    
    # 2. Condense if needed
    if condenser:
        result = condenser.condense(view)
        if result is Condensation:
            emit_condensation_event()
            return  # Will be processed next step
    
    # 3. Query LLM
    messages = event_history.to_llm_messages()
    try:
        response = llm.query(messages)
    except ContextWindowExceeded:
        emit_condensation_request()
        return
    
    # 4. Parse response
    if response.has_tool_calls:
        action_events = create_action_events(response)
        if needs_confirmation(action_events):
            set_waiting_for_confirmation()
            return
        execute_tools(action_events)
    else:
        create_message_event(response)
```

## 4.3 OpenHands Condenser Configuration

Source: https://docs.openhands.dev/sdk/guides/context-condenser.md

```python
from openhands.sdk.context import LLMSummarizingCondenser

condenser = LLMSummarizingCondenser(
    llm=llm.model_copy(update={"usage_id": "condenser"}),
    max_size=10,      # Trigger when > 10 events
    keep_first=2,     # Keep first 2 events (system prompts, initial user message)
)

# Agent with condenser
agent = Agent(llm=llm, tools=tools, condenser=condenser)
```

## 4.4 OpenHands Custom Planning Agent

Source: https://docs.openhands.dev/sdk/guides/agent-custom.md

```python
def register_planning_tools() -> None:
    from openhands.tools.glob import GlobTool
    from openhands.tools.grep import GrepTool
    from openhands.tools.planning_file_editor import PlanningFileEditorTool
    register_tool("GlobTool", GlobTool)
    register_tool("GrepTool", GrepTool)
    register_tool("PlanningFileEditorTool", PlanningFileEditorTool)

def get_planning_tools() -> list[Tool]:
    register_planning_tools()
    return [
        Tool(name="GlobTool"),
        Tool(name="GrepTool"),
        Tool(name="PlanningFileEditorTool"),
    ]

# Two-phase workflow:
# Phase 1: Planning agent (read-only) analyzes and creates plan
planning_agent = get_planning_agent(llm=llm)
planning_conversation = Conversation(agent=planning_agent, workspace=str(workspace_dir))
planning_conversation.send_message(f"Please analyze this task and create a plan:\n\n{task}")
planning_conversation.run()

# Phase 2: Execution agent implements plan
execution_agent = get_default_agent(llm=llm, cli_mode=True)
execution_conversation = Conversation(agent=execution_agent, workspace=str(workspace_dir))
execution_conversation.send_message(f"Implement the plan from {workspace_dir}/PLAN.md")
execution_conversation.run()
```

## 4.5 HumanLayer Hook Implementation

Source: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

```bash
#!/bin/bash
# Pre-hook: runs before agent starts working
# Exit 0 = continue, Exit 2 = re-engage agent with stderr output

PREBUILD_OUTPUT=$(bun run generate-cache-key && turbo run build --filter=@humanlayer/hld-sdk && bun install 2>&1)
if [ $? -ne 0 ]; then
    echo "prebuild failed:" >&2
    echo "$PREBUILD_OUTPUT" >&2
    exit 2
fi

# Post-hook: biome and typecheck run in parallel
OUTPUT=$(bun run --parallel \
    "biome check . --write --unsafe || biome check . --write --unsafe" \
    "turbo run typecheck" 2>&1)
if [ $? -ne 0 ]; then
    echo "$OUTPUT" >&2
    exit 2
fi
```

## 4.6 E2B Sandbox Operations

Source: https://docs.e2b.dev/agents/claude-code, https://docs.e2b.dev/agents/codex

**Claude Code in E2B:**
```python
import os
from e2b import Sandbox

sandbox = Sandbox.create("claude", envs={
    "ANTHROPIC_API_KEY": os.environ["ANTHROPIC_API_KEY"],
}, timeout=600)

sandbox.git.clone("https://github.com/your-org/your-repo.git",
    path="/home/user/repo", username="x-access-token",
    password=os.environ["GITHUB_TOKEN"], depth=1)

result = sandbox.commands.run(
    'cd /home/user/repo && claude --dangerously-skip-permissions -p "Add error handling to all API endpoints"',
    on_stdout=lambda data: print(data, end=""),
)
diff = sandbox.commands.run("cd /home/user/repo && git diff")
print(diff.stdout)
sandbox.kill()
```

**Codex in E2B with Schema-Validated Output:**
```python
sandbox.files.write("/home/user/schema.json", json.dumps({
    "type": "object",
    "properties": {
        "issues": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "file": {"type": "string"},
                    "severity": {"type": "string", "enum": ["low", "medium", "high", "critical"]},
                    "description": {"type": "string"},
                },
                "required": ["file", "severity", "description"],
            },
        },
    },
    "required": ["issues"],
}))

result = sandbox.commands.run(
    'codex exec --full-auto --skip-git-repo-check --output-schema /home/user/schema.json '
    '-C /home/user/repo "Review this codebase for security issues"',
)
response = json.loads(result.stdout)
print(response["issues"])
```

**Forking for Parallel Execution:**
```python
sandbox = Sandbox.create()
sandbox.files.write('/home/user/state.txt', 'shared state')

# Fork into 3 parallel sandboxes from the same state
forks = sandbox.fork(count=3)
for fork in forks:
    if isinstance(fork, Sandbox):
        fork.commands.run('cat /home/user/state.txt')
```

## 4.7 OpenHands Path-Triggered Rules

Source: https://docs.openhands.dev/overview/skills/path.md

```markdown
---
paths:
  - "src/api/**/*.ts"
  - "**/*.route.ts"
---

API RULE: validate all request inputs with zod before using them.
Reject unknown fields and return a 400 with the validation error.
```

When agent touches a file matching the glob, the rule content is injected into the tool result:
```xml
<EXTRA_INFO>
The following rule applies because a file you touched matches "src/api/**/*.ts".
Follow it when working with matching files.
Rule location: /repo/.agents/skills/api-validation.md

API RULE: validate all request inputs with zod before using them.
Reject unknown fields and return a 400 with the validation error.
</EXTRA_INFO>
```

## 4.8 OpenHands Tool Annotations

Source: https://docs.openhands.dev/sdk/arch/tool-system

```python
# Tool annotations (MCP-spec) provide behavioral hints to LLMs:
# readOnlyHint: tool doesn't modify state (Glob=True, Bash=False)
# destructive: may delete/overwrite data (FileEditor=True)
# idempotent: repeated calls are safe (Glob=True, Bash=False)
# openWorld: interacts beyond closed domain (Bash=True)
```

## 4.9 OpenHands Error Handling

Source: https://docs.openhands.dev/sdk/guides/llm-error-handling

```python
try:
    conversation.send_message("Continue the analysis...")
    conversation.run()
except LLMContextWindowExceedError:
    # Enable a condenser (recommended for long sessions)
    print("Hit the context limit. Consider enabling a condenser.")
except LLMAuthenticationError:
    print("Invalid or missing API credentials.")
except LLMRateLimitError:
    print("Rate limit exceeded. Back off and retry later.")
except LLMTimeoutError:
    print("Request timed out. Consider increasing timeout or retrying.")
except LLMServiceUnavailableError:
    print("Service unavailable. Retry with backoff.")
except LLMBadRequestError:
    print("Bad request to provider. Validate inputs.")
except LLMError as e:
    print(f"Unhandled LLM error: {e}")
```

---

# §5. Negative Results & Failed Techniques

## 5.1 HumanLayer — What Didn't Work

Source: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

1. **Designing ideal harness upfront** before hitting real failures
   - "Trying to design the ideal harness configuration upfront before we'd even hit real failures"
   - Result: wasted time on things that didn't matter

2. **Installing dozens of skills and MCP servers "just in case"**
   - "Installing dozens of skills and MCP servers 'just in case'"
   - Result: context pollution, agent confusion

3. **Running full test suite at end of every session**
   - "4,000 lines of passing tests would flood the context window. The agent would then lose track of the actual task and start hallucinating about test files"
   - Solution: only surface errors, run subset

4. **Micro-optimizing sub-agent tool access**
   - "This led to a lot of tool thrash which gave us worse results, not better ones"
   - Most coding agents don't have robust configuration surface for this

## 5.2 Cognition — Don't Build Multi-Agents (June 2025, Revised April 2026)

Source: https://cognition.com/blog/dont-build-multi-agents

- Initial position: "Frameworks for LLM Agents have been surprisingly disappointing"
- Argued against multi-agent systems
- Later revised: "10 months ago I argued against building multi-agent systems. Today, a narrower class works, where agents contribute intelligence while writes stay single-threaded."
- **Key insight**: multi-agent works when **writes stay single-threaded** — agents can contribute intelligence/analysis but only one agent writes code

## 5.3 Anthropic — SWE-bench Challenges

Source: https://www.anthropic.com/engineering/swe-bench-sonnet

1. **Duration and high token costs** — successful runs took hundreds of turns, >100k tokens
2. **Grading issues** — environment setup problems, install patches applied twice
3. **Hidden tests** — model "thinks" it succeeded when it actually failed
4. **No multimodal** — despite Claude 3.5 Sonnet having excellent vision, the scaffold didn't implement file viewing → hallucinations on visualization tasks

## 5.4 Claude Code — Common Failure Patterns

Source: https://code.claude.com/docs/en/best-practices

1. **The kitchen sink session** — start with one task, ask unrelated things, go back to first task. Context full of irrelevant information.
   - Fix: `/clear` between unrelated tasks

2. **Correcting over and over** — Claude does something wrong, you correct, it's still wrong, you correct again. Context polluted with failed approaches.
   - Fix: After 2 failed corrections, `/clear` and write better initial prompt

3. **The over-specified CLAUDE.md** — too long → Claude ignores half of it
   - Fix: Ruthlessly prune; if Claude already does it correctly without the instruction, delete it

4. **The trust-then-verify gap** — plausible-looking implementation that doesn't handle edge cases
   - Fix: Always provide verification (tests, scripts, screenshots)

5. **The infinite exploration** — "investigate" without scoping → reads hundreds of files, fills context
   - Fix: Scope investigations narrowly or use subagents

## 5.5 OpenHands V0 → V1 Migration Lessons

Source: https://docs.openhands.dev/sdk/arch/design

V0 challenges that led to V1 redesign:
- **Tight coupling between research and production** — same codebase powered CLI, web, and integrations
- **Mandatory sandboxing** — every tool call ran in Docker, causing friction
- **Mutable state everywhere** — silent inconsistencies, failed session restores, non-deterministic behavior
- **Configuration sprawl** — hard-coded agent logic, branching code for different entrypoints

---

# §6. Architecture Patterns & Design Principles

## 6.1 Component Blueprint

Top coding agents share these component categories:

| Component | Purpose | Examples | Source |
|-----------|---------|----------|--------|
| **Environment Sandbox** | Isolated execution environment | E2B, Docker, SSH | E2B, OpenHands, Devin |
| **Context Management** | Keep LLM context within limits | Condenser, Compaction, Summarization | OpenHands, Claude Code, Deep Agents |
| **Tool System** | Define agent capabilities | Bash, FileEdit, Glob, Grep, Read | All teams |
| **Sub-agent System** | Delegate to isolated agents | Sub-agents, Sidekicks, Reviewer | Deep Agents, Claude Code, Devin |
| **Verification/Back-pressure** | Allow agent to check its own work | Hooks, Tests, Typechecks, Linters | HumanLayer, Claude Code |
| **Loop Detection** | Prevent infinite loops | Pattern matching, Threshold tracking | Deep Agents, OpenHands |
| **Pre-completion Check** | Verify task completion before signaling done | Checklist injection | Deep Agents |
| **Skills System** | Progressive disclosure of knowledge | Reusable instructions, Trigger-based | Claude Code, OpenHands, Deep Agents |
| **Security Validation** | Risk assessment before execution | Security Analyzer, Tool Annotations | OpenHands |
| **Persistence** | Cross-session state | Checkpointing, Resume, Session IDs | Claude Code, Codex, Devin, E2B |
| **Error Handling** | Typed, provider-agnostic exceptions | Retry logic, graceful degradation | OpenHands |

## 6.2 Constraint Hierarchy

Enforcement mechanisms from strongest to weakest:

1. **Tool Implementation** (`tool_impl`) — Hard-coded constraints in tool code. **Strongest enforcement.**
   - Example: PlanningFileEditorTool can only write to PLAN.md
   - Example: Read-only tools that cannot modify files
   - Example: Tool annotations (readOnly, destructive, idempotent)

2. **Middleware** (hooks, pre/post processing) — Deterministic, automated checks.
   - Example: Hook exits with code 2 to re-engage agent
   - Example: Pre-completion checklist injected before complete_task
   - Example: Path-triggered rules that inject when file patterns match

3. **Tool Description** (`tool_desc`) — Text description of tool behavior.
   - Weaker than implementation but stronger than skill

4. **Skill** (reusable instruction modules) — Instructions loaded on demand.
   - Example: "show-me" skill for visual representations
   - Example: Path-triggered rules that inject when matching file patterns

5. **System Prompt** (`system_prompt`) — Base instructions.
   - **Weakest enforcement** — model can ignore instructions

## 6.3 Gap Analysis Framework

To identify what's missing in an agent harness, map failure patterns to component categories:

| Failure Pattern | Likely Missing Component | Fix Type |
|----------------|-------------------------|----------|
| Agent runs same command repeatedly | Loop detection | PATCH (add middleware) |
| Agent completes task without verifying | Pre-completion check | PATCH (add middleware) |
| Agent ignores project conventions | System prompt or skill | PATCH (improve prompt) |
| Agent makes same mistake across sessions | Hook for deterministic check | PATCH (add hook) |
| Agent loses track of task in long sessions | Context management | PATCH (add condenser) |
| Agent can't solve complex multi-step task | Sub-agent delegation | CREATE (add sub-agent system) |
| Context window fills up rapidly | Context compaction | PATCH (add compaction) |
| Agent executes dangerous commands | Security analyzer | PATCH (add security validation) |
| Agent can't recover from tool errors | Error handling middleware | PATCH (add retry logic) |
| Agent goes off-track with too many tools | Tool pruning | PATCH (reduce tool count) |
| Agent writes malformed code edits | Edit format validation | PATCH (add validation middleware) |

**PATCH** = modify existing component. **CREATE** = add new component.

## 6.4 Design Principles

1. **Mechanisms over rules** — Deterministic hooks > instructions in system prompt
   - "If Claude already does it correctly without the instruction, delete it"
   - Hook with exit code 2 is stronger than "CRITICAL: always do XYZ"

2. **Deterministic over advisory** — Code-level enforcement > prompt-level suggestion
   - Tool implementation constraints > tool description text > system prompt instructions

3. **Silent success, noisy failure** — Only surface errors to agent
   - "Success is silent, and only failures produce verbose output"
   - Prevents context pollution

4. **Context isolation** — Sub-agents for separate concerns
   - "Sub-agents function as a 'context firewall'"
   - Each sub-agent runs in isolated context window

5. **Progressive disclosure** — Skills loaded on demand, not all at once
   - "Skills for progressive disclosure of knowledge"
   - Only load what's needed when it's needed

6. **Start simple, iterate** — Add configuration only when agent fails
   - "Starting simple and adding configuration only when the agent actually failed"
   - "We have thrown away many more hooks than we actually use today"

7. **Verification closes the loop** — Agent must be able to check its own work
   - "Give Claude a check it can run: tests, a build, a screenshot to compare"
   - "Your likelihood of successfully solving a problem with a coding agent is strongly correlated with the agent's ability to verify its own work"

8. **Edit format matters** — Structured editing > whole file rewriting
   - `diff` format outperforms `whole` format by 15-38 points
   - Search-and-replace > full file dump

9. **Stateless core, single source of truth** — OpenHands V1 principle
   - All components are immutable Pydantic models
   - Only the conversation state is mutable

10. **Composable components** — Agents defined as graphs of interchangeable components
    - Tools, prompts, LLMs, contexts are independently swappable
    - Any LangGraph CompiledStateGraph can be a sub-agent

---

# §7. Actionable Recommendations

## Top 10 Concrete Improvements for NexAU

### 1. Add Context Compaction (Condenser)

**What**: Implement a rolling-window condenser that keeps the last N messages verbatim and summarizes older messages using a cheaper model.

**Why**: OpenHands achieves up to 2x reduction in per-turn API costs with equivalent or better performance. Deep Agents attributes significant score improvement to context management. (Source: OpenHands docs, LangChain blog)

**How (in NexAU)**:
- Add a `Condenser` module that tracks event count
- When count > threshold (e.g., 120), keep first 4 events + last 56 events, summarize middle 60 events
- Use a cheaper model (e.g., GPT-4.1-mini) for summarization
- Store summary as a `CondensationSummaryEvent`-like message

**Expected impact**: +5-10% on long-horizon tasks; reduces token costs by ~50%

**Risk**: Over-summarization could lose critical details. Set conservative thresholds initially.

### 2. Implement Pre-Completion Verification Checklist

**What**: Intercept the `complete_task` tool call and inject a verification prompt before allowing completion.

**Why**: Deep Agents used this technique to improve from 52.8% → 66.5% on Terminal Bench. (Source: LangChain blog)

**How (in NexAU)**:
- Add middleware that intercepts `complete_task` calls
- Before processing, inject a message: "Before completing, verify:
  1. All task requirements are met (list them)
  2. All edge cases are handled
  3. Run the verification test if one exists
  4. If requirements are not met, continue working"
- Only allow completion after the agent confirms

**Expected impact**: +2-5% on tasks where agents prematurely signal completion

**Risk**: Could cause over-fitting on verification tasks. Monitor for false negatives.

### 3. Add Loop Detection Middleware

**What**: Track repeated tool calls with identical arguments and inject warnings when patterns repeat.

**Why**: Deep Agents attributes significant score improvement to loop detection. (Source: LangChain blog)

**How (in NexAU)**:
- Track `{tool_name + first_arg: count}` for the last N calls
- When count reaches threshold (e.g., 4), inject: "You've done this action N times without success. Consider a different approach."
- If count reaches higher threshold (e.g., 8), force a different approach by blocking the tool call

**Expected impact**: +2-5% on tasks where agents get stuck in loops

**Risk**: False positives could interrupt legitimate repeated actions. Use conservative thresholds.

### 4. Switch to `diff`-style Edit Format

**What**: Use search-and-replace (unified diff) format for file editing instead of whole-file rewriting.

**Why**: Aider's leaderboard shows 15-38 point improvement from switching to `diff` format. (Source: https://aider.chat/docs/leaderboards/)

**How (in NexAU)**:
- Replace whole-file write tool with a search-and-replace tool
- Tool accepts: file path, search text (exact lines to replace), replace text (new content)
- Implement validation: verify search text exists exactly once in the file before applying
- If search text not found or found multiple times, return error with context

**Expected impact**: +10-20% on code editing tasks

**Risk**: Search-and-replace can fail on whitespace mismatches. Add fuzzy matching as fallback.

### 5. Implement Silent Success / Noisy Failure for Tool Outputs

**What**: Swallow successful tool outputs and only surface errors to the agent.

**Why**: HumanLayer found that "4,000 lines of passing tests would flood the context window" and cause the agent to "lose track of the actual task." (Source: HumanLayer blog)

**How (in NexAU)**:
- After running a command/test, check the exit code
- If exit code = 0 (success): return a short summary (e.g., "Tests passed (42/42). stdout/stderr suppressed.")
- If exit code != 0 (failure): return full output
- For build commands: same pattern — silent success, verbose failure

**Expected impact**: Reduces context pollution, improves task focus. Hard to measure directly but critical for long sessions.

**Risk**: Agent might miss important warnings in successful outputs. Allow configurable verbosity.

### 6. Add Deterministic Pre-Hooks and Post-Hooks

**What**: Add hooks that run before and after agent work, with exit code 2 to re-engage the agent on failure.

**Why**: HumanLayer and Claude Code both use hooks as the highest-leverage configuration point. (Source: HumanLayer blog, Claude Code docs)

**How (in NexAU)**:
- **Pre-hook**: Run setup commands (install deps, build, generate types)
  - If exit code != 0: surface error to agent, re-engage
  - If exit code = 0: silent, nothing in context
- **Post-hook**: Run verification (typecheck, lint, test)
  - If exit code = 2: surface error output, re-engage agent
  - If exit code = 0: silent

**Expected impact**: +3-8% on tasks requiring verification. Prevents agent from thinking it's done when it's not.

**Risk**: Hooks add overhead. Keep them fast (< 30 seconds). Use test subsets.

### 7. Add Sub-Agent Support for Context Isolation

**What**: Create a sub-agent system that can delegate investigation tasks to isolated context windows.

**Why**: Claude Code and HumanLayer both identify sub-agents as a "particularly powerful lever" for maintaining coherency across many sessions. (Source: Claude Code docs, HumanLayer blog)

**How (in NexAU)**:
- Add a `delegate` tool that creates a sub-agent with:
  - Isolated context window (fresh conversation)
  - Specific task description
  - Subset of tools (read-only for investigation)
- Sub-agent returns findings to parent
- Parent receives summary, not full sub-agent conversation

**Expected impact**: +5-10% on complex multi-step tasks. Reduces context pollution in parent thread.

**Risk**: Increased latency and cost. Use sub-agents only for truly independent subtasks.

### 8. Add Edit Format Validation

**What**: When the agent writes a file, validate the edit format before applying.

**Why**: Aider's data shows that "well-formed" responses strongly correlate with success. Models with >90% well-formed rate score much higher. (Source: https://aider.chat/docs/leaderboards/)

**How (in NexAU)**:
- For search-and-replace edits:
  - Verify search text exists exactly once in the file
  - If not found: return error with surrounding context to help agent correct
  - If found multiple times: return error asking for more specific search
- For whole-file writes:
  - Verify the file is syntactically valid (parse with language-appropriate parser)
  - If invalid: return error with parse error details

**Expected impact**: +5-10% on code editing tasks by reducing malformed edit attempts

**Risk**: Strict validation could reject valid but unusual edits. Add fallback modes.

### 9. Implement Progressive Skill Loading

**What**: Create a skill system that loads domain-specific instructions only when relevant.

**Why**: Claude Code and OpenHands both use skills for progressive disclosure — "only load what's needed." Over-specified CLAUDE.md causes Claude to ignore half of it. (Source: Claude Code docs, OpenHands docs)

**How (in NexAU)**:
- Create skill files with YAML frontmatter specifying trigger conditions
- Path-triggered skills: load when agent reads/edits files matching glob patterns
- Task-triggered skills: load when agent is working on specific task types
- Repo skills: always included (project conventions)

**Expected impact**: +2-5% by reducing context noise and improving instruction adherence

**Risk**: Skills add complexity to the agent configuration. Start with 2-3 skills max.

### 10. Add Retry Logic with Exponential Backoff

**What**: Implement retry logic for API calls with exponential backoff.

**Why**: API failures are common in long-running agent sessions. OpenHands has a dedicated exception handling system with typed, provider-agnostic exceptions. (Source: OpenHands SDK docs)

**How (in NexAU)**:
- Retry on status codes: 429 (rate limit), 500, 502, 503 (server errors)
- Backoff: 1s → 2s → 4s → 8s → 16s (max 5 retries)
- Jitter: add random ±25% to each wait time
- After max retries: return error to agent with clear message

**Expected impact**: Reduces failure rate by 80-90% on transient errors. Critical for production reliability.

**Risk**: Long retry chains could exceed task timeout. Set max total retry time to 60 seconds.

---

# References

| # | URL | Content |
|---|-----|---------|
| 1 | https://docs.e2b.dev/sandbox | E2B sandbox lifecycle, persistence, forking |
| 2 | https://docs.e2b.dev/sandbox/persistence.md | Sandbox pause/resume, auto-pause, timeout |
| 3 | https://docs.e2b.dev/sandbox/fork.md | Sandbox forking for parallel execution |
| 4 | https://docs.e2b.dev/agents/claude-code | Claude Code in E2B sandbox |
| 5 | https://docs.e2b.dev/agents/codex | Codex in E2B sandbox |
| 6 | https://docs.e2b.dev/filesystem/read-write.md | E2B filesystem operations |
| 7 | https://docs.e2b.dev/commands.md | E2B command execution |
| 8 | https://docs.e2b.dev/use-cases/coding-agents.md | E2B coding agents overview |
| 9 | https://code.claude.com/docs/en/best-practices | Claude Code best practices |
| 10 | https://www.anthropic.com/engineering/swe-bench-sonnet | Anthropic SWE-bench agent scaffold |
| 11 | https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents | HumanLayer harness engineering |
| 12 | https://docs.openhands.dev/sdk/arch/agent.md | OpenHands agent architecture |
| 13 | https://docs.openhands.dev/sdk/arch/events.md | OpenHands event system |
| 14 | https://docs.openhands.dev/sdk/arch/condenser.md | OpenHands condenser architecture |
| 15 | https://docs.openhands.dev/sdk/arch/conversation.md | OpenHands conversation architecture |
| 16 | https://docs.openhands.dev/sdk/arch/tool-system | OpenHands tool system & MCP |
| 17 | https://docs.openhands.dev/sdk/arch/design | OpenHands design principles |
| 18 | https://docs.openhands.dev/sdk/guides/agent-custom.md | OpenHands custom agent creation |
| 19 | https://docs.openhands.dev/sdk/guides/context-condenser.md | OpenHands context condenser guide |
| 20 | https://docs.openhands.dev/sdk/guides/skill | OpenHands skills & context guide |
| 21 | https://docs.openhands.dev/sdk/guides/llm-error-handling.md | OpenHands error handling |
| 22 | https://docs.openhands.dev/overview/skills/path.md | OpenHands path-triggered rules |
| 23 | https://aider.chat/docs/leaderboards/ | Aider LLM leaderboards |
| 24 | https://aider.chat/docs/more/edit-formats.html | Aider edit formats |
| 25 | https://cognition.com/blog | Cognition Devin blog posts |
| 26 | https://www.tbench.ai/ | Terminal Bench official site |
| 27 | https://github.com/langchain-ai/deepagents | LangChain Deep Agents GitHub |
| 28 | https://www.humanlayer.dev/blog | HumanLayer blog (context forking, context engineering) |