Course: Master Course — Harness Engineering (12+ Hours) Module: 3 — Context Management Duration: 90 minutes Level: Senior Engineer and above Prerequisites: Module 1 (the loop), Module 2 (tool outputs as 67.6% of context)
The hardest engineering problem in production harnesses.
After completing this module, you will be able to:
Context is the model's working memory. When it fills, the agent loses track of its task. This is measurable, not theoretical.
Stanford's "Lost in the Middle" study (Liu et al., 2023, arXiv:2307.03172) found a 30%+ performance drop when critical information is buried mid-context — even in models with million-token stated context limits. The model attends strongly to the beginning and end of its context; the middle is a performance desert.
This is not a quirk of weak models. It persists across model sizes and stated context lengths. The implication for harness engineering: where you place information in context matters as much as what the information is. The system prompt (beginning) and the latest user message (end) get strong attention; the tool output from turn 14 (middle) gets weak attention — exactly when the agent needs it most.
There is often a hard degradation threshold around 256k tokens regardless of the model's stated context limit. A model that "supports" 1M tokens may perform measurably worse above 256k. The stated limit is a ceiling, not a quality target.
The practical consequence: effective context is smaller than nominal context. A harness that fills 800k of a 1M window is operating past the quality cliff. The harness's job is to keep effective context below the degradation threshold — which means managing context proactively, not just fitting within the nominal limit.
In a typical production session:
| Layer | Share | Implication |
|---|---|---|
| Tool outputs | ~67.6% | The dominant lever; Module 2's truncation rule |
| History + rest | ~18% | Grows with every turn; the compaction target |
| Tool definitions | ~10.7% | Fixed at registration; reducible by tool-count cuts |
| System prompt | ~3.4% | Small but high-attention (it's at the beginning) |
| Model logic | ~1.6% | Not context; the model itself |
The asymmetry that drives this entire module: history grows unboundedly with turns; everything else is fixed or bounded. A 50-turn session has 50 turns of history piling into the middle of context, where attention is weakest. This is context rot — the slow degradation of the model's grip on its task as history accumulates.
The primary defense against history growth. Summarize the old; keep the recent raw.
The compaction decision: when context approaches a threshold, summarize older history. The question is what to keep and what to compress.
Preserve (verbatim or near-verbatim):
Compress or discard:
The ACON approach (a published context-optimization method) reports 26–54% token reduction with 95%+ accuracy preservation. The method: identify compressible segments (verbose tool outputs, redundant reasoning), summarize them into compact decision records, and replace the raw segment with the summary in context.
The 95% accuracy figure is the key claim: compaction, done well, does not meaningfully degrade task performance. The 26–54% reduction is the leverage — nearly doubling effective context budget without quality loss.
A specific compaction technique Claude Code uses: keep the reasoning chain visible, hide old raw tool outputs. The model's Thought (reasoning about what to do) is preserved; the Observation (the 1840-token file dump) is replaced with a one-line summary ("read auth.ts: 248 lines, contains validateToken function").
The benefit: the model sees its own decision history (why it did what it did) without the noise of the raw data that informed those decisions. The cost: if the raw data is needed later (e.g., to re-examine a specific line), it's gone — unless the harness retains it in a side store for JIT retrieval (Module 3.3).
Synchronous: compaction happens in the loop, blocking the next model call. Simple; adds latency on the compaction turn. The harness sees the compacted context immediately.
Asynchronous: a background process compacts older history while the loop continues. No latency penalty; but the loop may operate on un-compacted context for a turn or two before the compaction lands. More complex; race conditions possible.
Most production harnesses use synchronous compaction triggered at a token threshold (e.g., compact when history exceeds 50% of the context window). The threshold is tunable; too low wastes tokens on compaction calls, too high lets context rot set in before compaction fires.
Read this in real code: Tau's non-destructive compaction (DD-21). Tau implements compaction as an append-only operation on a JSONL session tree. A
CompactionEntrycarriesreplaces_entry_ids— during replay, those messages are swapped for a summary. The on-disk record is never rewritten. You can always re-derive the fuller history by replaying without the compaction entry. This is event-sourcing applied to agent context windows. Seesrc/tau_agent/session/—entries.py,tree.py, andcontext_window.py.
Claude Code's compaction logic is worth reading as the production reference. In ~50 lines of Python (per the Module 0.1 teaching doc's lab note), it:
The Module 3 lab reproduces this in a minimal form.
Don't load everything into context. Load an index; retrieve full content on demand.
Claude Code's industry-benchmark memory design has three tiers:
| Tier | What | When loaded | Token cost |
|---|---|---|---|
| Tier 1: Index | Lightweight summaries (~150 chars each) | Always in context | Low |
| Tier 2: Topic files | Detailed content per topic | On demand (model requests) | Medium |
| Tier 3: Raw logs | Full interaction history | Search only (never bulk-loaded) | High |
The principle: the index is always present, cheaply; detail is loaded only when needed. The model sees the index, knows what topics exist, and can request a specific topic file when relevant. It never pays the token cost of Tier 3 unless it explicitly searches.
This is the JIT (just-in-time) retrieval pattern applied to memory. It's the same idea as a database index: you don't load the whole table; you load an index, then fetch rows on demand.
Three retrieval methods, each with a use case:
Keyword search (ripgrep, grep): exact-match or regex. Best for code ("find where validateToken is defined"). Fast, deterministic, no infrastructure.
Vector retrieval (embeddings + similarity search): semantic match. Best for concepts ("find discussions about authentication approach"). Requires an embedding model and a vector store. Can hallucinate relevance.
File read: direct access to a known path. Best when the model knows exactly what it needs ("read the decisions.md file"). No search needed.
The harness's job: provide tools for all three, let the model choose. A harness with only keyword search can't do semantic retrieval; a harness with only vector search is unreliable for exact code lookups. The tool surface (Module 2) determines retrieval capability.
When multiple retrieval sources compete for context, the harness allocates a budget:
Total context budget: e.g., 100k effective tokens
- System prompt: 3k (fixed)
- Tool definitions: 10k (fixed)
- Tier 1 index: 5k (always loaded)
- History (recent): 20k (recent N turns)
- Tier 2 (on demand): 40k (retrieved topic files)
- Headroom: 22k (reserved for tool outputs this turn)
The budget is a design decision. Too little headroom → tool outputs overflow into history, triggering premature compaction. Too much headroom → wasted capacity. The Module 3 lab has learners build a JIT loader that fills a budget in priority order.
How the harness assembles the final message array. This directly determines what the model knows and prioritizes.
The standard priority stack (OpenAI's recommended ordering):
System Message ← highest attention (it's first)
↓ Tool Definitions
↓ Memory (Tier 1 index)
↓ Conversation History
↓ User Message ← high attention (it's last)
Core rule: anything placed early in the stack gets higher effective attention. This is the Lost-in-the-Middle effect operationalized. The system prompt is first → strongest attention. The user's latest message is last → also strong attention (recency). History is in the middle → weakest attention, which is why compaction targets it.
Ultra-thin (Pi: <1,000 tokens): minimal instructions; let the model figure it out. Co-evolves with model improvements (Module 1's future-proof test). Risk: less control over behavior; relies on model competence.
Dense (Claude Code: ~40,000 tokens estimated): explicit instructions for every scenario; harness controls behavior directly. Reliable, predictable, auditable. Risk: fights model capability growth; the prompt becomes a constraint.
The thickness spectrum (Module 0.1) applies to prompt assembly as much as to any other layer. The design question: how much behavior do you encode in the prompt vs. delegate to the model?
Anthropic's cache_control feature: the system prompt and tool definitions are cached on Anthropic's side, so repeated calls with the same prefix are cheaper and faster. This matters because the system prompt and tool definitions are fixed across turns — caching them turns a recurring cost into a one-time cost.
When to use it: any harness with a large, stable system prompt (>1k tokens) making multiple calls per session. The cache hit reduces latency and cost significantly.
When it doesn't help: if the system prompt changes every turn (dynamic prompt assembly), the cache invalidates. Caching rewards stability.
The format of history messages affects both token efficiency and model legibility:
<thought>...</thought><action>...</action><observation>...</observation>. More token-efficient for structured exchanges.The choice interacts with tool-output formatting (Module 2): if tool outputs are JSON, history containing them is JSON-mixed-with-text unless the harness normalizes the format.
Drop the oldest messages when context fills. Simple — but catastrophic loss of early context. The model loses the original task. Cure: compaction (preserve the task; compress the middle), not a blind sliding window.
Fill context until error. Early Aider, Pi (which relies on minimal tool output rather than active management). Works for short sessions; fails on long tasks. Cure: any of the five strategies.
A subagent returns full context to the parent (Module 1.3, Module 2). The parent's context fills with the subagent's noise. Cure: 1–2k token bound on subagent returns; summary-only by default.
Compacting too aggressively (threshold too low) wastes tokens on compaction calls and loses detail the model needs. Cure: tune the threshold; the ACON 26–54% reduction is the target efficiency, not 90%.
| Term | Definition |
|---|---|
| Lost in the Middle | 30%+ performance drop when critical info is mid-context; persists across model sizes |
| 256k degradation cliff | Hard quality threshold around 256k tokens regardless of stated limit |
| Compaction | Summarize older history; preserve task/decisions/recent; discard verbose outputs |
| Observation masking | Keep reasoning chain; hide old raw tool outputs |
| JIT retrieval | Load an index always; fetch full content on demand (3-tier design) |
| Priority stack | System → tools → memory → history → user; early = higher attention |
| cache_control | Anthropic's system-prompt caching; turns recurring cost into one-time |
| Context budget | Token allocation per layer; reserves headroom for tool outputs |
See 07-lab-spec.md. Build a compaction function (preserve task/decisions, compress tool outputs). Build a JIT context loader that fills a 100k budget in priority order. Run a long session, trigger compaction, and measure the before/after token count and task-performance difference.
# Module 3 — Context Management
**Course**: Master Course — Harness Engineering (12+ Hours)
**Module**: 3 — Context Management
**Duration**: 90 minutes
**Level**: Senior Engineer and above
**Prerequisites**: Module 1 (the loop), Module 2 (tool outputs as 67.6% of context)
> *The hardest engineering problem in production harnesses.*
---
## Learning Objectives
After completing this module, you will be able to:
1. Explain *why* context rots — the Lost-in-the-Middle effect, the 256k degradation cliff, and the token breakdown that makes tool outputs the dominant share.
2. Choose among the five context-management strategies (compaction, observation masking, JIT retrieval, note-taking, subagent delegation) and state the tradeoff each accepts.
3. Implement a compaction function that preserves decisions and current state while discarding verbose tool outputs.
4. Design the prompt-assembly priority stack and explain why anything placed early gets higher effective attention.
5. Diagnose a context-rot failure from token-usage logs and prescribe the correct intervention.
---
# 3.1 — Why Context Rots
*Context is the model's working memory. When it fills, the agent loses track of its task. This is measurable, not theoretical.*
## The Lost-in-the-Middle effect
Stanford's "Lost in the Middle" study (Liu et al., 2023, arXiv:2307.03172) found a **30%+ performance drop** when critical information is buried mid-context — even in models with million-token stated context limits. The model attends strongly to the beginning and end of its context; the middle is a performance desert.
This is not a quirk of weak models. It persists across model sizes and stated context lengths. The implication for harness engineering: *where you place information in context matters as much as what the information is*. The system prompt (beginning) and the latest user message (end) get strong attention; the tool output from turn 14 (middle) gets weak attention — exactly when the agent needs it most.
## The 256k degradation cliff
There is often a hard degradation threshold around **256k tokens** regardless of the model's stated context limit. A model that "supports" 1M tokens may perform measurably worse above 256k. The stated limit is a ceiling, not a quality target.
The practical consequence: effective context is smaller than nominal context. A harness that fills 800k of a 1M window is operating past the quality cliff. The harness's job is to keep *effective* context below the degradation threshold — which means managing context proactively, not just fitting within the nominal limit.
## The token breakdown (Module 0.3's anchor, restated)
In a typical production session:
| Layer | Share | Implication |
| --- | --- | --- |
| Tool outputs | **~67.6%** | The dominant lever; Module 2's truncation rule |
| History + rest | ~18% | Grows with every turn; the compaction target |
| Tool definitions | ~10.7% | Fixed at registration; reducible by tool-count cuts |
| System prompt | **~3.4%** | Small but high-attention (it's at the beginning) |
| Model logic | ~1.6% | Not context; the model itself |
The asymmetry that drives this entire module: **history grows unboundedly with turns; everything else is fixed or bounded.** A 50-turn session has 50 turns of history piling into the middle of context, where attention is weakest. This is context rot — the slow degradation of the model's grip on its task as history accumulates.
---
# 3.2 — Compaction and Summarization
*The primary defense against history growth. Summarize the old; keep the recent raw.*
## What to preserve vs. discard
The compaction decision: when context approaches a threshold, summarize older history. The question is *what to keep* and *what to compress*.
**Preserve (verbatim or near-verbatim):**
- The original task / user goal (lose this and the agent forgets why it's working)
- Key decisions made and their rationale
- The current state (what's been done, what's pending)
- Recent turns (the last 3–5, for coherence)
- Errors and their resolutions (the model needs to know what it already tried)
**Compress or discard:**
- Verbose tool outputs that have been acted on (the 1840-token file read that's been summarized into a decision)
- Redundant reasoning chains (the model often re-reasons the same point across turns)
- Successful confirmations ("ok", "done", "file written") that add no information
- Exploratory dead-ends (tool calls that returned nothing useful)
## The ACON approach
The ACON approach (a published context-optimization method) reports **26–54% token reduction with 95%+ accuracy preservation**. The method: identify compressible segments (verbose tool outputs, redundant reasoning), summarize them into compact decision records, and replace the raw segment with the summary in context.
The 95% accuracy figure is the key claim: compaction, done well, does not meaningfully degrade task performance. The 26–54% reduction is the leverage — nearly doubling effective context budget without quality loss.
## Observation masking
A specific compaction technique Claude Code uses: **keep the reasoning chain visible, hide old raw tool outputs.** The model's Thought (reasoning about what to do) is preserved; the Observation (the 1840-token file dump) is replaced with a one-line summary ("read auth.ts: 248 lines, contains validateToken function").
The benefit: the model sees its own decision history (why it did what it did) without the noise of the raw data that informed those decisions. The cost: if the raw data is needed later (e.g., to re-examine a specific line), it's gone — unless the harness retains it in a side store for JIT retrieval (Module 3.3).
## Synchronous vs. asynchronous compaction
**Synchronous**: compaction happens in the loop, blocking the next model call. Simple; adds latency on the compaction turn. The harness sees the compacted context immediately.
**Asynchronous**: a background process compacts older history while the loop continues. No latency penalty; but the loop may operate on un-compacted context for a turn or two before the compaction lands. More complex; race conditions possible.
Most production harnesses use synchronous compaction triggered at a token threshold (e.g., compact when history exceeds 50% of the context window). The threshold is tunable; too low wastes tokens on compaction calls, too high lets context rot set in before compaction fires.
> **Read this in real code: Tau's non-destructive compaction (DD-21).** Tau implements compaction as an append-only operation on a JSONL session tree. A `CompactionEntry` carries `replaces_entry_ids` — during replay, those messages are swapped for a summary. The on-disk record is never rewritten. You can always re-derive the fuller history by replaying without the compaction entry. This is event-sourcing applied to agent context windows. See `src/tau_agent/session/` — `entries.py`, `tree.py`, and `context_window.py`.
## Claude Code's compaction — the reference
Claude Code's compaction logic is worth reading as the production reference. In ~50 lines of Python (per the Module 0.1 teaching doc's lab note), it:
1. Detects when history exceeds the threshold.
2. Identifies compressible segments (tool outputs older than N turns).
3. Calls the model to summarize those segments into decision records.
4. Replaces the raw segments with the summaries in context.
5. Preserves the original task, recent turns, and key decisions verbatim.
The Module 3 lab reproduces this in a minimal form.
---
# 3.3 — JIT Retrieval and Memory Integration
*Don't load everything into context. Load an index; retrieve full content on demand.*
## The 3-tier design (Claude Code's memory model)
Claude Code's industry-benchmark memory design has three tiers:
| Tier | What | When loaded | Token cost |
| --- | --- | --- | --- |
| **Tier 1: Index** | Lightweight summaries (~150 chars each) | Always in context | Low |
| **Tier 2: Topic files** | Detailed content per topic | On demand (model requests) | Medium |
| **Tier 3: Raw logs** | Full interaction history | Search only (never bulk-loaded) | High |
The principle: **the index is always present, cheaply; detail is loaded only when needed.** The model sees the index, knows what topics exist, and can request a specific topic file when relevant. It never pays the token cost of Tier 3 unless it explicitly searches.
This is the JIT (just-in-time) retrieval pattern applied to memory. It's the same idea as a database index: you don't load the whole table; you load an index, then fetch rows on demand.
## When to use vector retrieval vs. keyword search vs. file read
Three retrieval methods, each with a use case:
**Keyword search** (ripgrep, grep): exact-match or regex. Best for code ("find where `validateToken` is defined"). Fast, deterministic, no infrastructure.
**Vector retrieval** (embeddings + similarity search): semantic match. Best for concepts ("find discussions about authentication approach"). Requires an embedding model and a vector store. Can hallucinate relevance.
**File read**: direct access to a known path. Best when the model knows exactly what it needs ("read the decisions.md file"). No search needed.
The harness's job: provide tools for all three, let the model choose. A harness with only keyword search can't do semantic retrieval; a harness with only vector search is unreliable for exact code lookups. The tool surface (Module 2) determines retrieval capability.
## Context budget allocation
When multiple retrieval sources compete for context, the harness allocates a budget:
```
Total context budget: e.g., 100k effective tokens
- System prompt: 3k (fixed)
- Tool definitions: 10k (fixed)
- Tier 1 index: 5k (always loaded)
- History (recent): 20k (recent N turns)
- Tier 2 (on demand): 40k (retrieved topic files)
- Headroom: 22k (reserved for tool outputs this turn)
```
The budget is a design decision. Too little headroom → tool outputs overflow into history, triggering premature compaction. Too much headroom → wasted capacity. The Module 3 lab has learners build a JIT loader that fills a budget in priority order.
---
# 3.4 — Prompt Assembly Engineering
*How the harness assembles the final message array. This directly determines what the model knows and prioritizes.*
## The priority stack
The standard priority stack (OpenAI's recommended ordering):
```
System Message ← highest attention (it's first)
↓ Tool Definitions
↓ Memory (Tier 1 index)
↓ Conversation History
↓ User Message ← high attention (it's last)
```
**Core rule: anything placed early in the stack gets higher effective attention.** This is the Lost-in-the-Middle effect operationalized. The system prompt is first → strongest attention. The user's latest message is last → also strong attention (recency). History is in the middle → weakest attention, which is why compaction targets it.
## Ultra-thin vs. dense system prompts
**Ultra-thin (Pi: <1,000 tokens)**: minimal instructions; let the model figure it out. Co-evolves with model improvements (Module 1's future-proof test). Risk: less control over behavior; relies on model competence.
**Dense (Claude Code: ~40,000 tokens estimated)**: explicit instructions for every scenario; harness controls behavior directly. Reliable, predictable, auditable. Risk: fights model capability growth; the prompt becomes a constraint.
The thickness spectrum (Module 0.1) applies to prompt assembly as much as to any other layer. The design question: *how much behavior do you encode in the prompt vs. delegate to the model?*
## System prompt caching
Anthropic's `cache_control` feature: the system prompt and tool definitions are cached on Anthropic's side, so repeated calls with the same prefix are cheaper and faster. This matters because the system prompt and tool definitions are *fixed across turns* — caching them turns a recurring cost into a one-time cost.
**When to use it**: any harness with a large, stable system prompt (>1k tokens) making multiple calls per session. The cache hit reduces latency and cost significantly.
**When it doesn't help**: if the system prompt changes every turn (dynamic prompt assembly), the cache invalidates. Caching rewards stability.
## Multi-turn message format
The format of history messages affects both token efficiency and model legibility:
- **Raw text**: flexible but verbose. The model parses natural language.
- **XML blocks**: structured, parseable. `<thought>...</thought><action>...</action><observation>...</observation>`. More token-efficient for structured exchanges.
- **JSON blocks**: maximally structured. Best when the harness needs to parse history programmatically (e.g., for compaction).
The choice interacts with tool-output formatting (Module 2): if tool outputs are JSON, history containing them is JSON-mixed-with-text unless the harness normalizes the format.
---
## Anti-Patterns
### The sliding window
Drop the oldest messages when context fills. Simple — but catastrophic loss of early context. The model loses the original task. Cure: compaction (preserve the task; compress the middle), not a blind sliding window.
### No context management
Fill context until error. Early Aider, Pi (which relies on minimal tool output rather than active management). Works for short sessions; fails on long tasks. Cure: any of the five strategies.
### Unbounded subagent returns (revisited)
A subagent returns full context to the parent (Module 1.3, Module 2). The parent's context fills with the subagent's noise. Cure: 1–2k token bound on subagent returns; summary-only by default.
### Premature compaction
Compacting too aggressively (threshold too low) wastes tokens on compaction calls and loses detail the model needs. Cure: tune the threshold; the ACON 26–54% reduction is the target efficiency, not 90%.
---
## Key Terms
| Term | Definition |
| --- | --- |
| **Lost in the Middle** | 30%+ performance drop when critical info is mid-context; persists across model sizes |
| **256k degradation cliff** | Hard quality threshold around 256k tokens regardless of stated limit |
| **Compaction** | Summarize older history; preserve task/decisions/recent; discard verbose outputs |
| **Observation masking** | Keep reasoning chain; hide old raw tool outputs |
| **JIT retrieval** | Load an index always; fetch full content on demand (3-tier design) |
| **Priority stack** | System → tools → memory → history → user; early = higher attention |
| **cache_control** | Anthropic's system-prompt caching; turns recurring cost into one-time |
| **Context budget** | Token allocation per layer; reserves headroom for tool outputs |
---
## Lab Exercise
See `07-lab-spec.md`. Build a compaction function (preserve task/decisions, compress tool outputs). Build a JIT context loader that fills a 100k budget in priority order. Run a long session, trigger compaction, and measure the before/after token count and task-performance difference.
---
## References
1. **Liu et al. (2023)** — *Lost in the Middle: How Language Models Use Long Contexts*. arXiv:2307.03172. The foundational context-position study.
2. **ACON** — context optimization approach; 26–54% reduction, 95%+ accuracy.
3. **Anthropic cache_control documentation** — system-prompt caching.
4. **Claude Code source** — the compaction logic reference (~50 lines).
5. **Module 0.3** — the 67.6% / 3.4% anchor.
6. **Module 2** — tool-output truncation as the first context-management lever.