Context Management

Module 3 · Master Course

90 min · The hardest engineering problem in production harnesses

The hardest problem

Context is the model's working memory. When it fills, the agent loses track of its task. This is measurable, not theoretical.

Module 0.3's anchor: tool outputs 67.6%, history 18%, and history is the only layer that grows unboundedly. By the end: why context rots, five strategies, the priority stack, and a compaction function you'll build.

3.1 Why Context Rots

Lost in the Middle

Stanford (Liu et al. 2023, arXiv:2307.03172): 30%+ performance drop when critical info is mid-context — even in million-token models.

The U-shape: strong attention at start (system prompt) and end (latest user message). The middle — where history piles up — is a desert.

Implication: where you place info matters as much as what it is.

The 256k cliff

Hard quality degradation around 256k tokens regardless of stated limit. A "1M token" model performs measurably worse above 256k.

Effective context < nominal context. The harness must keep effective context below the cliff — managing proactively, not just fitting within the nominal limit.

Where the tokens go (and grow)

Layer	Share	Grows?
Tool outputs	67.6%	per-turn, compressible
History	~18%	UNBOUNDED — the rot source
Tool defs	10.7%	fixed
System prompt	3.4%	fixed

History is the only layer that grows every turn. Compaction targets it.

3.2 Compaction & Summarization

Preserve vs. Compress

Preserve verbatim:
• original task/goal
• key decisions + rationale
• current state (done/pending)
• recent 3-5 turns
• errors + resolutions

Compress/discard:
• verbose tool outputs (acted on)
• redundant reasoning chains
• "ok"/"done" confirmations
• exploratory dead-ends

ACON: 26–54% token reduction, 95%+ accuracy preservation. Compaction done well doesn't degrade performance.

Observation masking

Keep the reasoning chain (Thought); hide old raw tool outputs (Observation).

Before: [Thought: read auth.ts] [Observation: 248 lines, full content, 1840 tokens]
After:  [Thought: read auth.ts] [Observation: auth.ts, 248 lines, contains validateToken — 40 tokens]

Model sees its decision history without the noise. Cost: raw data gone unless retained in a side store for JIT retrieval.

Synchronous vs async compaction

	Synchronous	Async
When	in-loop, blocks next call	background, loop continues
Latency	adds on compaction turn	none
Complexity	simple	race conditions possible
Production default	yes — trigger at threshold	rare

Most production harnesses: synchronous compaction at ~50% of context window. Tunable: too low wastes calls; too high lets rot set in.

3.3 JIT Retrieval & Memory

The 3-tier design (Claude Code benchmark)

Tier	What	Loaded	Cost
T1 Index	~150 char summaries	always	low
T2 Topic files	detailed content	on demand	medium
T3 Raw logs	full history	search only	high

Like a database index: don't load the table; load the index, fetch rows on demand. The model never pays T3 cost unless it explicitly searches.

Three retrieval methods

Method	Best for	Cost
Keyword (ripgrep)	exact code: "find validateToken"	fast, deterministic
Vector (embeddings)	semantic: "auth approach discussions"	infra; can hallucinate relevance
File read	known path: "read decisions.md"	no search needed

Provide all three; let the model choose. Tool surface (Module 2) determines retrieval capability.

3.4 Prompt Assembly

The priority stack

System Message      ← HIGHEST attention (first)
  ↓ Tool Definitions
  ↓ Memory (T1 index)
  ↓ History           ← WEAK attention (middle) — compaction target
  ↓ User Message      ← HIGH attention (last)

Core rule: anything placed EARLY gets higher effective attention. This is Lost-in-the-Middle operationalized.

The original task, placed in the system prompt, survives better than the same task in turn-1 history.

Thin vs. dense system prompts

Ultra-thin (Pi: <1k)
co-evolves with model; relies on competence

Dense (Claude Code: ~40k)
explicit control; fights capability growth

Thickness spectrum (Module 0.1) applies to prompt assembly. Question: how much behavior do you encode vs. delegate?

System prompt caching

Anthropic cache_control: system prompt + tool defs cached server-side. Repeated calls with same prefix are cheaper + faster.

System prompt and tool defs are FIXED across turns → caching turns recurring cost into one-time. Reward stability; dynamic prompts invalidate the cache.

Four anti-patterns

Sliding window. Drop oldest when full. Catastrophic — loses the task. Cure: compaction (preserve task, compress middle).

No context management. Fill until error. Works short-term; fails long. Cure: any of the five strategies.

Unbounded subagent return. Full context to parent. Cure: 1-2k bound.

Premature compaction. Threshold too low; loses needed detail. Cure: tune; ACON 26-54% is the target.

Takeaways

Context rots measurably — 30%+ mid-context drop; 256k cliff; history grows unboundedly.
Compaction: preserve task/decisions/recent/errors; compress verbose outputs. ACON: 26-54% reduction, 95%+ accuracy.
JIT retrieval: T1 index always; T2 on demand; T3 search-only. Like a database index.
Priority stack: early = high attention. System prompt first, user last, history middle (rot target).
Cache stability: cache_control rewards fixed prompts.

Next: Module 4 — Memory Architecture. Continuity across sessions.