Context Window Engineering: Compaction, Retrieval and Why Long Sessions Degrade

Anyone who has run a coding agent for a couple of hours knows the shape of the failure. The first half hour is genuinely impressive. By hour two it is re-reading files it read at the start, re-deriving decisions it already made, and confidently reintroducing a bug it fixed forty minutes ago.

The intuitive explanation is that it "ran out of room". It usually did not. The window was nowhere near full.

What actually happened is that the signal-to-noise ratio of the context collapsed, and every extra token spent on stale material made the next response slightly worse.

Bigger windows did not fix this

The standard reaction to context problems is to reach for a bigger window. Million-token windows exist. They did not make the problem go away, for three reasons that are worth separating.

Attention is not uniform. Retrieval accuracy varies systematically with where in the context a fact sits. Material at the beginning and end is recalled reliably; material buried in the middle of a very long context is recalled less reliably. Putting something in the window is not the same as the model using it.

Irrelevant context is not free — it is negative. Twenty files when three are relevant is not "the three plus some harmless extra". The seventeen compete for attention, and the model's probability of citing the wrong one goes up. This is the finding that most changes how you build: removing context can improve accuracy.

Contradiction accumulates. A long session contains the first attempt, the error, the second attempt, the correction, and a decision that reversed an earlier one. All of it is present as equally-weighted text. Nothing in the transcript says "that part is obsolete now" — and so the model sees the abandoned approach with the same clarity as the final one.

This bundle of effects has acquired a name — context rot — and it is the reason context engineering became a discipline rather than a prompt-writing tip.

Compaction: what to keep when you cannot keep everything

At some point the session must be compressed. The naive approach is to summarise the transcript into prose. That is better than truncation and much worse than what you can do with structure.

The insight is that a coding session contains several different kinds of information, each with a different compression ratio and a different loss profile.

The last row matters more than it looks. A resolved error is worse than useless in context — it is a vivid, detailed description of code that no longer exists, sitting in a window where the model treats vividness as relevance.

Structural compaction of code

The biggest single win is not compressing conversation, it is compressing files. A module's public shape is a fraction of its bytes and carries most of what an agent needs to reason about calling it.

src/lib/api-posts.ts  (6.2 KB → 340 bytes)

export const postBodySchema: ZodObject<…>
export type PostBody = z.infer<typeof postBodySchema>
export const POST_SELECT = { … } as const
export async function createPostFromApi(body: PostBody, authorId: string): Promise<Post>
export async function updatePostFromApi(id: string, body: Partial<PostBody>): Promise<Post>

// internal: freeSlug, connectTags, resolveContent, normaliseKeywords

Twenty times smaller, and sufficient for every question except "how is this implemented?" — at which point the agent reads the file. Generate it from the AST, not from the model: tsc's declaration emit, ts-morph, tree-sitter, or ctags will all do it, deterministically and for free.

The same idea applies to the repository as a whole. A file tree plus per-file one-liners is a map; the files themselves are the territory. An agent that has the map can navigate to the territory. An agent handed the whole territory has no map.

Checkpoint structurally, not in prose

When a session must compact, write a structured document, not a summary paragraph:

## Task
Add rate limiting to the public API.

## Decisions (do not revisit)
- Token bucket, not sliding window — matches the existing quota code in billing/
- Limits live in Postgres, not Redis — we are not adding a dependency for this
- 429 carries Retry-After; clients already handle it

## Done
- src/lib/rate-limit.ts — bucket implementation, 14 tests passing
- prisma/schema.prisma — RateLimitBucket model, migration applied

## In progress
- src/app/api/v1/posts/route.ts — limiter wired for POST, not yet for PATCH

## Rejected
- Middleware-level limiting: cannot see the resolved API key, which is the bucket key

## Next
- PATCH and DELETE handlers
- Integration test for the 429 path

Every heading answers a different question the agent will otherwise have to re-derive — and "Rejected" is the section that pays for itself, because without it the agent will propose the middleware approach again within two turns.

The other reason structure beats prose: it is diffable. You can update one section of the checkpoint without rewriting the whole thing, which means the durable decisions stay byte-identical across compactions instead of being paraphrased into drift.

Retrieval beats stuffing

The instinct "put the codebase in the context" is wrong at almost any repository size. The alternative is to give the agent the ability to look things up, and keep the resident context small.

This changes what you build. Instead of a large assembled prompt, you provide:

• A map — file tree, module summaries, an architecture note. Small and always present.
• Search — grep, symbol lookup, a semantic index. Cheap and on demand.
• Read — precise, ranged file access. Pulled only when needed.

A graph index is a meaningful upgrade over plain text search when the questions are structural: what calls this?, what would break if I change this signature?, what is the transitive blast radius? Those are graph traversals, not string matches, and answering them with grep means reading far more than necessary.

The practical rule that falls out: context should be pulled, not pushed. Anything pushed into every request must earn its place permanently.

Layout for the cache, because it is nearly free money

Prompt caching gives a large discount on the prefix of a request that has not changed since last time. That makes context layout an economic decision as well as an accuracy one.

┌─────────────────────────────┐
│ System prompt               │  never changes  ─┐
│ Tool definitions            │  never changes   │ cacheable prefix
│ Project conventions         │  rarely changes  │
│ Architecture map            │  rarely changes ─┘
├─────────────────────────────┤
│ Compacted session state     │  changes at checkpoints
├─────────────────────────────┤
│ Recent turns                │  changes every turn
│ Current tool results        │  changes every turn
└─────────────────────────────┘

Stable first, volatile last. One mutation near the top — a timestamp, a re-ordered tool list, a file list that sorts differently — invalidates everything after it and costs full price on every subsequent request.

Two failure modes to design out specifically: do not interleave volatile and stable content, and do not reorder stable content between requests. Both are easy to do accidentally when the prompt is assembled from a set rather than a list.

Failure modes worth naming

Context poisoning. An early hallucination — a function that does not exist, a wrong assumption about a schema — stays in context and gets reinforced by every subsequent reference. The fix is not correction, it is removal: drop the turn and restate the truth. Correcting in place leaves the wrong version in the window alongside the right one.

Distraction. With enough accumulated history, a model starts repeating past actions rather than reasoning about the current state. The tell is re-reading a file it has already read. It is a signal to compact, not to prompt harder.

Clash. Two parts of the context disagree — an old decision and a newer one that superseded it. The model has no principled way to pick. This is what the "Decisions (do not revisit)" section exists to prevent.

Confusion by tool. Fifty tool definitions makes tool selection measurably worse. Scope tools by phase — the planning agent does not need write access, and does not need the write tools described to it either. Definitions are context too.

What to actually do on Monday

1. Instrument it. Log tokens per request, split into system, resident context, and turns. You cannot manage what you have never measured, and most teams have never looked.
2. Generate a structural map of the repository and make it resident. AST-derived, regenerated in CI, never hand-written.
3. Write file summaries, not file contents, into context by default, and let the agent read the full file when it decides to.
4. Checkpoint with structure, with an explicit non-negotiable decisions section.
5. Drop resolved errors and old tool output aggressively. Aim to shrink context at checkpoints, not to re-summarise it at the same size.
6. Order the prompt for the cache and stop reordering it.

The discipline underneath all six is one sentence, and it is the same one that makes a good technical document good:

Every token in the window is competing for the model's attention. Make each one earn its place.

Part of a series on the highest-friction problems in modern software engineering.