Agent context window management: a practical guide for production

By the end of this guide, you'll have a working playbook for managing the context window of a production agent — one that stops crashing at turn 40, stops paying for redundant tokens, and stops losing the thread halfway through a task.

This is a how-to for engineers running agents against real tools. It assumes you've hit the wall: the run that worked in a demo, then fell over in production when the tool responses got bigger, the trajectory got longer, or a user handed the agent a task that took 30 tool calls instead of three.

Prerequisites: an agent already calling tools in production, access to the trace logs, and roughly 30 minutes to work through the steps. We'll cover context accounting, tool response shaping, trajectory compaction, and the failure modes that catch teams late.

Step 1 — Measure what's actually in the window

Before you optimise anything, you need a token budget. Most teams don't have one. They have a model limit — 200k (or 1M in beta) for Claude, 128k for GPT-4-class OpenAI models, 1M–2M for Gemini — and they assume the ceiling is the budget. It isn't.

Every turn, the window holds: the system prompt, the tool definitions, the conversation history, tool call arguments, tool responses, and the model's next output. On a long-running agent, tool responses dominate. On a tool-heavy agent, definitions dominate.

Instrument every turn. Log the token count of each component separately. You want a table that looks like this after a real run:
‍

Now you can see where the budget goes. The point isn't the numbers — it's owning them turn by turn.

Step 2 — Cap tool response size at the tool, not the model

The single biggest source of context bloat is tool responses returning more than the agent needs. A list_customers call returns 500 rows and 40 fields per row. The agent needed the ID and status of one customer.

Cap responses at the tool boundary. Three levers:

1. Default limits. Every list endpoint returns 20 items unless the agent asks for more. Not 100. Not 1000.
2. Field selection. Accept a fields parameter and honour it. If the agent asks for id,status, return only those.
3. Pagination as a first-class contract. Return a next_cursor and let the agent decide whether to fetch more. Don't dump the full result set on the first call.

Example response shape:

{
 "items": [{"id": "cus_123", "status": "active"}],
 "next_cursor": "eyJvZmZzZXQiOjIwfQ==",
 "total_estimated": 487
}



The agent gets what it needs, knows more exists, and can ask for the next page if the task requires it. In our internal customer traces, this one change typically cuts context growth materially on data-heavy agents — often by more than half. For deeper coverage of pagination and response contracts, see REST API design best practices for the agent era.

Step 3 — Prune tool definitions the agent doesn't need

Tool definitions are always in the window. Every turn. If you've registered 80 tools and the current task uses six, you're paying for 80 every turn.

Do tool selection before the run starts, not inside it. A retrieval step — semantic search over tool descriptions, or a router model that picks the top 15–25 tools based on the user's opening message — is cheaper than carrying the full catalogue.

Run the numbers: 80 tools at ~180 tokens each (typical schemas run 80–300 tokens depending on provider and complexity) is roughly 14,400 tokens per turn. Over a 30-turn trajectory, that's on the order of 430,000 tokens you paid for without the agent invoking most of them.

This matters more than most teams realise. How many tools should an AI agent have? covers the selection-accuracy ceiling — the practical limit isn't the context window, it's how well the model picks the right tool. Meaningful degradation can appear as early as 15–20 tools with noisy schemas, and by roughly 30 exposed at once it's a reliable pattern across evaluations.

Step 4 — Compact conversation history

By turn 20 of a real agent run, the history is full of tool calls the agent doesn't need to see verbatim any more. It made the call, it got the result, it acted on it. The full JSON of a list_orders response from 15 turns ago is dead weight.

Two compaction strategies, applied together:

Summarisation. Every N turns (start with N=10), replace older tool call/response pairs with a short model-generated summary: "Fetched 12 orders for customer cus_123, all shipped, IDs ord_1 through ord_12." Keep the summary, drop the raw payloads.

Elision of intermediate reasoning. The model's thinking-out-loud from turn 5 rarely matters at turn 25. Keep the final decisions and the tool calls; drop the deliberation.

Don't compact aggressively at the start of a run — the model needs the recent working set. Do compact anything older than the last 5–8 turns.

One caveat: compaction is lossy. If the agent needs to refer back to something specific ten turns later, it needs a way to fetch it. That means logging tool responses to an external store (S3, a database, whatever you have) and giving the agent a retrieve_previous_result(call_id) tool. The context window holds the summary; the external store holds the truth.

Step 5 — Set a hard budget and enforce it

Pick a soft limit and a hard limit. For a 200k model, reasonable defaults:

• Soft limit: 60% of the model window (120k tokens). At this point, trigger compaction.
• Hard limit: 80% of the model window (160k tokens). At this point, either terminate the run with a clean error, or force aggressive summarisation and continue.

Why not 100%? Because model quality degrades before you hit the limit. Studies from foundation model providers repeatedly show that recall accuracy on facts in the middle of a very long context drops well before the ceiling. Running at 90% of your window isn't running long — it's running degraded.

Wire the enforcement into the agent loop itself. Before each model call, count the projected total. If it exceeds soft, run compaction. If it exceeds hard, stop or bail.

Step 6 — Handle overflow explicitly

Even with all of the above, some tasks will genuinely exceed the window. A user asks the agent to reconcile 10,000 invoices. That doesn't fit and it shouldn't.

Design for the overflow case:

• Detect early. If a tool call is about to return more than a threshold (say 20k tokens of output), the tool should refuse and return a structured error: {"error": "response_too_large", "total_items": 10000, "suggestion": "paginate or filter"}.
• Give the agent a decomposition tool. A spawn_subagent(task, tool_subset) tool that runs a fresh context window for a bounded sub-task and returns only the result. The parent agent sees a summary; the child agent had its own clean window.
• Fail loudly. If none of the above work, terminate with a clear message. Don't silently truncate the tool response — the agent will act on partial data and you won't know until the audit.

This is where authenticated, per-user tool execution matters: the sub-agent runs as the same user, hits the same permissions, and its work shows up in the same audit trail. Shared service accounts break this whole pattern.

Common pitfalls

Trusting max_tokens on the tool response. That parameter usually controls model output, not tool payload. Cap payload size at the tool, in your code, before it returns.

Compacting the system prompt. Don't. It's static, it's cached by the provider on most platforms, and rewriting it mid-run breaks prompt caching and burns money.

Assuming bigger windows fix the problem. A 1M-token window doesn't make a badly designed agent good. It makes a badly designed agent expensive. Selection accuracy, response shaping, and trajectory discipline still decide whether the agent finishes the task correctly.

Not measuring in production. Local runs with test data don't hit the response sizes real users generate. Instrument in production or you're guessing.

Treating context management as a prompt problem. It's a runtime problem. The prompt sets intent; the runtime — tool responses, history compaction, budget enforcement — decides whether the agent actually holds together at turn 40. See AI agent guardrails: the wrong layer usually gets the blame for the same argument applied to safety.

Get the six steps above running and the agent stops falling over. Skip them and no model upgrade will save you.