Agent infrastructure
Agents in production
How many tools should an AI agent have? A deep-dive on the real ceiling, why selection accuracy matters more than context window, and how to design past it.
How many tools should an AI agent have? The honest answer: fewer than you think, and the ceiling isn't where most teams look for it. Context window size gets blamed. Model capability gets blamed. Both are red herrings. The real limits are selection accuracy, prompt bloat, and the operational surface you now have to maintain. This article makes the case that tool count is a design decision, not a scaling one — and that the per-turn working set sits between 10 and 25 well-scoped tools for most production agents, with hard caveats on either side.
We'll cover five things: why the naive "more tools equals more capable" instinct breaks, what the research and production data actually show about tool selection degradation, how context window consumption sneaks up on you, the architectural moves that let you exceed the naive ceiling, and how to think about tool inventory as a product surface rather than a bag of functions.
The intuition most teams start with: an agent's capability equals the union of the tools it can call. Add more tools, cover more ground. That model works for the first ten or fifteen tools. It stops working somewhere between 15 and 40, and the failure is quiet.
The published evidence is unambiguous. The RAG-MCP paper benchmarked tool-selection accuracy as the tool inventory grew, and reported accuracy dropping from around 43% with a small inventory to roughly 13.6% once the inventory was scaled up — and rising back to ~43% when retrieval was used to shrink the per-turn working set. The Berkeley Function Calling Leaderboard shows similar degradation curves as tool counts climb. Anthropic's own guidance on writing tools for agents recommends "a few thoughtful tools" and explicit consolidation rather than exposing every underlying capability. OpenAI ships a tool_search feature in the Responses API precisely to defer large tool inventories until runtime, on the same reasoning: keep the per-turn working set small, and reach for the rest via retrieval.
These aren't hard cliffs. They're the point at which the model starts picking the wrong tool, hallucinating parameter names, or calling two tools that overlap.
The failure mode is worth naming precisely. It isn't that the agent refuses to work. It's that the agent works — but picks the fourth-best tool for the job, or invents a parameter that doesn't exist, or calls a read-only lookup when the user asked for a write. You only catch it in evals or in production. Neither is a great place to find out.
So the naive ceiling is real. It sits lower than most teams expect. And the answer isn't to fight it — it's to design around it.
The most common explanation for the tool-count ceiling is context window: each tool's JSON schema takes up tokens, and you run out. This is true in the trivial sense and misleading in the practical one.
A typical tool definition — name, description, parameter schema with types and descriptions — runs somewhere between 300 and 1,000 tokens depending on schema complexity, with heavier tools (rich descriptions, nested parameters, examples) exceeding that. Anthropic's own engineering write-ups on tool design describe cases where 58 tools consumed roughly 55K tokens — an average close to 950 tokens per tool. Forty tools at 700 tokens each is 28,000 tokens of pure tool overhead, before you've added a system prompt, conversation history, or retrieved context. On a 200K-token model that's still fine. On a 32K model it's most of your budget.
But context isn't the binding constraint at 40 tools. Selection accuracy is. Even with a million-token window, the model still has to read every tool description on every turn and decide which one to call. Longer descriptions don't scale attention — they dilute it. Tool descriptions that overlap semantically confuse selection regardless of how much room they have.
The practical rule: budget tokens for tools, but don't assume more room fixes the problem. If your agent is picking the wrong tool at 40 tools on a 200K model, giving it a 1M model will not help. The bottleneck is somewhere else.
Every tool in your inventory has three ongoing costs beyond its share of the context window. Teams underweight all three when they first size the inventory.
Selection cost. The model spends attention on every tool description, whether it uses that tool or not. Beyond about 25 tools, the marginal tool starts making the model worse at picking correctly among the tools it already had. This is the counterintuitive one — adding a tool can reduce accuracy for calls that don't involve the new tool at all.
Maintenance cost. Every tool wraps some underlying capability — an API endpoint, a database query, a computation. When that underlying capability changes, the tool has to change with it. If you have 40 hand-written tools wrapping 40 endpoints across three internal services and two vendors, you have 40 things that can silently drift. We've written about third-party API change management as the tax nobody puts on the roadmap — the same tax applies to internal APIs, just less visibly.
Auth and permissions cost. Every tool has to execute as some identity. If your agent runs as a service account, every tool it exposes is available to every user of that agent — which is a security review problem the day you ship to a second tenant. If your agent runs as the authenticated user, every tool has to honour that identity, which means every tool has to be wired to the underlying auth surface correctly. More tools, more wiring, more places to get it wrong.
The honest read: past about 25 well-scoped tools, the operational cost of each additional tool starts exceeding its marginal capability benefit. That's the real ceiling, not the token count.
So the naive answer is a per-turn working set of 10 to 25, and the ceiling isn't context window — it's selection accuracy and maintenance load. What if your agent genuinely needs to reach hundreds of capabilities? Enterprise SaaS platforms typically have thousands of internal operations an agent could theoretically call. You can't expose all of them directly. You also can't leave them unreachable. There are three architectural moves that work.
Instead of putting every tool in the system prompt, index tool descriptions in a vector store and retrieve the top k relevant tools for each user turn. The agent sees 10-15 tools at a time, chosen for the query, not the whole catalogue. Selection accuracy stays high because the working set stays small. This is exactly the pattern the RAG-MCP paper measured, and roughly what OpenAI's tool_search feature automates on their side.
This works well when queries are semantically distinct. It works badly when the agent needs to chain tools from different parts of the catalogue in one turn — retrieval based on the initial query won't surface tools needed for step three of a five-step plan. Teams that use retrieval well typically pair it with an explicit planning phase where the agent lists intended operations before retrieval runs.
Most agents don't need atomic CRUD tools for every entity. They need a smaller set of task-level operations — "reschedule this meeting and notify attendees", "pull the last quarter's invoices and flag anything unusual" — each of which internally calls several atomic operations. Composition compresses tool count without losing capability.
The trade-off: composed tools are opinionated. Every composition encodes a decision about how to reschedule a meeting, which fields to check for "unusual" invoices, what to do if a step fails. That opinion is what makes composed tools useful for agents (small selection surface, clear intent) and dangerous for engineers who assumed the primitive was flexible. Design the composition explicitly, don't let it accrete.
If your capability surface really is 200+ operations across five distinct domains, one agent with all 200 tools is the wrong architecture regardless of how you retrieve them. Split into specialised agents — a scheduling agent, a billing agent, a reporting agent — each with its own bounded tool inventory of 15-25 tools. An orchestrator agent (or a router) decides which specialised agent handles a request.
This is more infrastructure to run, but it maps to how humans actually organise work, and it makes evals tractable — you can measure each specialised agent's tool-call accuracy independently. The orchestrator vs tools layer split matters here: the routing logic is orchestrator work, the specialised agents' capabilities are tools-layer work, and confusing the two is where these architectures usually go wrong.
Most of this article's argument is: tool count is a design constraint, and past 25 or so tools the marginal tool starts costing you more than it earns. That framing lands cleanly for agents built on greenfield capabilities. It gets harder when you're an established SaaS platform whose product surface is genuinely thousands of operations, most of which are invisible to your existing APIs.
That's the situation Pontil is built for. We sit in the tools layer of the agent stack. We generate connectors from the codebases and APIs you already own, so the underlying tool inventory isn't hand-written and doesn't compound as maintenance debt. The runtime executes tool calls as the authenticated user, so per-user auth and audit come out of the design rather than being wired to each tool one at a time. That lets you scope agent inventories the right way — down to what a specific agent needs — without having to choose between coverage and maintainability at the portfolio level. If your team is running into this ceiling now, our resource on why agent projects stall walks through the pattern in more depth.
How many tools should an AI agent have? For most production agents on today's foundation models, the working set the agent sees in any single turn should sit between 10 and 25 well-scoped tools. Past that, selection accuracy starts to slip in ways evals will catch and users will feel. Below that, you're probably not covering enough of the task surface for the agent to be useful.
But the working set isn't the whole inventory. The inventory can be much larger — hundreds of underlying capabilities — as long as you're using retrieval, composition, or agent specialisation to keep the per-turn working set small. The number that matters is what the model sees on the turn, not what exists in the catalogue.
The design question worth asking, then, isn't "how many tools can we get away with?" It's: given the tasks this agent has to do, what's the smallest working set that covers them, and what's the maintenance and auth surface behind each tool in that set? Get those two answers right and the tool count sorts itself out. Get them wrong and no amount of context window will save you.
Stay up to date on the ever changing agentic landscape.
GET STARTED
Start with a 30-minute discovery call
contactus@pontil.com
