Agents in production
Agent infrastructure
Agent tool selection breaks in ways that look like model problems but are design problems. Failure modes, retrieval and routing trade-offs, and what to fix first.
How does an agent decide which tool to call? The mechanism is simple on paper: the model sees a list of tool definitions, matches the user's request against them, and emits a call. In production, that mechanism breaks in ways that look nothing like a model quality problem. Agents call the wrong tool. They call the right tool with the wrong arguments. They call three tools when one would do. They ignore a tool that fits perfectly because a similarly-named one shows up first.
Our view: agent tool selection is a design problem masquerading as a model problem. Model choice matters at the margins. Tool surface design — how many tools, how they're named, how they're described, how they're presented — matters an order of magnitude more. This deep-dive covers what selection actually is, why it degrades, how retrieval and routing help, where they hurt, and what a production-grade selection layer looks like.
When a foundation model provider like Anthropic or OpenAI exposes tool calling, the model isn't reasoning about your tools in any deep sense. It's doing pattern matching against the tool schemas in its context — names, descriptions, parameter types, and (sometimes) examples. The tool schemas are text. The user's request is text. The model picks the tool whose text pattern best fits.
That framing matters because it explains most of what goes wrong. If two tools have overlapping descriptions, the model will flip between them. If a description is vague, the model will guess. If the parameter names don't match the request's vocabulary, the model will hallucinate arguments. Selection quality is a function of the schemas you write, not a fixed capability of the model.
There are three sub-problems inside "tool selection" that get conflated:
Most teams debug selection by staring at the second problem. The wins usually live in the first and third.
The too many tools problem is well documented at this point. Selection accuracy drops as tool count rises, and the drop starts earlier than most teams expect — well before the context window fills. The reason isn't context length. It's discrimination difficulty.
Every tool you add is another candidate the model has to distinguish from every other candidate. When you have five tools that all touch "customer records," the model isn't picking between clearly different capabilities — it's picking between siblings. A get_customer, find_customer, lookup_customer_by_email, search_customers, and list_customers surface is a selection failure waiting to happen. The model will pick one, be wrong often enough that no prompt tweak will save you, and there's nothing in the prompt that will fix it because the prompt isn't the problem.
Each failure mode has a different fix. Argument hallucination isn't solved by better tool descriptions — it's solved by tighter schemas. Under-selection isn't solved by fewer tools — it's solved by descriptions that name the trigger conditions explicitly. Treating all selection failures as the same problem is why teams throw prompt engineering at issues that need schema work.
When teams hit the tool ceiling, the standard move is agent tool retrieval: embed all your tool descriptions, retrieve the top-k relevant ones per turn, and only put those in the model's context. It's the same pattern as RAG, applied to tools instead of documents.
Retrieval works. It genuinely raises the ceiling on how many tools you can expose. But it introduces a new failure mode: if the right tool isn't in the top-k, the model literally cannot call it. Now you have two selection problems stacked — the retriever picking the candidate set, and the model picking within it.
Retrievers are also worse at discrimination than models are. A retriever operating on tool descriptions will happily return five sibling tools as the top-5, giving the model the exact ambiguity retrieval was supposed to solve. Retrieval helps most when your tool catalogue is diverse — many capabilities, few overlaps. It helps least when your catalogue is dense — many variants of similar operations.
Where retrieval genuinely earns its keep is when tool count crosses a few hundred and the catalogue spans multiple domains. Below that, better tool design usually beats retrieval. Above it, retrieval is unavoidable, but it needs to be paired with disciplined tool authoring — otherwise you're just moving the selection problem one layer up.
A related move is explicit tool routing LLM architecture: use a cheaper, faster model as a router that picks a category or a small handful of tools, then hand off to the main agent with only those tools in context. It's cousin to retrieval, but with a model doing the filtering instead of embeddings.
Routing has two advantages over retrieval. First, models are better than embeddings at handling multi-step intent — "I need to update the customer's plan and email them a receipt" is two capabilities, and a router can identify both. Second, routers can be fine-tuned on your specific catalogue in a way retrievers can't easily match.
The trade-offs are real though. You've added a call to the critical path, which adds latency and cost. You've introduced a second model that can be wrong, and its errors compound with the main agent's. And you now have two things to evaluate — router accuracy and agent accuracy — instead of one.
Routing is worth it when: tool count is large, your catalogue has clear categorical structure, and your evals show the main agent's selection accuracy dropping below acceptable thresholds. It's not worth it as a default. Most agents don't need it.
Before you reach for retrieval or routing, exhaust tool design. The interventions here have the highest leverage and the lowest complexity.
Name tools by capability, not by resource. cancel_subscription beats update_subscription_status. The model is matching against the request's verbs; make your tool names verbs the request will use.
Write descriptions that name the trigger. The first sentence of every tool description should answer "when should the model call this?" — not "what does this do?". Trigger-first descriptions cut wrong-tool selection dramatically.
Collapse siblings. If you have get_customer_by_id, get_customer_by_email, and get_customer_by_phone, that's one tool with a polymorphic parameter, not three tools. Consolidation reduces the model's discrimination burden without reducing capability.
Constrain parameters with enums and patterns. Every free-text string is an invitation to hallucinate. If a parameter has three valid values, encode all three as an enum. If it's a date, specify the format. The OpenAPI schema you generate tools from should already do this — if it doesn't, the API's contract is the first thing to tighten.
Give examples in descriptions. A one-line example of a valid call inside the tool description is worth more than three paragraphs of prose. Models are excellent at pattern-matching examples.
Remove tools nobody uses. Every tool in context is a potential distractor. If a tool hasn't been called in production in the last 30 days, ask whether it belongs in the default surface at all.
These are unglamorous fixes. They're also the ones that survive contact with a real agent workload. Retrieval and routing help when you've done this work and still have too many tools. They don't substitute for doing this work.
Selection quality lives or dies on the tool schemas the model sees. That's not something you can fix by wiring up a better orchestrator — it's a property of the tools layer itself. Pontil sits in that layer. We generate tools from the systems you already own, and the schemas we produce are designed for selection: trigger-first descriptions, enum-constrained parameters, consolidated siblings, examples baked in.
Because tools are generated from your codebase rather than reverse-engineered or hand-written, the schemas stay accurate as your product evolves. When a parameter's valid values change, the tool's enum changes with it. When a capability is deprecated, the tool leaves the catalogue. That's the maintenance discipline selection depends on — and the piece that's hardest to hold together when tools are bespoke. If you want the longer argument for why this belongs in a dedicated layer rather than the agent framework, our product page is the place to start.
The selection problem gets more interesting, not less, as agents take on longer trajectories. A single-turn selection error is annoying. A selection error on turn three of a fifteen-turn trajectory is catastrophic — every downstream call inherits the error. The teams building serious agents are already treating tool selection as a first-class evaluation surface, grading tool calls as rigorously as they grade final outputs.
The design principles won't change much. Fewer, sharper tools beat many fuzzy ones. Trigger-first descriptions beat capability-first ones. Constrained schemas beat free text. Retrieval and routing help at scale but don't substitute for the underlying work. What will change is the tooling around selection — better evals, better observability into which tools the model considered before picking, better ways to A/B test schema changes without retraining anything.
The teams that treat selection as a schema problem will get further than the teams that treat it as a prompt problem. The model isn't going to get smart enough to fix bad tool design. The tools have to get well-designed enough for the model to succeed.
Stay up to date on the ever changing agentic landscape.
GET STARTED
Start with a 30-minute discovery call
contactus@pontil.com
