Semantic tool retrieval for AI agents: when RAG for tools beats loading them all

When an agent has 12 tools, you load all of them. When it has 400, you can't. Somewhere between those two numbers, the way you present tools to the model has to change — and most teams hit that wall without realising it's what's breaking their agent.

This piece is for engineers and platform leads deciding how their agent finds the right tool at the right time. The two main options are static loading (put every tool in the prompt, let the model pick) and semantic tool retrieval (index the tools, retrieve the top-k relevant ones per turn, load only those). One is simple and predictable. One scales but adds a whole new failure surface.

Here's the short version: static loading wins until roughly 30–50 tools or when tool selection accuracy starts dropping. After that, semantic retrieval is the honest answer — but only if you're willing to own the retrieval quality problem it creates.

How static tool loading works

Static tool loading is the default pattern in every foundation model provider's SDK. You define your tools — a JSON schema per tool with a name, description, and parameter definitions — and pass the full list to the model on every call. The model sees every tool it could possibly use and picks one (or several) per turn.

This is what Anthropic, OpenAI, and every agent framework ship out of the box. It's easy to reason about: the tool set is deterministic, the model's context is predictable, and debugging is straightforward because you can see exactly what the model was offered. Every token used by a tool definition is a token not available for reasoning or conversation history, but at small tool counts nobody notices.

The pattern breaks in two ways as tools grow. First, context window pressure: empirical measurements of real-world MCP tool suites put typical tool definitions in the 200–500 token range, with complex ones reaching 1,000+. At 400 tools that's roughly 80K–200K tokens of overhead before the user even asks a question. Second, and worse, tool selection accuracy drops. More options means more chances the model picks a plausible-but-wrong tool. Research from Anthropic and others has consistently shown selection accuracy degrading well before the context window fills — the ceiling is behavioural, not mechanical.

How semantic tool retrieval works

Semantic tool retrieval — sometimes called tool RAG or dynamic tool loading — treats the tool catalogue like a search index. At startup, you embed each tool's name, description, and (optionally) example use cases into a vector store. At runtime, you take the user's message (or the current agent state), embed it, and retrieve the top-k tools by semantic similarity. Only those k tools go into the model's context that turn.

Mechanically, this is RAG applied to tool definitions instead of documents. The building blocks are familiar: an embedding model, a vector index (pgvector, Pinecone, Weaviate, or similar), a retrieval query per turn, and some logic to merge the retrieved tools with any tools that must always be available (auth helpers, cancellation, escalation).

The pattern shows up in production frameworks. LangChain supports it via standard vector-store retrievers over embedded tool descriptions, wired into a tool-calling agent. LlamaIndex offers a first-class ObjectIndex for tool retrieval. Anthropic's Claude and OpenAI's Responses API both accept dynamically constructed tool lists per call, and both providers now ship first-party implementations of the pattern too — Anthropic's Tool Search Tool and OpenAI's hosted tool search in the Responses API. And there are hybrid variants worth knowing: hierarchical retrieval (pick a tool category first, then a specific tool), keyword pre-filter plus semantic re-rank, or LLM-as-router where a cheaper model picks the tool namespace before the main model runs.

What you gain: the model sees a small, relevant set of tools every turn, freeing context and improving selection accuracy on the tools it does see. What you pay for: a whole new component — the retriever — that can fail, drift, or return the wrong candidates. When the retriever misses, the model literally cannot see the correct tool. That failure looks like a hallucination but is actually a retrieval bug, and it's harder to debug than static loading.

Static loading vs semantic retrieval: the trade-offs that matter

The row that catches teams out is the last-but-one: when retrieval hides the right tool, the model has no way to know a better option existed. Static loading fails loudly (the model picks a wrong tool from tools you can see). Retrieval fails silently.

When to choose static tool loading

Stick with static loading when:

• Your tool count is bounded and small. Under about 30 tools, static loading is almost always correct. Under 15, it's not even a question.
• The tools are heterogeneous and always relevant. If every tool could plausibly apply to any user turn — a coding agent with read_file, write_file, run_command, grep — retrieval will just re-select the same tools every time. You're adding latency for nothing.
• Debuggability matters more than scale. For internal agents, prototypes, or workflows where a wrong answer is expensive to review, seeing exactly what the model was offered is worth the token cost.
• You haven't measured selection accuracy yet. Do not reach for semantic retrieval as a preemptive fix. Ship static, measure where selection accuracy drops, then decide. Frameworks like BFCL and τ-bench give you a starting point for this — we covered how the major benchmarks measure tool selection elsewhere.

A useful heuristic: if you can list every tool your agent has from memory, you don't need retrieval yet.

When to choose semantic tool retrieval

Reach for retrieval when:

• The catalogue is 100+ tools, or growing unpredictably. Platform teams exposing hundreds of internal capabilities to an agent, or SaaS companies with multiple products each exposing dozens of tools, hit the static-loading ceiling early.
• Selection accuracy has measurably dropped. You've run evals, you've watched the model pick create_invoice when it should have picked create_estimate, and adding another tool description isn't fixing it. The signal is a decline in trajectory-level accuracy that correlates with catalogue growth.
• Tools cluster into semantically distinct groups. Retrieval works best when tools are separable: CRM tools vs analytics tools vs billing tools. When the tools are all subtle variations of each other (get_user, get_user_by_email, get_user_by_id), retrieval struggles for the same reason humans do — the descriptions blur.
• You're building for a multi-product platform. This is the case we see most often at Pontil. A SaaS company with five products, each with meaningful surface area, will have hundreds of tools before the first agent ships. Static loading was never going to work.

One caveat: if you're going to use retrieval, budget for retrieval evals from day one. A retriever that returns the wrong top-k is worse than static loading because the failure is invisible to the model. Measure retrieval hit rate against a curated eval set, not just end-to-end agent accuracy.

How Pontil fits

Semantic tool retrieval is a solution to a problem most SaaS platforms only get to because they've already generated the tools. The hard part isn't retrieval — vector search is a solved problem. The hard part is having a large, well-described, up-to-date tool catalogue in the first place. Most agent projects stall long before retrieval matters, because their APIs expose 2% of what their product can do and nobody's generated the other 98% yet.

Pontil is a Tools-as-a-Service platform that generates and maintains that catalogue directly from your codebase. Once the tools exist, layering semantic retrieval on top — whether ours or your own — is straightforward. Tool descriptions are structured, versioned, and stay current as your products change, which is the actual precondition for retrieval quality. If your agent project is stuck deciding between static and dynamic loading, the honest question is usually one step earlier: do you have enough tools, described well enough, for retrieval to even be worth building?

What we'd choose

Default to static loading. Ship it. Measure selection accuracy on real user traffic. Only switch to semantic retrieval when the data tells you to — usually somewhere between 50 and 100 tools, sometimes earlier if the tools are semantically close.

When you do switch, treat the retriever as a first-class component with its own evals, its own monitoring, and its own on-call rotation. The teams that get burned by tool retrieval are the ones that treat it as a config change rather than a new distributed system. It isn't. It's another layer of the tools stack, and like every layer, it has its own failure modes worth respecting.

And whichever pattern you pick: describe your tools like the model has to read them, because it does. Good tool descriptions do more for selection accuracy than any retrieval architecture will.