Agent infrastructure
Agents in production
RAG vs tool calling for AI agents: how each pattern works, where each one fits, and why most production agents need tool calling before retrieval.
Retrieval-augmented generation and tool calling get lumped together in agent architecture discussions. They shouldn't be. They solve different problems, they fail in different ways, and picking the wrong one is one of the more expensive mistakes an agent project can make early on.
This comparison is for engineering leads and heads of AI deciding how their agent reaches information and takes action. The short version: RAG is how an agent reads. Tool calling is how an agent does. Most production agents need both, but the order you build them in — and which one you lean on for a given capability — decides whether the project ships or stalls.
Retrieval-augmented generation is a pattern for grounding a model's output in a specific corpus. At query time, the system takes the user's input, converts it to an embedding, searches a vector store for semantically similar chunks, and passes those chunks into the model's context window alongside the original prompt. The model then generates a response conditioned on that retrieved content.
The pipeline has moving parts. You need an ingestion step that chunks source documents and embeds them. You need a vector database — Pinecone, Weaviate, pgvector, or similar. You need a retrieval step that handles the query embedding, similarity search, and often re-ranking. And you need prompt assembly logic that decides how retrieved chunks are formatted and truncated before hitting the model.
RAG shines when the answer already exists somewhere in text and the agent needs to find and summarise it. Documentation lookup, policy questions, historical ticket search, knowledge-base Q&A — all classic RAG territory. It's a read pattern. Nothing changes in the underlying system as a result of the retrieval.
Tool calling is the mechanism by which a model requests that the runtime execute a defined function. The model doesn't run the function itself — it emits a structured call (name plus arguments), the runtime executes it against the real system, and the result is fed back into the model's context. Modern foundation models from Anthropic, OpenAI, and Google all support this natively through their APIs.
Each tool is a contract: a name, a schema for arguments, and an implementation that does something. That something can be anything a program can do. Query a database. Send an email. Update a CRM record. Cancel a subscription. Kick off a workflow. Tool calling is how an agent stops being a chat interface and starts being a system that affects the world.
The reliability question with tool calling is different from RAG. RAG can be wrong (retrieve the wrong chunk, hallucinate around it). Tool calling can be destructive — wrong arguments to a real API don't just produce a bad answer, they produce a bad outcome. Which is why the runtime around tool calls matters as much as the tool definitions themselves. If you want to go deeper on where those two failure modes diverge, our post on tool calling vs function calling covers the production reality.
The row that matters most is the one about state change. Read-only patterns and mutating patterns should not share failure modes, retry logic, or observability. Conflating them is how agent projects end up with tool calls being retried three times because the runtime treated them like a search query.
Reach for RAG when the capability is fundamentally about finding and synthesising information from a body of text.
RAG is also the right choice when you don't control the underlying system and can't get proper API access. Scraping documentation into a vector store isn't ideal, but it's better than fabricating.
What RAG cannot do: take action. If the user asks "what's my current subscription status?" and you've indexed the billing docs, RAG will tell them what subscriptions exist as a concept. It won't tell them theirs. For that, you need tool calling.
Reach for tool calling when the capability is about interacting with a live system — either to read current state or to change it.
The hard part of tool calling in production isn't the model side — foundation models handle the schema and the call emission well now. The hard part is the runtime: authenticating as the right user, honouring their permissions, retrying safely, handling rate limits, catching upstream drift when the underlying API changes. Our guides on API idempotency for AI agents and AI agent guardrails both dig into that runtime layer, which is where most production incidents actually happen.
The interesting middle ground is agentic RAG. Instead of a fixed retrieve-then-generate pipeline, the agent decides when to retrieve, what to retrieve, and whether to retrieve at all — usually by exposing retrieval itself as a tool.
In this pattern, search_docs is just another tool in the agent's toolbox, alongside create_ticket or update_contact. The model decides based on the user's request which combination of tools to invoke and in what order. That's more flexible than a hard-coded RAG pipeline, and it lets the same agent handle read-heavy questions and mutating actions without switching architectures.
Agentic RAG doesn't remove the read/write distinction — it just moves the decision into the model. You still need to design the retrieval tool differently from the mutating tools. Different auth model, different retry semantics, different observability. But the interface the agent sees is uniform, and that uniformity is what makes complex flows tractable.
This is also where the "RAG vs agents" framing gets clearer. Agents use retrieval. Retrieval isn't a competing architecture; it's one capability among many.
The reason the read/write distinction matters is that most established SaaS companies building agents run into the write side and stall. Their APIs cover a small fraction of what the product can do, so their agents can search documentation about a feature but can't invoke it. RAG covers the read gap by pointing at text. It doesn't cover the action gap.
Pontil is a Tools-as-a-Service platform that generates and runs the tools an agent needs to reach the parts of a product that its existing APIs don't expose. Tool calling only works if there are tools to call, and for most SaaS products those tools don't exist yet. We generate them from the codebase, run them through a runtime that executes as the authenticated user, and maintain them as the product changes. If your agent project is currently sitting at "the retrieval works but the agent can't actually do anything", that's the gap we close. A product walkthrough is the fastest way to see what that looks like on a real codebase.
Start with the read/write question, not the technology question. For every capability your agent needs, ask: does answering this require reading a corpus, or does it require touching a live system?
If it's read-only over stable text, RAG. If it's user-specific, live, or mutating, tool calling. If it's both — and most production agents are — build the tool calling layer first, expose retrieval as one of the tools, and let the model orchestrate. That order matters. Retrieval on its own produces a chatbot that knows things. Tool calling on its own produces an agent that can do things. The combination, in that order, produces an agent that can actually be trusted with real work.
The teams that get stuck are usually the ones that built impressive RAG demos and then discovered that their agent can't reach the vast majority of their product that isn't documented as text. Answer the read/write question early and the rest of the architecture follows.
Stay up to date on the ever changing agentic landscape.
GET STARTED
Start with a 30-minute discovery call
contactus@pontil.com
