Orchestrator vs tools layer: where agent work actually happens

If you're building an agent system inside an established SaaS company, you've probably had this argument: where does the orchestrator end and the tools layer begin? It sounds like a semantics question. It isn't. The line you draw decides which team owns what, which vendors you buy, and which problems show up in production six months later.

This piece is for Heads of AI and platform engineers picking apart the agent stack for the first serious build. The short version: an orchestrator decides what should happen next. A tools layer decides how that thing actually gets done against real systems. Confusing the two is the most common architectural mistake we see, and it's expensive to unwind.

Here's when each wins. The orchestrator wins on reasoning, planning, and routing. The tools layer wins on capability access, auth, and reliability against your actual product surface. You need both, and they should not be the same component.

How an agent orchestrator works

An orchestrator is the control loop. It takes a goal — from a user, a workflow, another agent — and decides the sequence of steps to reach it. In practice that means a model call, a decision about which tool to invoke, a check on the result, and a decision about what to do next. Frameworks like LangChain/LangGraph, CrewAI, and OpenAI's Agents SDK all sit at this layer, with different opinions about how much structure to impose on the loop.

A good orchestrator handles the things that are genuinely hard about coordinating models: managing context windows, routing between models of different cost and capability, retrying on model errors, decomposing a vague goal into concrete sub-tasks, and holding state across a multi-step run. It also exposes the surface for human-in-the-loop checkpoints, observability of agent reasoning, and policy enforcement on what the agent is allowed to attempt.

What an orchestrator does not do — at least, not well — is the actual work against your product. It calls tools. The tool is the thing that makes a real change in a real system. The orchestrator's job ends at the point of saying "invoke this tool with these arguments." Everything that happens between that decision and a successful, audited, permission-checked side effect is the tools layer's problem.

How an AI orchestration layer's tools layer works

The tools layer is where agent intent meets your actual systems. Each tool is a callable capability — create an invoice, reassign a ticket, update a customer record — with a defined input schema, an output schema, and a runtime that executes the call against your APIs, your auth, your rate limits, and your observability stack.

A tools layer has to do four things reliably. First, it has to expose the capabilities your product actually has, not just the slice that happens to be on a public API. Second, it has to execute calls under the authenticated user's identity, so permissions, data visibility, and audit trails are honest — never under a shared service account. Third, it has to handle the operational mess: retries, idempotency, partial failures, rate limits, deterministic invocation. Fourth, it has to stay current as the underlying product changes, or every release silently breaks half the tools.

This is the layer where established SaaS companies get stuck. The orchestrator is mostly a solved buy-or-build choice. The tools layer is where the real structural problem lives — your APIs were built for the UI, the UI grew faster than the API kept up, and now your agents can't reach what your customers can already do by clicking.

Orchestrator vs tools layer: a comparison

The rows that matter most are the last three. The orchestrator is portable across domains because reasoning loops are roughly the same whether you're triaging tickets or reconciling invoices. The tools layer is the opposite — it's the most product-specific code in your entire agent stack, and it's the layer that decides whether the agent can actually do the job.

When to choose orchestrator-first thinking

Lead with the orchestrator when the hard problem really is reasoning. A few situations where that's true:

• You're routing between models. If half your engineering effort is about choosing the right model for a sub-task, balancing cost against capability, and managing fallback chains, the orchestrator is your centre of gravity.
• You're composing many agents. Multi-agent systems with handoffs, debate patterns, or supervisor-worker structures live or die on orchestration quality.
• Your tool set is small and stable. If you only need a handful of tools and they're already exposed cleanly via existing APIs, the tools layer is mostly a thin wrapper. The orchestrator is where the interesting decisions are.
• You're doing research or content workflows. Tasks dominated by retrieval, summarisation, and generation — rather than side effects in business systems — are reasoning-heavy and tooling-light.

In all of these, picking a good orchestrator framework, writing reasonable prompts, and wiring up a small number of tools is enough. You don't need a tools-layer strategy because the tools layer isn't where your problems live.

When to choose tools-layer-first thinking

Lead with the tools layer when capability access is the bottleneck. This is the situation most established B2B SaaS companies are actually in, even if they started by shopping for orchestrators.

• Your agents need to do what your UI does. If the goal is "replace this manual workflow our customers do in our product," you're in tools-layer territory. The orchestrator is the easy part. Reaching the large majority of your product surface your APIs don't expose is the work.
• You have multiple products in a portfolio. Bespoke connectors per product per use case don't compound. A second product doubles the maintenance load. The hidden cost of bespoke connectors compounds quickly.
• You need real audit trails. If a regulator or a security review will ask "who did this and were they allowed to," service-account tool execution is a non-starter. You need user-scoped runtime, which is a tools-layer concern.
• Your APIs are older than your UI. This is the structural agent project stall pattern. The orchestrator works fine in the proof-of-concept. The build hits the wall when the agent needs to do something the API never exposed. No orchestrator fixes that.

If two or more of these are true, your agent project will succeed or fail on the tools layer. Picking a better orchestrator won't help. Investing months in orchestrator tuning before solving access is the most common way teams burn a year and ship nothing.

What we'd choose

For most established SaaS companies building agents on their own products, the right shape is: buy a thin, opinionated orchestrator, and put serious investment into the tools layer.

The orchestrator choice matters less than people think. Most of the major frameworks are converging on similar primitives — tool calling, structured outputs, context management, observability. Pick one your team already understands, keep your usage of it shallow enough to swap, and move on. The orchestrator is not where you'll differentiate or where your project will fail.

The tools layer is the opposite. It's where the structural problem lives, where the maintenance cost compounds, and where the difference between a demo and a production system gets decided. Treat it as a first-class architectural concern. That means deciding deliberately between three real options — rewrite the API surface, build bespoke connectors per product, or use a generation platform — with the honest trade-offs of each on the table.

The one trap to avoid: don't let the orchestrator vendor convince you the tools layer is a thin wrapper they handle for you. Every major framework exposes a Tool (or equivalent) abstraction. None of them solve capability access against your real product, user-scoped execution, or maintenance as your product changes. Those are your problems, and they live in a layer the framework doesn't own.

How Pontil fits

The argument above — that the tools layer is where established SaaS companies' agent projects actually succeed or fail — is the problem Pontil is built for. We're a Tools-as-a-Service platform: we generate connectors from the codebases you already own, run them under the authenticated user's identity, and keep them current as your product changes. That sits cleanly behind whichever orchestrator you've picked, and it doesn't ask you to rewrite your API layer first.

If you're trying to work out whether your access problem is big enough to need a dedicated tools-layer strategy, a demo is the fastest way to see what generation against your existing surface looks like in practice — and to compare it honestly against the bespoke-connector path most teams default to.