How to reduce agent latency: a practical guide to faster tool calls

Agent latency is the time between a user request and the agent's next useful action. Most teams measure the model's token stream and stop there. The real cost sits in the tool call: the round trip your agent makes to your APIs, third-party APIs, and internal services. Fix that layer and response time drops without changing the model.

By the end of this guide you'll know how to measure agent latency end-to-end, identify which tool calls dominate the budget, and apply six concrete techniques to cut response time. Prerequisites: an agent in staging or production, access to its trace data, and the ability to change tool definitions or the runtime that executes them. Time required: two to four hours to instrument, a day or two to roll out the fixes that apply.

Step 1 — Measure agent latency end-to-end, not per-model

Agent latency is a sum, not a single number. Break it into four segments: prompt assembly, model inference, tool call execution, and response synthesis. Most teams only see the model number because that's what the foundation model provider returns. That's the wrong number to optimise.

Instrument every trace so each turn logs:

• time to first token (TTFT) from the model
• total inference time per turn
• per-tool-call wall time, including network, auth, and downstream API time
• number of turns per task

If you already run agent evals or an agent observability platform, the data is usually there — you just need to aggregate by segment. If you don't, add OpenTelemetry spans around the tool executor. A weekend of instrumentation pays back the first time you find that one tool eats 60% of the budget.

Expected output: a per-turn breakdown that tells you which segment to attack first.

Step 2 — Rank tool calls by contribution to total latency

Once you have per-tool-call timings, rank them. The distribution is almost always long-tailed: two or three tool calls account for most of the wall time. Focus there.

Build a simple table from a week of production traces:
‍

The tool at the top of the p95 × frequency column is your target. Don't optimise search_customers first because it's easy — optimise get_order_history because it's costing you seconds per task.

Expected output: a ranked list, with the top two or three tools flagged as latency hotspots.

Step 3 — Cut tool call round trips with parallel execution

Most agents call tools sequentially: think, call, wait, think, call, wait. If two tool calls don't depend on each other, run them in parallel. The savings are additive across turns.

Check two things in your orchestrator:

1. Does the model provider support parallel tool calls? OpenAI and Anthropic both do — a single model response can contain multiple tool invocations (OpenAI returns them in a tool_calls array; Anthropic returns multiple tool_use blocks inside the assistant message's content array).
2. Does your tool executor actually run them concurrently? Some runtimes serialise despite the model returning them together. Check the trace.

If your executor serialises, switch to a concurrent runner (async task group, worker pool). Cap concurrency per user so one agent can't exhaust a downstream rate limit.

Expected output: a turn that used to take 2.4 seconds across three sequential calls now takes 900ms across three parallel ones. Model inference cost is unchanged; wall time drops by the slowest call.

Step 4 — Compress tool responses before they hit the context window

Large tool responses hurt twice: the network transfer and the next model turn (which now has more tokens to process). A 40KB JSON blob from a list endpoint is a common offender.

Apply three techniques, in order:

• Filter server-side. If the agent only needs id, status, and updated_at, return only those. Extend your tool definition to accept a fields parameter and enforce it at the runtime. This is easier if your tools sit on APIs you own — see REST API design best practices for the agent era.
• Paginate aggressively. Default page size 20, not 200. Let the agent request more if it needs more.
• Return summaries by default, details on request. A get_orders tool returns a compact list; a get_order_detail(id) tool returns the full record. The agent decides when to spend the tokens.

Expected output: median tool response drops from tens of KB to single-digit KB, and downstream inference time drops with it.

Step 5 — Cache what doesn't change every second

A lot of tool calls fetch data that's stable for minutes or hours — org settings, product catalogues, user profiles, permission lookups. Cache those at the runtime layer, not in the agent.

Use a short TTL cache (30 seconds to 5 minutes depending on the data) keyed on the tool name and arguments. Two rules:

• Never cache per-user data under a shared key. If tool calls execute as the authenticated user, the cache key must include the user identity. Otherwise you'll leak data across sessions.
• Invalidate on writes. If the agent calls update_customer, evict the get_customer entry for that ID immediately. Stale reads after writes are worse than no cache.

An HTTP cache with ETag and If-None-Match gets you most of this for free if the upstream API supports it. Otherwise a Redis layer at the runtime does the job.

Expected output: cache hit rate above 40% on read-heavy tools, with the corresponding drop in p50 latency for those calls.

Step 6 — Set timeouts and circuit breakers so slow calls don't dominate p95

Your p50 is fine. Your p95 is where the user experience actually lives. A single slow downstream API — a third-party CRM having a bad afternoon — can push the whole turn past the timeout the user is willing to wait.

Set per-tool timeouts based on the p95 you measured in Step 1, plus headroom. If a tool's p95 is 800ms, a 2-second timeout is reasonable. Beyond that, cancel and return an error the agent can handle.

Add a circuit breaker around any tool that calls a third-party API. When the breaker is open, fail fast instead of waiting for the timeout. The agent gets a clear signal to try a different approach or tell the user, rather than burning three seconds waiting.

Expected output: p95 tool call time bounded by your timeout, and tail latency stops dragging the mean.

Step 7 — Reduce turns by giving the agent the right tools, not more tools

Every extra turn costs a full model inference. If the agent needs three turns to do what one composite tool could do in one, you're paying inference latency twice for no reason.

Look at your traces for common multi-call sequences:

• find_customer → get_orders(customer_id) → get_order_detail(order_id) shows up in 30% of tasks? Consider a get_customer_recent_orders_detailed tool that does all three server-side.
• The agent frequently pages through results looking for one specific match? Add a filter parameter so the search endpoint does it in one call.

Don't overdo this — every tool you add costs selection accuracy. But collapsing a proven three-call sequence into one composite tool is almost always a win: fewer turns, less context, less latency.

Expected output: median turns per task drops by one, which typically halves the perceived response time.

Common pitfalls

• Optimising the model when the tools are the problem. Switching to a faster model saves 200ms of inference and hides a 2-second tool call. Fix the tool layer first, then look at the model.
• Adding a cache without user-scoped keys. This will pass tests and fail security review. Every cache key that touches user-scoped data must include the authenticated user identity.
• Removing timeouts because they cause errors. Errors are the point. A tool call that hangs for 30 seconds is a worse user experience than one that fails at 2 seconds and lets the agent recover.
• Parallelising dependent calls. If call B needs the result of call A, running them in parallel doesn't help — and race conditions in your executor will make debugging miserable. Only parallelise calls the model actually returned in the same turn.
• Measuring in staging only. Staging latency and production latency rarely match. Real auth, real rate limits, and real data volumes reveal problems synthetic traffic doesn't.