Parallel tool calls: how to run agent tools concurrently without breaking production

You've got an agent that makes three tool calls in sequence. Each takes 800ms. Your users wait 2.4 seconds for a response that could have taken 800ms. The tools are independent — there's no reason they run one after the other.

This guide walks through implementing parallel tool calls in production. By the end you'll have a working parallel execution loop, a way to decide which calls are safe to parallelise, and the guardrails that stop concurrency from silently corrupting your data. Prerequisites: familiarity with tool calling on OpenAI or Anthropic's APIs, a working agent loop, and a language runtime with async support. Time required: roughly two hours to implement, longer to test properly.

Step 1 — Confirm your model and API support parallel tool calls

Parallel function calling is a model capability, not something you can bolt on client-side. The model has to emit multiple tool calls in a single response. If it emits them one per turn, no amount of client-side concurrency helps.

Check your model:
‍

On OpenAI, set parallel_tool_calls: true in your request. It defaults to true on supported models, but be explicit — some fine-tuned models flip the default. If you're on an o-series reasoning model, skip this parameter entirely; passing it returns a 400 (Unsupported parameter: parallel_tool_calls is not supported with this model). On Anthropic, parallel tool use is on by default (controlled via disable_parallel_tool_use); you can nudge it further with a system prompt that tells the model to batch independent calls. Note that some Claude versions (e.g., 3.7 Sonnet) are more conservative about emitting parallel calls even with the default enabled; Anthropic recommends a batch-tool wrapper as a reliable workaround.

Run a smoke test. Ask the agent something that clearly needs two independent lookups ("get the weather in Paris and Tokyo"). If you get one tool_calls array with two entries, you're set. If you get sequential turns, either your model doesn't support it or something in your request is disabling it.

Step 2 — Decide which tools are safe to run concurrently

Not every tool is safe to parallelise. The model will happily emit ten concurrent calls to a tool that must run serially, and your production data will pay for it.

Classify each tool in your registry along two axes:

• Idempotent vs non-idempotent. A read is idempotent. A write that creates a new resource each call is not. If the model calls create_invoice twice in parallel, do you want two invoices?
• Independent vs order-dependent. get_user(123) and get_order(456) are independent. list_orders() followed by refund_order(order_id) is not — the second call depends on the output of the first.

The model can't reliably reason about this. You have to encode it. Add a concurrency field to each tool definition:

{
 "name": "get_customer",
 "description": "Fetch customer record by ID.",
 "concurrency": "parallel-safe",
 "input_schema": { "...": "..." }
}



Use three values:

• parallel-safe — read-only, idempotent, no shared state. Run these concurrently without a second thought.
• serial — must run one at a time within a turn. Writes to shared resources, order-dependent workflows.
• exclusive — must run alone. Nothing else in the batch runs while this executes. Reserve this for destructive or state-changing operations that can't tolerate concurrent siblings.

This classification is the same discipline you'd apply for API idempotency for AI agents — parallel execution just makes the cost of getting it wrong more visible.

Step 3 — Group the tool calls the model returned

When the model returns a batch of tool calls, split them into execution groups based on the concurrency field.

The rule is simple: any run of adjacent parallel-safe calls runs together as one group. A serial call runs alone in its own group. An exclusive call runs alone and flushes anything queued before it.

Given a batch like [get_user, get_order, refund_order, get_invoice] where the first two and last are parallel-safe and refund_order is serial, you get three groups:

Group 1 (parallel): get_user, get_order
Group 2 (serial):   refund_order
Group 3 (parallel): get_invoice



Groups execute in order. Within a group, calls execute concurrently. This preserves the model's intended sequence for anything order-sensitive while still parallelising the safe stuff.

Don't reorder within groups without a reason. The model chose an order; respect it unless you know better.

Step 4 — Execute a group concurrently with a bounded concurrency limit

Don't fire ten HTTP requests at once because the model emitted ten calls. Cap concurrency per group.

In Python with asyncio:

import asyncio

async def run_group(calls, executor, max_concurrent=5):
   semaphore = asyncio.Semaphore(max_concurrent)

   async def run_one(call):
       async with semaphore:
           return await executor.invoke(
               call.name,
               call.arguments,
               call_id=call.id,
           )

   return await asyncio.gather(
       *(run_one(c) for c in calls),
       return_exceptions=True,
   )



In TypeScript, use Promise.allSettled with a semaphore library like p-limit. Pick a limit based on your downstream capacity, not the model's ambition. Five is a reasonable default. If your tools hit the same upstream API, that API's rate limit is your real ceiling.

Use return_exceptions=True (or allSettled) — never gather with default settings. One failed call should not cancel the others. The model needs every result to reason about what to do next.

Step 5 — Return every result to the model, including failures

This is the step teams get wrong. Concurrent execution means some calls will succeed and some will fail. The model can only recover from failures it can see.

For each call in the batch, append a tool result message with the matching tool_call_id. Include the result on success, a structured error on failure:

{
 "role": "tool",
 "tool_call_id": "call_abc123",
 "content": {
   "status": "error",
   "error": "rate_limited",
   "message": "Upstream returned 429. Retry-After: 30s.",
   "retryable": true
 }
}



Every tool_call_id in the assistant message must have a matching tool result message, and those tool result messages must immediately follow the assistant message with no intervening user or system messages. Matching is by tool_call_id, not by position — but returning them in the original order of the tool_calls array is still good practice for readability and to avoid provider-specific quirks.

Structured errors matter more here than in serial execution. In a serial loop, a failure stops the world and you handle it. In a parallel batch, the model sees a mixed result set and has to decide whether to retry, work around the failure, or bail. Give it enough structure to make that call. Our guide on AI agent error handling covers the error taxonomy in more detail.

Step 6 — Add timeouts, tracing, and cancellation

One slow tool call blocks the model's next turn. If nine calls finish in 200ms and one hangs for 30 seconds, your user waits 30 seconds.

Set a per-call timeout aggressive enough that the slowest acceptable call still fits comfortably inside it. Two to five seconds is a reasonable range for most SaaS API calls. On timeout, return a structured error to the model rather than raising:

try:
   result = await asyncio.wait_for(
       executor.invoke(call.name, call.arguments),
       timeout=3.0,
   )
except asyncio.TimeoutError:
   result = {
       "status": "error",
       "error": "timeout",
       "message": "Tool did not respond within 3s.",
       "retryable": True,
   }



Trace every call with a shared correlation ID and a per-call span. When something goes wrong in production — and it will — you need to reconstruct which calls ran, in what order, with what latency, and which upstreams they hit. Without per-call spans, a parallel batch is a black box.

Also implement cancellation. If the user closes the session or the parent request is cancelled, the remaining in-flight tool calls should be cancelled too. Otherwise you're paying for work nobody's waiting on.

Step 7 — Test the concurrency contract, not just the happy path

Unit tests where every tool returns instantly won't catch the failures that matter. Test the cases that only appear under concurrency:

• Partial failure. Make three tools succeed and two fail. Confirm the model receives all five results and continues.
• Slow neighbour. Make one tool sleep for 10 seconds while others finish in 100ms. Confirm the timeout fires and the batch completes.
• Rate limit cascade. Make all five calls hit the same rate-limited endpoint. Confirm your semaphore holds and you don't emit a burst that gets throttled.
• Order-dependent misuse. Emit two parallel-safe calls that actually share state (accidentally misclassified). This should surface in your test data as a race — that's the signal to fix the classification.
• Duplicate detection. Emit the same tool call twice in parallel. Confirm your idempotency layer catches it or your tool tolerates it.

Run these as part of your agent eval suite, not just unit tests. Parallel execution changes trajectories, and trajectory regressions are what break production.

Common pitfalls

Assuming the model will batch optimally. It won't, always. If you see the model making three sequential turns for three independent lookups, tighten the system prompt: "When multiple tools are needed and they don't depend on each other, call them all in one response."

Sharing HTTP clients unsafely. Check your client's docs before assuming. In Python, httpx.AsyncClient is safe to share across coroutines. In Node, both the global fetch and a custom undici Agent/Pool are safe to share across concurrent tasks — in fact, sharing a single Agent process-wide is the recommended pattern. Watch out for socket-reuse race conditions under high concurrency (see undici issue #5450) and tune keepAliveTimeout below your upstream's idle timeout.

Ignoring per-user rate limits. If your tool calls execute as the authenticated user (which they should — see our note on agent identity vs user identity), five parallel calls hit one user's rate limit budget five times faster. Size your concurrency limit accordingly.

Forgetting the empty batch. If every tool in a group errors out and the model gets only errors back, it may loop retrying. Cap retry attempts at the agent loop level, not just per tool.

Parallelising writes. The temptation to speed up a batch of writes by running them concurrently is strong. Resist it unless every write is idempotent and independent. A misclassified serial tool run in parallel is the fastest way to corrupt production data.