OpenAI tool calling: a step-by-step guide to shipping it in production

By the end of this guide, you'll have a working OpenAI tool calling loop that survives real traffic: schemas the model can actually fill, an execution layer that handles parallel calls and partial failures, and evals that catch regressions before your users do.

Prerequisites: Python 3.10+ or Node 20+, an OpenAI API key with access to the Responses API, and a service you actually want the model to call (a real endpoint, not a toy). Time required: about 90 minutes if you already have the endpoint; add a couple of hours if you're wiring auth from scratch.

This guide uses the Responses API, which is where OpenAI is consolidating tool calling. The older Chat Completions function_call shape still works, but everything below assumes you're on Responses.

Step 1 — Define the tool schema the model will see

The schema is the contract. If it's ambiguous, the model will fill it ambiguously. If it's bloated, the model will pick the wrong tool. Start with one tool and make it precise.

Write the schema as JSON Schema and register it as a function type tool on the request. Aim to keep the description under ~200 characters (the API accepts up to 1024, but shorter descriptions consistently select better), describe when to call it (not what it does internally), and mark every parameter that's actually required as required.

tools = [{
    "type": "function",
    "name": "get_order_status",
    "description": "Look up the current status of a customer order by order ID. Use when the user asks about an order they've placed.",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {
                "type": "string",
                "description": "The order ID, formatted like 'ORD-123456'."
            }
        },
        "required": ["order_id"],
        "additionalProperties": False
    },
    "strict": True
}]

Set strict: true. It forces the model's output to conform to your schema exactly, which removes a whole category of "the model returned a slightly wrong shape" bugs. It costs a small amount of first-token latency on new schemas; worth it.

If you want a deeper treatment of what makes a schema the model can actually use — parameter naming, enums, output shape — see our guide on tool schema design for AI agents.

Step 2 — Send the first request and read the tool call

Call responses.create with the user's message and your tools array. The model returns an output array. Each item has a type. When the model wants to call a tool, you get a function_call item with a name, call_id, and arguments (a JSON string, even with strict).

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-4.1",
    input=[{"role": "user", "content": "Where's my order ORD-482910?"}],
    tools=tools
)

for item in response.output:
    if item.type == "function_call":
        print(item.name, item.call_id, item.arguments)

Expected output:

get_order_status fc_abc123 {"order_id":"ORD-482910"}

If you get back a message item instead of a function_call, the model decided it didn't need the tool. That's fine — don't force it. If it never calls the tool when it should, revisit Step 1: the description is probably vague.

Step 3 — Execute the tool and return the result

Parse the arguments, call your actual function, and send the result back as a function_call_output item with the matching call_id. Include the previous response's id as previous_response_id so the model has the conversation state.

import json

tool_call = next(i for i in response.output if i.type == "function_call")
args = json.loads(tool_call.arguments)
result = get_order_status(args["order_id"])  # your real function

follow_up = client.responses.create(
    model="gpt-4.1",
    previous_response_id=response.id,
    input=[{
        "type": "function_call_output",
        "call_id": tool_call.call_id,
        "output": json.dumps(result)
    }],
    tools=tools
)

print(follow_up.output_text)

Two things to get right here. First, call_id must match exactly — the model uses it to associate the result with the call. Second, output is a string. Serialise your result, don't pass a dict.

Step 4 — Handle parallel tool calls

The Responses API returns multiple function_call items in a single response when the model decides calls can run concurrently. If you execute them serially, you leave a lot of latency on the table. If you execute them in parallel without thinking, you'll corrupt state.

Classify each tool as safe (idempotent reads, no side effects) or unsafe (writes, external side effects). Run safe calls concurrently; run unsafe calls serially, in the order the model returned them.

import asyncio

async def dispatch(item):
    args = json.loads(item.arguments)
    result = await TOOL_REGISTRY[item.name](**args)
    return {
        "type": "function_call_output",
        "call_id": item.call_id,
        "output": json.dumps(result)
    }

calls = [i for i in response.output if i.type == "function_call"]
safe = [c for c in calls if TOOLS_META[c.name]["safe"]]
unsafe = [c for c in calls if not TOOLS_META[c.name]["safe"]]

safe_results = await asyncio.gather(*(dispatch(c) for c in safe))
unsafe_results = [await dispatch(c) for c in unsafe]

all_results = safe_results + unsafe_results

Cap concurrency — a semaphore of 5–10 is a sensible default depending on your downstream rate limits. For a full treatment including partial failure semantics and testing the concurrency contract, see our guide on parallel tool calls.

Step 5 — Handle errors so the model can recover

A tool call will fail. Rate limit, bad input, downstream 500, expired token. When it does, the worst thing you can do is throw and drop the loop. Return the error to the model as a structured result — the model can retry, ask the user for clarification, or explain the failure.

async def dispatch_safe(item):
    args = json.loads(item.arguments)
    try:
        result = await TOOL_REGISTRY[item.name](**args)
        payload = {"ok": True, "data": result}
    except ValidationError as e:
        payload = {"ok": False, "error": "invalid_input", "detail": str(e)}
    except RateLimitError:
        payload = {"ok": False, "error": "rate_limited", "retry_after": 30}
    except Exception as e:
        payload = {"ok": False, "error": "internal", "detail": str(e)[:200]}
    return {
        "type": "function_call_output",
        "call_id": item.call_id,
        "output": json.dumps(payload)
    }

Structured errors — a shape the model has seen in your tool descriptions — mean the model can reason about them. Free-text stack traces mean it can't. If you're new to this, our AI agent error handling guide covers retries, circuit breakers, and recovery in depth.

Step 6 — Loop until the model stops calling tools

A single tool call is the tutorial case. Real interactions involve several turns: the model calls a tool, reads the result, calls another, decides it has enough, and writes a final message. Wrap Steps 2–5 in a loop that exits when the model returns no function_call items.

async def run_agent(user_input, max_turns=10):
    resp = client.responses.create(
        model="gpt-4.1",
        input=[{"role": "user", "content": user_input}],
        tools=tools
    )
    for _ in range(max_turns):
        calls = [i for i in resp.output if i.type == "function_call"]
        if not calls:
            return resp.output_text
        outputs = await execute_calls(calls)  # from Steps 4 and 5
        resp = client.responses.create(
            model="gpt-4.1",
            previous_response_id=resp.id,
            input=outputs,
            tools=tools
        )
    raise RuntimeError("max turns exceeded")

Always cap turns. A model in a bad state will loop — call a tool, get an error, call the same tool again, forever. Ten is a reasonable ceiling for most interactive workloads; drop it lower for latency-sensitive paths.

Step 7 — Instrument and evaluate

Before you ship, you need to know two things: are the tool calls correct, and are they fast enough. Log every turn — the input, the tools registered, each function_call name and arguments, each result, the total token count, and wall-clock time per call.

Build an eval set of 20–50 real user inputs. For each, record the expected tool name and the expected arguments. Run the eval on every model version, prompt change, and schema change. Grade three things:

1. Tool selection accuracy: did it pick the right tool?
2. Argument accuracy: did it fill the arguments correctly?
3. Trajectory: for multi-turn cases, did it get to the answer in a reasonable number of steps?

Output-only evals miss the failures that matter. A run that returned the "right" answer via three wrong tool calls is a regression waiting to happen in production. See our guide on agent evals for the full grading harness.

Common pitfalls

• Registering too many tools at once. OpenAI recommends keeping fewer than 20 tools available at any one time; beyond that, selection accuracy tends to degrade. If your catalogue is large, use dynamic loading based on the user's intent rather than shipping all of them on every request.
• Vague descriptions. "Gets customer data" is not a description. "Look up a customer's billing address by customer ID; use when the user asks about their address on file" is.
• Forgetting the previous_response_id. Without it, the model doesn't know the tool result relates to the earlier call, and you'll get repeated calls or confused output.
• Using a shared service account for tool calling in production. Every tool call should execute as the authenticated end user so permissions, data visibility, and audit trails match the real identity. Our piece on agent identity vs user identity walks through why this matters and what breaks when it isn't there.
• No timeout on tool execution. A tool that hangs will burn through your max_turns doing nothing. Set an explicit per-tool timeout and return a structured timeout error to the model.
• Assuming strict: true fixes semantic errors. It guarantees the shape, not the meaning. The model can still pass order_id: "unknown" when it doesn't know the ID. Validate on your side.