Function call: what it means for LLMs, and where it breaks in production

A function call in the context of an LLM is the moment the model stops generating prose and emits a structured request to invoke code — a named function with a set of typed arguments — that something outside the model will actually run. It's the mechanism that turns a chat interface into an agent. And it's the seam where most agent projects quietly fall apart.

The mechanism itself is straightforward. Every major foundation model provider now supports it, and the shape has converged: describe the functions you want the model to be able to call, pass that description in with the prompt, and the model responds with either text or a structured call. The interesting part isn't the API. The interesting part is what happens between the model deciding to call a function and the function returning a useful answer.

This piece walks through what a function call actually is, how the major providers implement it, what changes when function calling meets a real product, and why the failure modes have almost nothing to do with the model.

What a function call actually is

Strip the branding off and a function call is a structured output. The model has been trained to emit a JSON object with a function name and arguments when the prompt and the available function definitions make that the right response. The runtime — your code, not the model — parses the JSON, dispatches to the named function, executes it, and feeds the result back into the next turn.

The important word there is emit. The model doesn't call anything. It produces text that describes a call. Everything after that is your problem: validating the arguments, checking authorisation, running the code, handling failure, deciding what to send back. The mental model that trips teams up is treating function calling like a remote procedure call where the model is the client. It isn't. The model is a very good structured-output generator that happens to have been fine-tuned to produce output shaped like function invocations.

This matters for three reasons. First, the model can produce a call that's syntactically valid and semantically wrong — right shape, wrong function, wrong arguments. Second, everything about reliability, security, and observability sits on your side of that seam, not the provider's. Third, the same model behaviour that makes function calling work — pattern-matching a description to a request — is the behaviour that makes it fail when descriptions are ambiguous, overlapping, or too numerous.

How the major providers implement it

OpenAI introduced function calling in June 2023 and has since rebranded it as tool calling in the Responses API, though the mechanism is identical. You pass a tools array with JSON Schema definitions; the model returns a tool_calls array on the response. Anthropic shipped tool use GA for Claude in May 2024 with a similar shape — JSON Schema for parameters (under input_schema), round-tripped via tool_use and tool_result blocks. Google's Gemini uses functionDeclarations with an OpenAPI 3.0 Schema subset. The Model Context Protocol (MCP), which Anthropic released in late 2024, standardises the tool-definition and tool-call shape across providers.

The convergence is real but the details still matter. Providers differ on parallel tool calling (whether the model can emit multiple calls in one turn), streaming behaviour (whether tool calls arrive incrementally or as a complete object), error surfaces (how you tell the model a call failed), and JSON Schema coverage (not every provider supports every schema keyword). Cross-provider agent code that assumes uniform behaviour breaks in ways that look like model regressions but are actually provider quirks.
‍

We've written elsewhere on where MCP and REST APIs actually fit different problems and on the distinction between tool calling and function calling as terms. The short version: the mechanism is the same, the production reality diverges once tools stop being toy demos.

Why function calling breaks in production

The demo works. Two functions, one clear intent, the model picks the right one, arguments look sensible. Then the project scales — five functions, then twenty, then a hundred — and accuracy collapses. Function calling has three failure modes that compound as the surface grows.

Selection failure. The model picks the wrong function. Two functions with overlapping descriptions, or a function whose description doesn't quite match what it does, and the model routes to the wrong one. This isn't fixable by trying a bigger model. It's a design problem: the tool descriptions the model reads at inference time need to be unambiguous relative to each other, not just individually correct. We've dug into the mechanics in agent tool selection.

Argument failure. The model picks the right function and fills the arguments wrong. Off-by-one on enum values, hallucinated IDs, dates in the wrong format, required fields omitted. Some of this is schema design — the model can only produce what your schema describes — and some is missing constraints. If your schema says status: string when it should say status: "active" | "paused" | "archived", the model will invent values that look plausible. Tool schema design is where a lot of this gets fixed before it reaches the model.

Execution failure. The call is right, the arguments are right, and the function fails anyway. Rate limits, expired tokens, a downstream API that returned a shape the tool didn't handle, a transient network error. The model wasn't wrong. Your runtime was underbuilt. Retries need to be idempotent, tokens need to be scoped to the calling user, errors need to come back in a shape the model can reason about. See AI agent error handling for the retry and circuit-breaker patterns that hold up.

A fourth failure mode sits above all of these: surface failure. The function the agent needs doesn't exist. The product can do the thing — the UI does it every day — but there's no API for it, so no function to describe to the model. This is the failure mode that stalls agent projects at established SaaS companies. The model isn't picking wrong; there's nothing to pick.

Schema design decides most of the outcome

Before you tune prompts, before you upgrade models, before you add a router, look at the schemas. The JSON Schema you hand the model is the entire contract. Ambiguous names, missing enums, optional fields with unclear semantics, overlapping tool descriptions — every one of those turns into a runtime failure that looks like a model problem.

A few patterns that hold up:

• Name the function after the outcome, not the endpoint. cancel_subscription beats patch_subscription_v2. The model matches intent to name; endpoint verbs don't help it.
• Constrain enums explicitly. If a field has three legal values, list them in the schema. The model will fill in one of them. Leave it as string and you'll see all three plus a dozen the model invented.
• Describe what the function does, not what it is. "Cancels the customer's subscription at period end and stops future billing" is a description. "Subscription cancellation endpoint" is a label.
• Keep parameters flat where you can. Deeply nested objects are harder for the model to fill correctly. If a nested shape is required by the underlying API, flatten at the tool layer and reconstruct in your runtime.
• Return structured errors. When a call fails, send back JSON with a machine-readable error code and a short human-readable message. The model can reason about { "error": "insufficient_permissions", "message": "User lacks admin role" }. It cannot reason about a 500 with an HTML body.

The more schemas you have, the more the second point compounds. Two functions with similar names and vague descriptions are worse than one function with a clear name. If you can't tell which of two tools should be called for a given request, the model can't either.

How Pontil fits

The hardest part of function calling in an established SaaS company isn't writing the schema. It's that most of the product's capability isn't exposed by an API in the first place — years of UI-first development left the API layer covering a thin slice of what the product can actually do. So the functions the agent needs don't exist to describe.

Pontil is a Tools-as-a-Service platform. We generate tools directly from your existing codebase, so agents can reach the capability that's already there without waiting on an API rewrite. Every tool call runs through a managed runtime that executes as the authenticated user, with the auth, rate limits, and observability agents actually need in production. Schemas are generated with the constraints and descriptions that make function calling work — enum values pinned, outcomes named clearly, errors returned in shapes the model can reason about. If your agent project has stalled because the functions your model needs don't exist yet, book a demo and we'll walk through what changes.

Where does function calling go from here?

The mechanism is settled. Every provider supports it, MCP is standardising the shape across providers, and the ergonomics will keep improving at the API layer. That's not where the interesting work is anymore. The interesting work is upstream and downstream of the call itself.

Upstream: how do agents discover which functions exist when the catalogue has thousands of tools rather than dozens? Static tool lists in the context window don't scale past somewhere between a few dozen and a hundred functions before selection accuracy degrades, depending on model and tool similarity. Dynamic retrieval — indexing tool descriptions and loading only the relevant subset per turn — is where large tool surfaces are heading. We've written on semantic tool retrieval for the pattern and its trade-offs.

Downstream: how do you run function calls reliably when the agent is acting on behalf of a real user, in a real product, with real permissions and audit requirements? The runtime is where security reviews happen, where the failure modes get expensive, and where the difference between a demo and a production system actually lives. Getting the schema right is table stakes. Getting the runtime right is what ships.

The teams that treat function calling as an API integration problem tend to stall. The teams that treat it as a product-surface-and-runtime problem tend to ship.