API strategy
Platform integration
Mock vs live third party APIs for integration tests: how each approach works, where each one breaks, and the combined setup that holds when agents are calling.
Every integration test suite eventually faces the same fork. You either mock the third party APIs your code depends on, or you call them for real. Both have obvious tax. Mocks drift from reality. Live calls are slow, flaky, and cost money.
This piece is for engineers deciding how to test integrations that agents will invoke in production — where the caller is non-deterministic, the tool count is high, and silent contract drift is the failure mode that kills projects. If you're wiring up a handful of REST calls for a checkout flow, the trade-offs look different than they do for a tools layer feeding an AI agent.
One-sentence summary: mock third party APIs for speed, coverage, and CI hygiene; call live APIs on a scheduled cadence to catch the drift your mocks can't see. Most production teams end up running both, deliberately.
A mock stands in for the real service. Your code makes the same HTTP call it would in production, but the request is intercepted and answered from a fixture — a canned response you wrote or recorded earlier. Tools like WireMock, Mockoon, Nock, and Prism sit on this side of the line. Contract-aware tools like Pact go further: they turn recorded consumer expectations into a contract the provider is meant to honour.
The test path is fast and deterministic. No network. No auth juggling. No rate limits. The suite runs in milliseconds per test, which means you can run it on every commit and inside every pull request. You can force error paths — 429s, 500s, malformed JSON, timeouts — that would be hard or unethical to trigger against a live sandbox.
The trade-off is that mocks encode your understanding of the third party's behaviour at the moment you wrote them. When the provider changes a field name, tightens a validation rule, or starts returning a new error code, your mock keeps happily returning the old shape. Your tests stay green. Production breaks. This is the classic mock drift problem, and it's the reason mock-only strategies fail at scale.
A live integration test calls the actual third party — usually against a sandbox or dedicated test tenant. You issue real HTTP requests, real auth tokens flow, and the responses come from the same code path that serves production traffic. Postman collections, custom test runners, and Playwright-style API test suites sit here.
The upside is honesty. If the provider changed something, your test catches it — often before your customers do. Live tests are the only way to verify that your assumptions about pagination, rate limits, idempotency, webhook signatures, and error semantics still match what the provider actually does. For agent projects that depend on dozens of third party surfaces, this is not optional. Silent drift across a large tool catalogue compounds in ways that unit tests can't detect.
The downside is everything about running them. Sandboxes are rate-limited. Test data has to be seeded and cleaned up. Auth tokens expire. Some providers don't have sandboxes at all, so your "live" tests hit production and you have to be careful about side effects. Runs take minutes, not seconds. Flakiness from network conditions or upstream incidents makes them a bad fit for per-commit CI. If you gate merges on live tests, your team will start ignoring the red builds — which is worse than not running them.
Mock when the value of the test is exercising your code, not the provider's. Unit and integration tests for business logic, error handling, retry behaviour, and edge cases belong on mocks. If you want to prove that your circuit breaker opens after five consecutive 503s, you need a mock that can return five consecutive 503s on demand. A live sandbox won't cooperate.
Mock when the suite runs on every commit. Per-commit CI has a tight time budget — the continuous delivery literature often cites something under ten minutes end to end as the goal. Live third party calls blow through that budget quickly, and they introduce a class of flakiness that has nothing to do with your code. If you want green builds to mean something, keep the network out of them.
Mock when you're testing error paths that are hard to trigger for real. Rate limit responses, partial failures, malformed payloads, expired tokens, webhook signature mismatches. These are exactly the paths agents will hit at scale, and mocks are the only practical way to cover them.
One caveat: mocks work best when paired with a discipline that keeps them honest. That's either contract testing (Pact-style, where the mock is generated from a contract the provider verifies) or a scheduled live check that flags drift. A mock with no such discipline is a lie waiting to happen.
Call live when the risk you care about is contract drift. Any test that exists to answer the question "does the provider still behave the way we think it does" has to hit the provider. There is no substitute. For agent projects, this matters more than it does for classic integrations, because the failure mode isn't a broken button — it's a tool that quietly returns the wrong data and a model that confidently acts on it.
Call live in a nightly or hourly job, not on every commit. Give it its own queue, its own reporting, and its own on-call expectation. Alert on failure, but don't gate merges. The point is fast detection of upstream change, not blocking your team's work.
Call live before every release. A pre-deploy smoke suite that exercises the critical paths against real third parties catches the drift that landed since the last nightly run. Ten minutes of real calls before ship is cheap insurance.
Call live for anything auth-related. Token refresh flows, OAuth exchanges, scope enforcement — mocks can model these, but the interesting failures happen at the provider's edge and only show up when you actually hit it. This is doubly true for delegated auth flows agents rely on, where the identity in the token has to survive the round trip.
Run both. The interesting question isn't mock versus live — it's how to combine them so the mocks stay honest and the live checks stay affordable.
A setup that holds in production:
For agent projects specifically, this is where the discipline gets harder. If your product exposes fifty tools to an agent, you have fifty surfaces to keep honest — and the third party API change management tax grows linearly. This is where teams underestimate the operating cost. Detecting OpenAPI spec drift in one connector is a Tuesday afternoon. Detecting it across fifty is a full-time engineering function.
The reason we keep coming back to drift detection is that it's the failure mode agents don't recover from. A broken UI button gets a user ticket. A broken tool returns plausible-looking JSON and the agent keeps going. That's the shape of the problem the tools layer has to hold.
Pontil is a Tools-as-a-Service platform. We make SaaS products accessible to AI agents by generating and running tools against the APIs that already exist — including the third party APIs your product depends on. Because those tools are generated from source contracts and run through a managed runtime, drift detection is built into the maintenance loop rather than being a separate testing project you have to fund. The mocks-versus-live trade-off doesn't disappear, but it stops being fifty teams' problem and starts being one platform's problem.
If you're staring at a tool catalogue and doing the maths on how many nightly live checks you'd need to keep it honest, that's the moment worth a conversation.
The decision isn't binary and it isn't fixed. Start with mocks for CI, add scheduled live checks against the surfaces you actually depend on, and treat the two suites as complementary rather than competing. Then measure how often the live suite catches something the mocks missed. That number tells you whether your contract discipline is working — and whether the tools your agents call are as reliable as they look.
Stay up to date on the ever changing agentic landscape.
GET STARTED
Start with a 30-minute discovery call
contactus@pontil.com
