What is a webhook? A deep-dive on the pattern, its failure modes, and what agents change

Ask ten engineers what a webhook is and you'll get ten answers that all sound right and none of which are complete. "An HTTP callback." "A reverse API." "A push notification for servers." All true, all missing the part that matters in production.

A webhook is a delivery mechanism. One system tells another that something happened, over HTTP, with a payload. That's the definition. Everything hard about webhooks — signatures, retries, ordering, idempotency, dead letters — is what you have to build around that definition to make it survive contact with reality.

This piece is a deep-dive on the pattern itself: what a webhook actually is, how the mechanics work end to end, where webhooks sit against APIs and polling, the failure modes that break most implementations, and what changes now that AI agents are on both sides of the wire. If you're evaluating whether webhooks fit a problem you have, this is the ground you need to cover first.

The pattern in one sentence, then the honest version

A webhook is an HTTP POST from one system to a URL you own, sent when a specific event happens on their side. Stripe charges a card, a POST hits your endpoint. GitHub gets a push, a POST hits your endpoint. Same shape every time: an event happens, an HTTP request goes out, your server responds with a status code.

The honest version is longer. A webhook is a fire-and-forget HTTP request from a sender that has no persistent connection to you, no shared session, no guaranteed delivery, and no built-in ordering. The sender knows nothing about your health, your capacity, or whether you've seen this event before. If your endpoint is down when the event fires, the sender's retry policy — which you didn't write and probably haven't read — decides whether you ever see it. If your endpoint is up but slow, the sender may time out and retry, and now you have two copies of the same event racing through your system.

That gap between the tidy definition and the operational reality is where every production webhook problem lives.

How a webhook actually works, end to end

Walk through a single event. A user in Stripe's dashboard refunds a charge. Stripe's internal system emits an event — internally, probably to a queue. A dispatcher reads the event, looks up which endpoints the merchant has registered for charge.refunded, and enqueues a delivery job per endpoint. A worker picks up the job, serialises the event as JSON, signs the body with the endpoint's secret, opens an HTTPS connection to the merchant's URL, and POSTs.

On the merchant side, a load balancer routes the request to a receiver process. The receiver reads the raw body, verifies the signature against the shared secret, checks whether it's seen the event ID before, and then — if it's smart — enqueues the event for asynchronous processing and immediately returns 200. If it's not smart, it does the actual work inline: updating the order, notifying the customer, refunding the loyalty points, and only then returns 200. If any of that takes longer than Stripe's timeout, Stripe assumes failure and retries. Now you have two workers processing the same refund.

On the return trip, the receiver's status code tells the sender what to do next. Policies vary by sender, but a common pattern is: 2xx means success — mark the delivery complete, move on. 4xx (other than 429) means the receiver rejected the event permanently — don't retry, log it, maybe alert the endpoint owner. 5xx or a network timeout means transient failure — retry, usually with exponential backoff and jitter, up to some maximum. Some senders are more aggressive: Stripe, for instance, retries on any non-2xx response for up to 72 hours before auto-disabling the endpoint. After the maximum, the event goes to a dead letter queue on the sender's side, and the endpoint owner gets an email nobody reads.

That's the whole mechanic. Six components — event source, dispatcher, delivery worker, receiver, processing pipeline, retry policy — and the interesting engineering happens in the gaps between them.

Webhook vs API: not the debate people think it is

People frame this as "webhook vs API" as if they're alternatives. They aren't. A webhook is an API — it's an HTTP endpoint with a contract, versioning problems, and auth. The real distinction is who initiates the call.

In a normal API interaction, your code initiates. You want data, you ask for it. You control when the call happens, how often, and what you do with the response. The server is passive; it waits.

In a webhook interaction, the roles flip. The other system initiates. You expose an endpoint and wait. You don't control when calls arrive, how many arrive at once, or the order they arrive in. The client side of the connection — the sender — is now something you don't own.

That inversion has three practical consequences. First, capacity planning is different: your receiver needs to handle bursts you didn't schedule. Second, error handling is different: you can't just retry your own failed call; you have to signal failure back to a system whose retry policy you didn't write. Third, security is different: anyone can POST to a public URL, so you need a way to prove the request came from the sender you expected.

Compared to polling — where your code repeatedly asks "anything new?" — webhooks are more efficient and lower latency when events are sparse, and worse when events are dense or your receiver is fragile. We covered the trade-off in detail in our webhook vs polling decision guide; the short version is that polling gives you control at the cost of latency, and webhooks give you latency at the cost of control.

Where webhooks fit and where they don't

Webhooks win when events are sparse, latency matters, and the receiver is genuinely production-grade infrastructure. Polling wins when events are dense, ordering matters, or the receiver is fragile enough that catching up on restart is safer than absorbing a live push.

Signatures, timestamps, and why HTTPS isn't the whole answer

Your webhook URL is public. Anyone who guesses it can POST to it. "But it's HTTPS" is not an answer — HTTPS proves the transport is encrypted, not that the sender is who they claim to be.

The standard answer is HMAC signing. The sender computes an HMAC of the raw request body using a shared secret, puts the result in a header, and your receiver recomputes the HMAC and compares. If they match, the request came from someone who knows the secret. If they don't, drop the request. We wrote the full mechanic up in our HMAC-SHA256 signature verification guide, including the two things people get wrong most often: signing over the parsed JSON instead of the raw bytes, and using string equality instead of constant-time comparison.

Signatures aren't enough on their own. An attacker who captures a valid signed request can replay it — same body, same signature, same everything. The fix is a timestamp inside the signed payload and a small acceptance window. Typical implementations reject anything older than a few minutes (five is a common default, used by Stripe and Slack among others; some providers pick a different window or leave it to the receiver), and replay attempts die.

And secrets rotate. Or they should. A shared secret that's been in production for three years and shared across four vendor onboarding docs is not a secret. The pattern for rotating without downtime — dual-key acceptance windows, JWKS distribution for asymmetric schemes — is a whole topic on its own, covered in webhook signing key rotation without downtime.

The failure modes that break most implementations

Every webhook receiver eventually meets the same four failures. If you haven't hit them, you haven't run the pattern in production long enough.

Duplicates. The sender retries. Your receiver processes the retry. Now the customer gets two refund emails, or the order gets shipped twice, or the audit log has two entries for one event. The fix is idempotency: extract a stable event ID from the payload, claim it atomically before you process, and no-op on the second attempt. Full seven-step version in the webhook idempotency key guide.

Out-of-order delivery. The sender fired order.created and order.updated a millisecond apart. Their workers picked them up in reverse order. Your receiver applies the update to an order that doesn't exist yet, gets a foreign key error, and returns 500. The sender retries, this time in the right order, and everything works — except now your metrics show a spike in errors that isn't a real problem. The fix isn't to demand ordering from the sender (most senders don't offer it); it's to make your processing tolerant of it. Look up the resource, apply the change if it makes sense, defer if it doesn't.

Receiver overload. A batch job on the sender's side fires ten thousand events in thirty seconds. Your receiver, which was sized for the average rate, falls over. Requests time out, the sender retries them all, your receiver falls over harder. The fix is queue-first architecture: your receiver's only job is to verify the signature, enqueue the event, and return 200. Actual processing happens asynchronously with its own backpressure. We explain why this pattern holds where inline processing doesn't in queue-first webhook architecture.

Silent drop. Your receiver is down for two hours during a deploy. The sender retries for a while, then gives up. When you come back up, there's no signal that anything is missing. The events are just gone. Fixes here are layered: sender-side dead letter queues you can replay from, receiver-side reconciliation jobs that check for missing sequence numbers, and — if the sender supports it — a replay API you can hit after any outage. Handling the dead letter side of this is covered in the webhook dead letter queue guide.

None of these are exotic edge cases. They're what happens on a normal Tuesday if you shipped a naive implementation.

What changes when AI agents are involved

Webhooks have been around since Jeff Lindsay coined the term in 2007. The pattern is old. What's new is that agents are now on both sides of the wire, and that shifts the requirements in three ways.

First, agents consume webhooks as triggers for autonomous action. A support agent listening for ticket.created doesn't just log the event — it reads the payload, decides what to do, and calls tools. If the webhook payload is missing context, the agent hits your API to fetch it, then hits it again, then again. Webhooks designed for a human-driven UI to refresh a badge count are usually too thin for an agent that needs to act. Fatter payloads, richer type information, and stable event schemas matter more when the consumer is an agent.

Second, agents produce events that other systems subscribe to. When your agent creates an invoice or updates a CRM record, downstream webhooks fire. If the agent retried the operation because it timed out, the webhook fires twice. Idempotency at the tool call layer and idempotency at the webhook layer are now the same problem, and both have to hold. See API idempotency for AI agents for the tool-side view.

Third, the delivery guarantees agents need are stricter than the ones humans tolerated. A human noticing a missing notification will refresh the page. An agent won't. If the payment.succeeded event dropped, the agent's decision tree hits the wrong branch and stays there. This is why the reliability patterns — retries, dead letters, idempotency, reconciliation — stop being nice-to-haves and start being the difference between an agent that works and an agent that quietly corrupts data.

Standardisation is trying to catch up. The CloudEvents specification from CNCF fixes some of the payload-shape variance across senders. It doesn't fix delivery semantics, and it doesn't answer the questions agents actually ask — what capability does this event grant me, what should I do next, what's the schema of the resource this event refers to. Those are tools-layer questions, not transport-layer questions, and the transport layer isn't going to answer them.

So when should you actually use webhooks?

Use webhooks when four things are true. Events are sparse enough that polling would waste more resources than the webhooks cost you. Latency matters enough that waiting for the next poll would break the user experience. Your receiver is production-grade enough to handle bursts, verify signatures, deduplicate, queue, and reconcile. And the sender's delivery guarantees are strong enough — or your reconciliation logic is thorough enough — that a silent drop won't corrupt your state.

Don't use webhooks when you need ordering that the sender doesn't provide, when your receiver can't hold up under bursts, when the sender's retry policy is unknown or thin, or when the cost of a missed event is high and there's no reconciliation path. Those cases are what polling and pull-based APIs still exist for.

The deeper answer to "what is a webhook" isn't the definition — it's the honest map of everything that has to be true around the pattern for it to hold. The pattern is simple. The engineering isn't. And now that agents are on both sides, the gap between the simple pattern and the real engineering is where most integration projects will spend their next twelve months.