Agents in production
Platform integration
Queue-first webhook architecture: why the receiver should never do the work, how backpressure actually behaves, and where the pattern breaks in production.
Every webhook receiver eventually meets the same problem. The sender fires an event. The receiver picks it up, does the work inline, and returns a 200. That works until traffic spikes, a downstream system slows down, or the receiver restarts mid-request. Then events vanish, retries hammer the wrong endpoint, and the on-call channel fills up.
Queue-first webhook architecture answers this by splitting the receiver into two jobs. The HTTP endpoint accepts the event, persists it, and acknowledges. A separate worker does the actual processing. It sounds obvious. Most teams still don't do it, because the inline pattern feels simpler and the failure modes only show up under load.
This piece takes a position: for anything past a prototype, the receiver should never do the work. Below, five sections on why the queue belongs at the front, how backpressure actually behaves, what changes for agents, where the pattern breaks, and what a production-grade setup looks like.
The inline pattern collapses two responsibilities into one HTTP handler: acknowledging the event and acting on it. Under normal load, this is fine. Under any of the four conditions webhook systems actually hit — traffic bursts, downstream slowness, receiver deploys, or partial failures — it fails in ways that look like other problems.
Senders typically retry on non-2xx responses or timeouts. If your handler takes 8 seconds because a downstream API is slow, and the sender's timeout is 5, the sender retries. You now process the same event twice. If you added idempotency at the top of the handler, you'll reject the retry — but only after the sender has already burned attempts and possibly moved the delivery to a dead letter. If you didn't, you've duplicated the side effect.
Worse: the receiver often has no memory of what it saw. If it crashes between accepting the request and finishing the work, the event is gone from your side. The sender thinks it delivered. You have no record. This is the class of failure a dead letter queue is designed to catch, but a DLQ only helps if something wrote the event down before the crash.
The fix isn't faster handlers. It's separating the two responsibilities.
A queue-first receiver has three parts. An HTTP endpoint that does the minimum needed to accept the event. A durable queue that holds it. A worker that processes it asynchronously.
The HTTP endpoint's job is narrow:
That's it. No database writes to business tables. No calls to downstream APIs. No enrichment. If the handler can't finish those three steps in under 100ms, something is wrong at the infrastructure level, not the application level.
The queue itself can be a hosted broker (SQS, Pub/Sub, Kafka, RabbitMQ), a database-backed queue (Postgres with SELECT ... FOR UPDATE SKIP LOCKED, or something like river or Oban), or a durable execution platform like Temporal that exposes queue-like semantics through task queues backed by workflow state. The choice matters less than the property: once the event is durably recorded, it survives receiver restarts, worker crashes, and downstream outages.
The worker pulls from the queue, does the actual work — updating your database, calling downstream services, notifying agents — and acknowledges the message only after the work completes. If it crashes mid-processing, the message becomes visible again after the visibility timeout and another worker picks it up.
Most writing on webhook queues frames the queue as a durability mechanism. It is, but that undersells it. The bigger property is backpressure — the ability to let the queue absorb load the workers can't yet process, without breaking the contract with the sender.
Without a queue, backpressure has nowhere to live. If the receiver is overloaded, it either slows down (and the sender times out) or crashes (and the sender retries). Both are lossy in practice. The sender's retry policy — usually exponential backoff with a cap on attempts — is designed to survive brief hiccups, not sustained downstream slowness. If your worker pool needs 20 minutes to catch up because a downstream partner is degraded, sender retries will move events to their dead letter long before you're ready to accept them.
With a queue in front, the shape changes. The HTTP endpoint keeps returning 200 because it's only writing to the queue. Queue depth grows. Workers process at the rate they can sustain. When the downstream recovers, workers catch up. The sender sees a healthy endpoint the whole time.
This only works if you actually monitor queue depth and worker throughput. A queue that grows without alerts is a delayed outage. Two metrics matter more than any others:
Alerting on both catches the two failure modes: workers falling behind steadily (depth) and workers stuck on a poison message that's blocking the head (age).
Webhooks in a purely system-to-system context have well-understood semantics. When agents are downstream of the webhook — reacting to a Stripe charge, a Salesforce record change, a support ticket update — three things get harder.
Fan-out is asymmetric. One incoming webhook can trigger multiple agent actions, some of which want the event immediately and some of which want it batched or deduplicated. A queue-first architecture lets you write once and read many: the raw event lands in the queue, and different consumers pull it for different purposes. Inline processing forces you to decide, at request time, what all the downstream reactions are. You never actually know that.
Latency budgets are visible. An agent waiting for a webhook to trigger the next step is measuring end-to-end time — from the source system's event to the agent's next tool call. If your receiver takes 3 seconds because it's doing the work inline, that's 3 seconds the agent's context is stalled. A queue-first receiver returns in under 100ms and the worker handles processing on its own clock. The agent gets the resulting state change when the worker finishes, but the source system isn't waiting on you.
Idempotency becomes a runtime property, not a database constraint. When agents are consuming the events downstream of your queue, you can no longer assume that "process this event once" means "call the database once." It means "reach a consistent final state at the tool boundary regardless of how many times the worker retries." That's a design constraint on the tools, not just on the handler.
This is why the pattern matters beyond hyperscaler webhook volumes. Any team building agents on top of event-driven signals ends up needing the same shape.
Queue-first isn't free, and honest deep-dives should name the trade-offs.
Ordering guarantees weaken. A single-threaded inline handler processes events in the order they arrive. A worker pool with N workers pulling from a queue does not, unless you partition by key. If ordering matters — say, all events for a given account must process in order — you need a partitioned queue or a key-based routing layer. Most brokers support this (Kafka partitions, SQS FIFO with message group ID, Pub/Sub ordering keys), but you have to design for it. Losing implicit ordering silently is a common bug.
Debugging gains a hop. With inline processing, a failed webhook is one log line. With queue-first, you have three: the HTTP request, the queue enqueue, the worker attempt. Tracing across those needs a delivery ID propagated through all of them and a log aggregator that can join on it. Without that, debugging becomes archaeology.
Poison messages block progress. If a specific event causes the worker to crash — bad payload, unhandled edge case, downstream returning a permanent error — that event will be re-delivered until it hits the max attempts and moves to the DLQ. If you don't configure max attempts, or set them too high, a single bad message can stall the queue behind it (for FIFO queues) or waste worker capacity (for unordered ones). This is well-covered ground; the fix is disciplined DLQ handling and alerting on DLQ depth, not avoiding queues.
The queue itself becomes infrastructure you own. Broker outages, disk-full events on the queue host, mis-set visibility timeouts — these are now your problems. Managed queues push most of this to the cloud vendor, but not all. Self-hosted brokers push it all back to you.
None of these are reasons to skip the pattern. They're reasons to implement it deliberately.
Webhooks are one half of the event story for agent projects. The other half is the outbound tool call — how the agent acts on the state change that just arrived. Both halves share the same production discipline: durable delivery, retries with backoff, idempotency at the boundary, per-user identity on every action.
Pontil sits in the tools layer of the agent stack. We generate the tools agents call and run them through a managed runtime — the same runtime that handles retries, failure classification, and executing as the authenticated user rather than a shared service account. When a queue-first webhook receiver hands work to a worker that then invokes agent tools, the runtime is the boundary that decides whether the retry is safe, whether the user still has permission, and whether the downstream call went through. See our product overview for how the pieces connect.
Most teams reach queue-first by accident, after an incident. The receiver went down during a spike, or a downstream slowness burned through sender retries, or a poison message loop chewed through capacity. The rebuild that follows is usually the queue-first design described above.
The better path is to start there. Signature verification, persist, ack, return — everything else is a worker's problem. Once that split is in place, you can add backpressure monitoring, DLQ handling, key-based ordering, and consumer fan-out without rewriting the receiver.
The harder question is what runs behind the queue. Workers that update your own database are straightforward. Workers that call downstream APIs — or agent tools — need the same discipline the receiver just gained: bounded retries, idempotent operations, per-user identity, and observability across every hop. That's where the next round of production work usually lands.
Stay up to date on the ever changing agentic landscape.
GET STARTED
Start with a 30-minute discovery call
contactus@pontil.com
