Platform integration
Agent infrastructure
Webhook dead letter queue guide: build retry logic, capture failed webhooks in a DLQ, alert on depth, and replay safely with idempotency in seven steps.
Webhooks fail. Endpoints time out, downstream services return 500s, signatures don't verify, payloads land malformed. When agents depend on those events, silent failures become silent bugs. A dead letter queue (DLQ) is the pattern that catches what your retry logic can't fix, so nothing gets dropped and every failure is inspectable.
By the end of this guide, you'll have a working webhook DLQ pattern: a retry policy with exponential backoff, a DLQ that captures poison messages, an inspection path for humans, and a replay mechanism. Prerequisites: a webhook consumer running somewhere you control, access to a queue (SQS, RabbitMQ, Redis Streams, or equivalent), and about an hour.
The first mistake most teams make: they run business logic inside the HTTP handler that receives the webhook. That couples delivery success to processing success. If your database is slow, the sender retries. If your logic throws, the sender gives up.
Split it in two. The HTTP handler does three things and nothing more: verify the signature, write the raw payload to a queue, return 200. Processing happens on a worker that pulls from that queue.
@app.post("/webhooks/stripe")
def receive_webhook(request):
raw_body = request.body
signature = request.headers["Stripe-Signature"]
if not verify_signature(raw_body, signature, SECRET):
return Response(status=401)
queue.send(
MessageBody=raw_body,
MessageAttributes={
"source": {"StringValue": "stripe", "DataType": "String"},
"received_at": {"StringValue": now_iso(), "DataType": "String"},
},
)
return Response(status=200)
Expected result: senders typically get their 200 in tens of milliseconds — actual latency depends on signature verification cost, queue put latency, and whether your handler is colocated with the queue — and your processing pipeline is now free to fail without triggering upstream retries.
If you haven't nailed signature verification yet, do that first — see our practical HMAC-SHA256 guide before continuing.
Retries belong on the consumer side of the queue, not in the HTTP handler. The worker pulls a message, tries to process it, and on failure the queue re-delivers according to a policy you set.
The policy has three knobs: maximum attempts, backoff schedule, and what counts as a retryable error. A common shape:
On SQS, this maps to VisibilityTimeout and maxReceiveCount. On RabbitMQ, use a retry exchange with TTL. On Redis Streams, track attempt counts in the message metadata.
def process_message(msg):
try:
payload = json.loads(msg.body)
handle_event(payload)
msg.delete()
except NonRetryableError as e:
send_to_dlq(msg, reason=str(e))
msg.delete()
except RetryableError:
if msg.receive_count >= 5:
send_to_dlq(msg, reason="max_retries_exceeded")
msg.delete()
# else: leave in queue, visibility timeout handles the delay
Expected result: transient failures resolve on their own. Permanent failures move to the DLQ fast, without wasting five retry cycles on a payload that will never process.
The DLQ is a separate queue with no consumers by default. Its job is to hold failed messages until a human decides what to do.
On SQS, you configure the source queue's redrive policy to point at the DLQ. On RabbitMQ, set x-dead-letter-exchange on the source queue. The mechanics vary; the principle doesn't.
What matters is what you write into the DLQ message. The raw payload isn't enough. You need:
Without this metadata, DLQ inspection turns into archaeology. With it, an on-call engineer can triage a message in under a minute.
One message in the DLQ is not an incident. Fifty messages in an hour is. Alert on the rate and the depth, not on every arrival.
A reasonable starting policy:
The per-source threshold matters. A single misbehaving sender shouldn't hide behind aggregate metrics. If Stripe suddenly starts sending payloads that fail schema validation, you want to know that specifically — not just that "webhooks are failing."
If you're thinking about observability more broadly, our guide on monitoring third-party API integrations covers the same principles for outbound calls.
Humans need to look at DLQ messages. Give them a way that doesn't involve SSH-ing to a queue node.
Minimum viable inspection: a CLI or admin UI that lists DLQ messages with their metadata, lets you view the full payload, and shows the failure reason. If you're on AWS, the SQS console works for small volumes but breaks past a few hundred messages. Build the tool before you need it.
$ dlq inspect --source stripe --limit 5
MSG_ID SOURCE RECEIVED REASON ATTEMPTS
7f3a-c2b1-... stripe 2026-05-04 14:22:03 ValidationError: missing 5
7f3a-c2b1-... stripe 2026-05-04 14:19:11 ValidationError: missing 5
7f3a-c2b1-... stripe 2026-05-04 14:15:47 DownstreamTimeout 5
The inspection tool should also let you filter by source, failure reason, and time range. When 200 messages land at once, filtering is how you find the pattern.
Once you've fixed the bug that caused messages to fail, you need to reprocess them. Replay is the operation that pulls a message from the DLQ and puts it back on the main queue.
Two rules for replay:
$ dlq replay --source stripe --reason "ValidationError: missing" --dry-run
Would replay 47 messages. Confirm with --execute.
$ dlq replay --source stripe --reason "ValidationError: missing" --execute
Replayed 47 messages. Audit log: /var/log/dlq-replay-2026-05-04.log
Selective replay by source, reason, or time range is worth building on day one. Bulk-replaying everything in the DLQ is almost always the wrong operation.
DLQs grow. Set a retention policy — 14 days is common — and archive older messages to cold storage before they expire. You want the audit trail even after the operational window closes.
On SQS, MessageRetentionPeriod handles this. On other queues, run a scheduled job that pulls expiring messages, writes them to S3 or equivalent, and deletes them from the DLQ. The archived format should match what your inspection tool reads, so an engineer can still replay a message from three months ago if a customer complaint surfaces.
Retrying non-retryable errors. A malformed payload will fail on attempt 5 the same way it failed on attempt 1. Classify errors at the point of failure and skip retries for anything a retry can't fix.
Using the same DLQ for every source. When a Stripe outage floods the DLQ, GitHub webhook failures get lost in the noise. Separate DLQs per source, or at minimum, filterable metadata.
No idempotency on replay. The most expensive DLQ bug is the one that double-charges a customer after replay. Idempotency keys are not optional.
Alerting on individual DLQ messages. Every alert train the on-call engineer to ignore the DLQ. Alert on rate and depth thresholds.
Building the DLQ but not the replay path. A DLQ without replay is a graveyard. Messages accumulate, no one processes them, and eventually the retention window drops them. Build inspection and replay before you consider the DLQ done.
Forgetting to test the DLQ path. In staging, force failures on purpose — bad payloads, downstream timeouts, unhandled exceptions — and verify messages land where they should, with the metadata you expect. Untested DLQs fail silently in exactly the way they're meant to prevent.
Webhook infrastructure that agents can trust is built on this shape: fast receipt, decoupled processing, disciplined retries, a DLQ that captures the rest, and a replay path that closes the loop. Everything else is variations on the theme.
Stay up to date on the ever changing agentic landscape.
GET STARTED
Start with a 30-minute discovery call
contactus@pontil.com
