Webhook retry with exponential backoff: a practical implementation guide

Webhook deliveries fail. Receivers time out, return 5xx, or drop the connection mid-request. Retrying naively — every second, forever — turns a small outage into a self-inflicted DDoS on your own consumers. Exponential backoff with jitter is the pattern that holds.

By the end of this guide you'll have a working webhook retry policy: a schedule that spaces attempts exponentially, jitter that stops retry storms, a bounded attempt cap, and the classification logic that decides what to retry in the first place. Prerequisites: a webhook dispatcher that already signs and sends events, and a job queue or scheduler that can defer work. Time required: about an hour to implement, another hour to test.

Step 1 — Classify the failure before you retry it

Not every failed delivery deserves a retry. Retrying a 400 wastes cycles and can amplify bad data. Retrying a 401 hides an auth problem that needs human attention. The retry policy starts with a classifier.

Run this check on every delivery response:

def should_retry(response, exception):
    if exception is not None:
        # Network errors, timeouts, DNS failures — always retry
        return True
    status = response.status_code
    if status in (408, 429):
        return True  # Timeout or rate limited
    if 500 <= status < 600:
        return True  # Server error
    return False  # 2xx = success, 4xx (except above) = permanent

Expected result: 2xx marks the delivery done. 4xx (except 408/429) goes straight to your dead letter queue — no retry, human review. 5xx, 408, 429, and network errors enter the backoff schedule.

Step 2 — Pick the base delay and multiplier

The schedule is delay = base * (multiplier ^ attempt). Two knobs: base (the first retry delay) and multiplier (how fast the gap grows).

Sensible defaults for most webhooks:

• base = 1 second
• multiplier = 2
• max_delay = 1 hour (cap so the tail doesn't stretch to days)

That gives you 1s, 2s, 4s, 8s, 16s, 32s, 1m, 2m, 4m, 8m, 16m, 32m, 1h, 1h, 1h. Fast enough to recover from a 30-second blip, patient enough to survive a two-hour outage without hammering the receiver.

If your consumers are latency-sensitive (fraud signals, real-time chat), drop base to 250ms and multiplier to 1.5. If they're batch-oriented (billing exports, nightly syncs), raise base to 5s.

Step 3 — Add jitter

Exponential backoff without jitter creates thundering herds. If ten thousand deliveries all fail at the same moment — say, because the receiver rebooted — they all retry at exactly t+1s, then t+3s, then t+7s. Every retry wave hits the receiver in lockstep, right as it's coming back up.

Jitter randomises each attempt so the herd spreads out. Full jitter (recommended for most cases) picks a random value between 0 and the calculated delay:

import random

def backoff_delay(attempt, base=1.0, multiplier=2.0, max_delay=3600):
    exponential = min(base * (multiplier ** attempt), max_delay)
    return random.uniform(0, exponential)

Expected result: attempt 5 with base=1, multiplier=2 gives you a delay somewhere between 0 and 32 seconds, uniformly distributed. Ten thousand retries no longer arrive in a spike — they smear across the window.

Equal jitter (delay/2 + random(0, delay/2)) is a middle ground if you want a floor. Full jitter is the AWS Architecture Blog's default recommendation and it's the right call for outbound webhooks.

Step 4 — Cap the attempts

Infinite retries are a resource leak dressed up as reliability. Set a hard cap.

For most webhook consumers, 15–25 attempts is the right envelope. With base=1, multiplier=2, max_delay=3600, 15 attempts covers roughly 4 hours of intermittent failure (the pre-cap ramp adds about a minute, then twelve capped 1-hour retries). Every additional attempt after that adds another hour, so 25 attempts stretches the tail to just over half a day — long enough that the receiver's on-call has time to notice and fix, short enough that stale events don't pile up forever. Pick the number based on how long you're willing to hold a payload before giving up.

MAX_ATTEMPTS = 20

def schedule_retry(delivery, attempt, response, exception):
    if not should_retry(response, exception):
        return move_to_dlq(delivery, reason="permanent_failure")
    if attempt >= MAX_ATTEMPTS:
        return move_to_dlq(delivery, reason="exhausted_retries")
    delay = backoff_delay(attempt)
    enqueue_delivery(delivery, delay_seconds=delay, attempt=attempt + 1)

Expected result: every delivery eventually resolves — success, permanent-failure DLQ, or exhausted-retries DLQ. Nothing loops forever.

Step 5 — Respect Retry-After when the receiver sends it

A well-behaved receiver returning 429 or 503 will include a Retry-After header telling you exactly when to try again. Overriding it with your own backoff is rude and counterproductive — you'll get rate-limited again.

def compute_delay(attempt, response):
    if response is not None:
        retry_after = response.headers.get("Retry-After")
        if retry_after:
            try:
                # RFC 9110 specifies delta-seconds as a non-negative integer,
                # but some servers emit floats, so parse defensively.
                return float(retry_after)
            except ValueError:
                # HTTP-date format — parse to seconds-from-now
                return parse_http_date_to_seconds(retry_after)
    return backoff_delay(attempt)

If Retry-After is longer than your calculated backoff, honour it. If it's shorter, still honour it — the receiver knows something you don't. Cap the maximum you'll accept (say, 1 hour) so a misconfigured header can't park a delivery indefinitely.

Step 6 — Persist attempt state across restarts

Backoff logic in memory dies with the worker process. When your dispatcher restarts mid-outage, every in-flight retry resets to attempt 0 and the schedule collapses back to t+1s for everything at once. That's the thundering herd you just designed jitter to prevent.

Store attempt count and next-scheduled-time in durable storage — the same row that holds the delivery payload. Every worker instance reads from that table (or queue with visibility timeouts) and the schedule survives deploys, crashes, and autoscaling events.

CREATE TABLE webhook_deliveries (
    id UUID PRIMARY KEY,
    endpoint_url TEXT NOT NULL,
    payload JSONB NOT NULL,
    attempt INTEGER NOT NULL DEFAULT 0,
    next_attempt_at TIMESTAMPTZ NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    last_response_status INTEGER,
    last_error TEXT
);

CREATE INDEX ON webhook_deliveries (next_attempt_at) WHERE status = 'pending';

Expected result: a worker restart is a no-op for retry state. The next poll picks up whatever's due, at whatever attempt it was on.

Step 7 — Instrument the whole schedule

A webhook retry policy you can't observe is a webhook retry policy you can't tune. Emit metrics per delivery, per attempt, and per endpoint:

• delivery_attempts_total (counter, labelled by endpoint and outcome)
• delivery_retry_delay_seconds (histogram)
• delivery_final_status (counter: succeeded, permanent_failure, exhausted_retries)
• delivery_time_to_success_seconds (histogram, first-attempt-to-2xx)

Alert on two things: DLQ depth crossing a threshold (something's broken on a receiver that used to work), and time-to-success p99 climbing (retries are working but the schedule is too patient — or too aggressive).

Pair this with the circuit breaker pattern at the endpoint level. If a specific consumer has been failing for 20 minutes straight, stop retrying every delivery to that URL and start short-circuiting new ones straight to DLQ until the breaker half-opens. Backoff protects the receiver from you; the breaker protects you from the receiver.

Common pitfalls

Retrying non-idempotent operations. If the receiver processes the payload but the response gets lost, your retry causes a duplicate. Every webhook needs an idempotency key the receiver can dedupe on — usually the event ID in the payload. See our guide on API idempotency for safe retries for the mechanics.

Confusing Retry-After seconds with milliseconds. The header is seconds. Sending your first retry 30 minutes later because someone read 1800 as milliseconds is a debugging session nobody wants.

Sharing one queue across all endpoints. A single slow consumer can back up deliveries for every other consumer. Partition by endpoint or tenant so a bad receiver only hurts itself.

Retrying 4xx blindly. 400, 410, 422 are the receiver telling you the request is wrong. No amount of backoff fixes wrong. Route these to DLQ on the first response. (404 is a judgement call — for webhooks specifically it can be transient during a receiver deploy or load-balancer reconfiguration, so some teams do give it a limited number of retries.)

Forgetting that agents are now the consumer. Webhook receivers used to be someone's Rails endpoint. Increasingly they're feeding event-driven agent workflows where a duplicate delivery kicks off duplicate tool calls. The dedupe key doesn't matter less in an agent world — it matters more.

Testing only the happy path. Fault-inject 500s, timeouts, connection resets, and slow responses. Confirm the schedule matches what you expect, jitter is spreading the load, and the DLQ catches what it should.