Tuning Retry Budgets for Webhook Provider SLAs

To size a webhook retry budget, read the provider’s documented retry window and per-attempt timeout, then choose a total window, a maximum attempt count, and a per-attempt timeout whose cumulative time fits inside that window — while setting your idempotency-key TTL strictly longer than the whole budget so no late retry ever replays an event whose key has expired. The budget is bounded above by the provider SLA and below by the need to give a struggling downstream enough attempts to recover; the idempotency TTL is the hard ceiling that keeps retries from turning into duplicate processing.

This page is part of Exponential Backoff & Jitter for Spatial Webhooks, which sits within the Queue Management, Retries & Delivery Guarantees architecture — the reference section for delivering spatial events reliably under failure.

When to use this pattern

Compute an explicit retry budget when:

• Your provider publishes a finite retry window (Stripe retries for ~3 days, GitHub for a fixed set of attempts, many GIS platforms document a 24–72 hour window) and you must decide how many consumer-side re-deliveries fit inside it.
• You deduplicate with an idempotency key stored in Redis with a TTL, and you need the TTL and the retry schedule to agree so a slow-recovering downstream never causes a replay after expiry.
• You run a dead-letter queue and need a precise, testable moment — budget exhaustion — at which an undeliverable spatial event is handed off rather than retried forever.

It is not the right tool when your provider guarantees indefinite retries with a stable event ID and you persist that ID directly — there the sender owns the budget and you only need at-most-once storage, not a computed schedule.

Why an unbudgeted retry loop breaks

A retry loop written as “back off exponentially until it works” has no relationship to the two clocks that actually govern correctness: the provider’s retry window and the idempotency-key TTL. When those three timelines drift apart, spatial pipelines fail in two directions.

Give up too early and you drop a delivery the provider still considered in-flight — the sender sees a 5xx, exhausts its budget against your prematurely-closed consumer, and a valid tile update or sensor event is lost. Retry too long and the more insidious failure appears: your idempotency key expires mid-retry. Because the key was the only record that this event had been seen, the next attempt looks brand-new. A Feature in EPSG:4326 (WGS84) gets written twice, a boundary-change event replays, and downstream spatial indexes diverge. The budget’s job is to make the three timelines nest correctly, as the diagram shows.

The invariant is simple: retry budget ≤ provider window ≤ idempotency TTL. The budget hands off to the dead-letter queue before the key expires, so every event ends in exactly one terminal state.

A worked example: a 72-hour provider window

Suppose your spatial webhook provider documents a 72-hour retry window and re-delivers failed events on its own schedule during that window. You are the consumer, and each of your internal delivery attempts to a downstream geometry service can itself fail. You want to:

1. Retry internally with exponential backoff and jitter so a downstream that is briefly down recovers without a thundering herd.
2. Keep your idempotency key alive in Redis for the entire window plus a margin — say 78 hours — so even the last retry finds a live key. The margin absorbs clock skew and queue lag, the same TTL-alignment reasoning covered in Cache-Backed Idempotency Checks.
3. Stop retrying and hand the event to a dead-letter queue once the budget is spent — well before the 78-hour TTL — so the payload is captured while its key still exists. That hand-off is the entry point to Dead-Letter Queues for Spatial Payloads.

The code below turns a budget into a concrete attempt schedule and proves the schedule fits.

Complete runnable implementation

This module is self-contained and uses only the Python standard library. Given a total budget in seconds it produces an exponential-backoff-with-jitter schedule, caps the attempt count so the cumulative delay plus per-attempt timeouts never exceeds the budget, and derives the idempotency TTL that must accompany it.

import random
from dataclasses import dataclass, field
from typing import List


@dataclass
class RetryBudget:
    """
    A retry budget sized against a webhook provider's documented SLA.

    All durations are in seconds. The core invariant the schedule guarantees:

        sum(delays) + max_attempts * per_attempt_timeout <= total_window

    and separately, callers must set:

        idempotency_ttl >= total_window   (enforced in __post_init__)

    so a late retry always lands on a live idempotency key.
    """
    total_window: float          # provider retry window, e.g. 72h = 259200s
    base_delay: float = 2.0      # first backoff delay, seconds
    factor: float = 2.0          # exponential growth factor
    max_delay: float = 3600.0    # cap on any single backoff (1h)
    per_attempt_timeout: float = 30.0   # request timeout per delivery attempt
    ttl_margin: float = 21600.0  # extra TTL over the window (6h) for skew/lag
    idempotency_ttl: float = field(init=False)

    def __post_init__(self) -> None:
        # The idempotency key must outlive the entire retry window.
        self.idempotency_ttl = self.total_window + self.ttl_margin

    def schedule(self, seed: int | None = None) -> List[float]:
        """
        Build the list of backoff delays (seconds) that fit inside the budget.

        Uses "full jitter": each delay is uniform in [0, capped_exponential].
        Stops adding attempts once the cumulative delay plus the timeout the
        NEXT attempt would consume would breach the total window.
        """
        rng = random.Random(seed)
        delays: List[float] = []
        elapsed = 0.0
        n = 0
        while True:
            capped = min(self.max_delay, self.base_delay * (self.factor ** n))
            delay = rng.uniform(0.0, capped)   # full jitter
            # Reserve room for this attempt's request timeout as well.
            projected = elapsed + delay + self.per_attempt_timeout
            if projected > self.total_window:
                break
            delays.append(delay)
            elapsed = projected
            n += 1
        return delays

    def max_attempts(self, seed: int | None = None) -> int:
        return len(self.schedule(seed))

    def total_scheduled_seconds(self, seed: int | None = None) -> float:
        """Cumulative wall-clock the schedule reserves: delays + timeouts."""
        delays = self.schedule(seed)
        return sum(delays) + len(delays) * self.per_attempt_timeout


if __name__ == "__main__":
    # 72h provider window -> 78h idempotency TTL.
    budget = RetryBudget(total_window=72 * 3600)
    delays = budget.schedule(seed=7)
    print(f"attempts fitted:      {len(delays)}")
    print(f"scheduled seconds:    {budget.total_scheduled_seconds(seed=7):,.0f}")
    print(f"total window:         {budget.total_window:,.0f}")
    print(f"idempotency TTL (s):  {budget.idempotency_ttl:,.0f}")
    # Set the Redis key with exactly this TTL when you first see the event:
    #   redis.set(idem_key, b"1", nx=True, px=int(budget.idempotency_ttl * 1000))
    assert budget.total_scheduled_seconds(seed=7) <= budget.total_window

The total_scheduled_seconds check is the load-bearing guarantee: it reserves not just the backoff delays but also one per_attempt_timeout per attempt, because a downstream that hangs until timeout consumes budget just as surely as a delay does. Feeding the derived idempotency_ttl straight into the Redis SET NX PX call keys the whole system off one number.

Parameter reference

Gotchas and spatial edge cases

1. Retry window exceeding the idempotency TTL causes re-processing. This is the headline failure. If the schedule can still fire an attempt after the Redis key has expired, that attempt reads no cached key, treats the delivery as new, and double-writes the spatial feature. Always derive the TTL from the window (total_window + ttl_margin) rather than hard-coding both independently, so the two can never drift apart during a config change.

2. Retry amplification across chained services. A single edge delivery that fans out to a CRS-normalization service and then a tile builder multiplies attempts: 5 × 5 × 5 = 125 downstream calls for one event. Budget the total end-to-end and divide it across hops — each inner hop must get a strictly smaller window than its caller, or a transient failure deep in the chain melts the pipeline.

3. Budget per-tenant, not globally. One tenant emitting invalid polygons (self-intersecting rings, unclosed geometries) can generate a wall of failures that exhausts a shared retry pool and starves every other tenant. Key the attempt counter and scheduled-time accounting on the tenant ID so a noisy neighbour cannot consume capacity reserved for others.

4. Jitter must not be applied to the TTL. Add jitter to backoff delays to de-correlate retries, but compute the idempotency TTL from the deterministic, un-jittered total_window. If jitter shortened the TTL, a worst-case (maximum-delay) retry schedule could outlive the key it depends on.

5. Per-attempt timeout counts against the budget. A downstream that hangs for the full per_attempt_timeout consumes budget identically to a backoff delay. Heavy EPSG:4326 → EPSG:3857 (Web Mercator) reprojection or large-geometry validation can push real request time toward the timeout, so reserve it explicitly (as the code does) rather than assuming attempts are instantaneous.

6. Provider windows are documented in wall-clock, your queue runs in processing time. If events sit in a broker backlog for hours before your consumer first sees them, the provider’s window has already partly elapsed. Anchor the budget to the provider’s original event timestamp, not to your first processing attempt, or a backlog can push retries past the real deadline.

Verification

Paste this into a test file and run with pytest. The critical assertion is that total scheduled time never exceeds the budget, and that the TTL always outlasts the window — the two invariants that keep retries from becoming duplicates.

import pytest
from your_module import RetryBudget  # adjust import path


def test_scheduled_time_never_exceeds_budget():
    """Cumulative delays + per-attempt timeouts must fit inside the window."""
    for seed in range(200):  # exercise many jitter draws
        budget = RetryBudget(total_window=72 * 3600)
        assert budget.total_scheduled_seconds(seed=seed) <= budget.total_window


def test_ttl_strictly_outlasts_window():
    """A late retry must still land on a live idempotency key."""
    budget = RetryBudget(total_window=72 * 3600)
    assert budget.idempotency_ttl > budget.total_window
    # The last possible attempt fires within the window, before TTL expiry.
    assert budget.total_scheduled_seconds() < budget.idempotency_ttl


def test_shorter_window_yields_fewer_attempts():
    """A tighter budget must not produce more attempts than a looser one."""
    small = RetryBudget(total_window=3600).max_attempts(seed=1)
    large = RetryBudget(total_window=72 * 3600).max_attempts(seed=1)
    assert small <= large


def test_chained_hop_budget_is_strictly_smaller():
    """Each downstream hop must get a strictly smaller window to avoid amplification."""
    edge = RetryBudget(total_window=72 * 3600)
    inner = RetryBudget(total_window=edge.total_window / 3)
    assert inner.total_window < edge.total_window
    assert inner.total_scheduled_seconds() <= inner.total_window

FAQ

Related

• Exponential Backoff & Jitter for Spatial Webhooks — parent: the backoff and jitter strategy the schedule in this page is built on
• Cache-Backed Idempotency Checks — aligning the Redis idempotency-key TTL with the retry window this budget derives
• Dead-Letter Queues for Spatial Payloads — where an event goes when the retry budget is exhausted before delivery succeeds