Exponential Backoff & Jitter for Spatial Webhooks

Exponential backoff with decorrelated jitter is the correct retry timing for redelivering failed spatial webhook events: it spreads retry attempts across time so a recovering receiver is not hit by a synchronised thundering herd, and — paired with an idempotency key derived from the geometry — it guarantees a retried write never double-inserts a feature.

This topic sits under Queue Management, Retries & Delivery Guarantees, the section covering how spatial event pipelines move payloads reliably from producer to consumer. Retry timing is the first line of defence against transient failure; when retries are exhausted, events fall through to Dead-Letter Queues for Spatial Payloads, and the overall correctness contract is set by Delivery Guarantees & Event Ordering.

Prerequisites

Confirm your stack meets this baseline before wiring retries into a spatial webhook dispatcher. Check off each item as you verify it:

• Python 3.11+ — asyncio.timeout() and tomllib simplify per-attempt deadlines and config loading
• aiohttp 3.9+ — the async HTTP client used to POST GeoJSON, with fine-grained ClientTimeout control
• shapely 2.0+ — to validate geometry topology so invalid-geometry failures are classified as terminal, not retryable
• redis-py 5.0+ (redis.asyncio) — for the idempotency claim that makes each retry safe, and for a shared retry-budget token bucket
• A retry-window SLA from every downstream provider — you cannot size backoff ceilings or TTLs without knowing how long the receiver stays down
• A dead-letter sink — a place to send events once the retry budget is exhausted, so nothing is silently dropped

Why Constant and Linear Retry Fail at Scale

A single client retrying a single failed delivery at a fixed one-second interval is harmless. The danger is correlation. When a shared downstream — a tile server, a PostGIS writer, a routing engine — has a brief outage, every producer that was mid-delivery fails at nearly the same instant. If they all retry on the same fixed schedule, they retry together, in a synchronised wave. The receiver comes back up, is immediately hit by the entire backlog at once, tips over again, and the cycle repeats. This is the classic thundering-herd retry storm, and constant or linear backoff actively creates it because it preserves the alignment of the original failure.

Spatial workloads make this worse in three specific ways. First, individual requests are expensive: a webhook carrying a MultiPolygon triggers geometry validation, a spatial join, and often a tile invalidation, so a retry storm consumes far more receiver CPU per request than a plain JSON ping. Second, payloads are large — a dense boundary geometry is orders of magnitude bigger than a status event — so synchronised retries saturate bandwidth as well as CPU. Third, spatial events are frequently bursty by nature: a fleet of sensors crossing a region boundary emits correlated events, so the population that fails together is already clustered.

Exponential backoff breaks the synchronisation over time by making each successive delay grow multiplicatively (base · 2^attempt), which thins the retry rate as an outage lengthens. But pure exponential backoff alone does not break synchronisation between clients: a thousand producers that all failed at t=0 and all use identical exponential delays still retry in lockstep, just at exponentially spaced instants. The fix is jitter — randomising each delay so the herd disperses across the interval instead of stacking on its edge.

Architecture Blueprint

The retry path is a small state machine wrapped around a single HTTP POST. Every failure is classified before a delay is ever computed, every retry re-claims an idempotency key, and exhaustion has exactly one destination.

Layer breakdown:

1. Dispatch — the payload is POSTed as GeoJSON under a stable idempotency key. The key is derived once and reused for every retry of the same event, so the receiver can recognise a redelivery.
2. Classification — the outcome is sorted into three buckets before any timing logic runs: success acknowledges and exits; a terminal fault (4xx or invalid geometry) goes straight to the dead-letter queue; a retryable fault (5xx, timeout, 429) proceeds to timing.
3. Budget gate — a shared token bucket decides whether the fleet can afford this retry. If the retry-to-success ratio is already too high, the event is shed to the dead-letter queue instead of amplifying an outage.
4. Jittered delay — a decorrelated-jitter delay, capped at a ceiling, is slept before looping back to dispatch for the next attempt.

Step-by-Step Implementation

Step 1 — Classify the Failure Before Scheduling Anything

Timing logic must never run for a failure that will never succeed. A 422 for a self-intersecting polygon is deterministic: it fails identically on attempt one and attempt fifty. Retrying it burns the budget and delays the moment a human sees it in the dead-letter queue.

from dataclasses import dataclass
from enum import Enum, auto
import aiohttp


class Outcome(Enum):
    SUCCESS = auto()
    RETRYABLE = auto()   # transient: 5xx, timeout, connection reset, 429
    TERMINAL = auto()    # permanent: 4xx, invalid geometry — send to DLQ


# 4xx codes that must never be retried; a retry cannot change the result.
NON_RETRYABLE_STATUS = {400, 401, 403, 404, 409, 410, 422}
# 5xx and 429 are worth another attempt after a backoff delay.
RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def classify_response(status: int) -> Outcome:
    if 200 <= status < 300:
        return Outcome.SUCCESS
    if status in NON_RETRYABLE_STATUS:
        return Outcome.TERMINAL
    if status in RETRYABLE_STATUS or 500 <= status < 600:
        return Outcome.RETRYABLE
    # Unknown 4xx: treat as terminal so we fail fast rather than loop.
    return Outcome.TERMINAL


def classify_exception(exc: Exception) -> Outcome:
    # Transport faults are transient by nature and safe to retry.
    if isinstance(exc, (aiohttp.ClientConnectionError,
                        aiohttp.ServerTimeoutError,
                        TimeoutError)):
        return Outcome.RETRYABLE
    return Outcome.TERMINAL

Validate geometry topology before the POST so that an invalid ring is classified as terminal locally, never sent over the wire only to bounce back as a 422. The topology check is the same shapely gate used across the pipeline; the CRS must carry its EPSG code (payloads default to EPSG:4326 (WGS84) per RFC 7946).

from shapely.geometry import shape
from shapely.validation import explain_validity


def is_terminal_geometry(feature: dict) -> str | None:
    """Return a reason string if the geometry can never succeed, else None."""
    geom = shape(feature["geometry"])
    if geom.is_empty:
        return "empty geometry"
    if not geom.is_valid:
        return f"invalid topology: {explain_validity(geom)}"
    return None

Step 2 — Compute the Delay with Decorrelated Jitter

The three jitter strategies differ in how they draw the random delay for attempt n, given a base, a cap, and (for decorrelated) the previous delay:

• Full jitter — delay = uniform(0, min(cap, base · 2^n)). Widest spread, lowest collision probability, but can retry almost immediately after a failure.
• Equal jitter — half = min(cap, base · 2^n) / 2; delay = half + uniform(0, half). Guarantees a minimum wait, so no retry lands too early, at the cost of a narrower spread.
• Decorrelated jitter — delay = min(cap, uniform(base, prev · 3)). The delay random-walks upward from the previous value, combining a rising floor with a wide ceiling. This is the recommended default.

import random


def full_jitter(attempt: int, base: float, cap: float) -> float:
    return random.uniform(0, min(cap, base * (2 ** attempt)))


def equal_jitter(attempt: int, base: float, cap: float) -> float:
    ceiling = min(cap, base * (2 ** attempt))
    half = ceiling / 2
    return half + random.uniform(0, half)


def decorrelated_jitter(prev_delay: float, base: float, cap: float) -> float:
    """Random-walk the delay upward; independent of an attempt counter."""
    return min(cap, random.uniform(base, prev_delay * 3))

The SVG below compares the delay envelopes the three strategies produce across attempts. All grow, but the shaded spread — the range a given attempt can land in — is what disperses the herd.

Step 3 — An Async Retry Decorator Around an aiohttp POST

The decorator wraps any coroutine that performs one delivery attempt. It uses decorrelated jitter, classifies every outcome, respects a Retry-After header when present, and raises a dedicated exception when the ladder is exhausted so the caller can route to a dead-letter queue.

import asyncio
import functools
import logging
import random
from typing import Awaitable, Callable

logger = logging.getLogger(__name__)


class RetriesExhausted(Exception):
    """All attempts consumed; the caller should dead-letter the event."""


class TerminalFailure(Exception):
    """Non-retryable outcome; fail fast and dead-letter immediately."""


def retry_decorrelated(
    max_attempts: int = 6,
    base: float = 0.5,     # seconds; first-attempt floor
    cap: float = 30.0,     # seconds; ceiling for any single delay
):
    def wrapper(fn: Callable[..., Awaitable]):
        @functools.wraps(fn)
        async def inner(*args, **kwargs):
            prev = base
            for attempt in range(1, max_attempts + 1):
                try:
                    return await fn(*args, **kwargs)
                except TerminalFailure:
                    raise  # never retry a permanent fault
                except Exception as exc:
                    if attempt == max_attempts:
                        raise RetriesExhausted(str(exc)) from exc
                    # Decorrelated jitter: walk upward from the last delay.
                    delay = min(cap, random.uniform(base, prev * 3))
                    prev = delay
                    logger.warning(
                        "attempt %d/%d failed (%s); sleeping %.2fs",
                        attempt, max_attempts, exc, delay,
                    )
                    await asyncio.sleep(delay)
        return inner
    return wrapper


@retry_decorrelated(max_attempts=6, base=0.5, cap=30.0)
async def deliver_geojson(
    session: aiohttp.ClientSession,
    url: str,
    feature: dict,
    idempotency_key: str,
) -> None:
    """Single delivery attempt. Raises to signal the decorator to retry."""
    reason = is_terminal_geometry(feature)
    if reason is not None:
        raise TerminalFailure(f"unshippable geometry: {reason}")

    headers = {
        "Content-Type": "application/geo+json",
        # The same key on every retry lets the receiver dedupe redeliveries.
        "Idempotency-Key": idempotency_key,
    }
    timeout = aiohttp.ClientTimeout(total=10)
    try:
        async with session.post(url, json=feature, headers=headers,
                                timeout=timeout) as resp:
            outcome = classify_response(resp.status)
            if outcome is Outcome.SUCCESS:
                return
            if outcome is Outcome.TERMINAL:
                raise TerminalFailure(f"HTTP {resp.status}")
            # Retryable: honour Retry-After if the server sent one.
            retry_after = resp.headers.get("Retry-After")
            if retry_after and retry_after.isdigit():
                await asyncio.sleep(min(float(retry_after), 30.0))
            raise RuntimeError(f"retryable HTTP {resp.status}")
    except (aiohttp.ClientConnectionError, aiohttp.ServerTimeoutError,
            TimeoutError) as exc:
        raise RuntimeError(f"transport fault: {exc}") from exc

Step 4 — Pair Retries with Idempotency So a Geometry Write Cannot Double-Insert

This is the non-negotiable rule of retries: a retry fires exactly when the client cannot know whether the first attempt succeeded. A read timeout can occur after the receiver has already inserted the polygon and committed the transaction — the write landed, but the acknowledgement never made it back. If the retry re-POSTs blindly, the receiver inserts the feature a second time: duplicate geometry, double-counted area, a corrupted spatial join downstream.

The idempotency key sent in Step 3 is what makes the retry a no-op instead of a duplicate. On the receiver side, that key is derived deterministically from the normalised geometry and claimed atomically before the write, exactly as described in Cache-Backed Idempotency Checks within the broader discipline of Idempotency & Spatial Deduplication. Retries and idempotency are two halves of one mechanism: retries provide liveness (the event eventually lands), idempotency provides safety (it lands at most once).

import hashlib
import json


def idempotency_key_for(feature: dict, device_id: str, event_type: str,
                        precision: int = 7) -> str:
    """
    Deterministic key over the rounded geometry plus a business identifier.
    The SAME event always yields the SAME key, so a retried POST is recognised
    as a redelivery on the receiver and does not double-insert. Coordinates are
    rounded (~1.1 cm at the equator, EPSG:4326) to absorb float drift.
    """
    def _round(coords):
        if coords and isinstance(coords[0], (int, float)):
            return [round(c, precision) for c in coords]
        return [_round(part) for part in coords]

    canonical = {
        "type": feature["geometry"]["type"],
        "coordinates": _round(feature["geometry"]["coordinates"]),
    }
    blob = json.dumps(canonical, sort_keys=True, separators=(",", ":"))
    composite = f"{device_id}:{event_type}:{blob}"
    return "idem:v1:" + hashlib.sha256(composite.encode()).hexdigest()

Step 5 — Enforce a Retry Budget Across the Fleet

Per-event backoff disperses a herd in time, but it does not cap total retry load. During a long outage, every event still climbs its full ladder, and a fleet of thousands multiplies that into a sustained flood aimed at an already-failing receiver. A retry budget bounds retries as a fraction of successful traffic — a token bucket refilled from successes and drained by retries. When the bucket is empty, retries are shed straight to the dead-letter queue.

import time


class RetryBudget:
    """
    Token-bucket retry budget shared across the dispatcher. Every success adds
    `ratio` tokens (e.g. 0.1 → retries may add at most ~10% extra load); every
    retry costs 1 token. When empty, retries are refused and the event is
    dead-lettered instead of amplifying a downstream outage.
    """
    def __init__(self, ratio: float = 0.1, ceiling: float = 100.0,
                 min_per_sec: float = 1.0):
        self.ratio = ratio
        self.ceiling = ceiling
        self.min_per_sec = min_per_sec
        self.tokens = ceiling
        self._last = time.monotonic()

    def _leak(self) -> None:
        now = time.monotonic()
        self.tokens = min(self.ceiling,
                          self.tokens + self.min_per_sec * (now - self._last))
        self._last = now

    def record_success(self) -> None:
        self._leak()
        self.tokens = min(self.ceiling, self.tokens + self.ratio)

    def try_retry(self) -> bool:
        self._leak()
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False  # budget exhausted → caller must dead-letter

Sizing the ratio and the ceiling against each provider’s published retry window and error-rate SLA is its own discipline, covered in Tuning Retry Budgets for Webhook Provider SLAs.

Delivery Guarantees: What Retries Actually Buy You

Retries move a pipeline from best-effort toward at-least-once delivery — with a bounded budget, every event is either acknowledged, dead-lettered, or shed, and none is silently lost. At-least-once is the honest ceiling for a network with timeouts: the client can always be forced to retry after a lost acknowledgement, so duplicates are unavoidable at the transport layer. What upgrades the observable behaviour to effectively-once is the idempotency key from Step 4 — duplicates still arrive, but the receiver collapses them, so a feature is written at most once regardless of how many times it is delivered.

True exactly-once across a network boundary requires a distributed transaction coordinator and is rarely justified for spatial ingestion given its latency cost. The pragmatic contract for almost every spatial webhook fleet is therefore at-least-once delivery plus idempotent writes. Ordering is a separate axis — retries reorder events by construction, since a retried event lands after later events that succeeded first — and reconciling that with spatial correctness is the subject of Delivery Guarantees & Event Ordering. When the retry ladder and the budget are both exhausted, the event must land somewhere durable and inspectable: Dead-Letter Queues for Spatial Payloads.

Verification

A retry policy is only correct if its delay sequence stays inside the bounds you designed. This test asserts that decorrelated jitter never drops below the base floor, never exceeds the cap, and — over many trials — actually disperses rather than collapsing onto a single value.

import statistics


def decorrelated_sequence(attempts: int, base: float, cap: float,
                          rng_seed: int) -> list[float]:
    rnd = random.Random(rng_seed)
    prev, out = base, []
    for _ in range(attempts):
        delay = min(cap, rnd.uniform(base, prev * 3))
        out.append(delay)
        prev = delay
    return out


def test_backoff_sequence_bounds():
    base, cap = 0.5, 30.0
    for seed in range(500):
        seq = decorrelated_sequence(8, base, cap, seed)
        # Bound 1: no delay is below the base floor.
        assert min(seq) >= base - 1e-9, f"delay below base: {min(seq)}"
        # Bound 2: no delay exceeds the cap.
        assert max(seq) <= cap + 1e-9, f"delay above cap: {max(seq)}"

    # Bound 3: across seeds, attempt-1 delays are dispersed, not constant.
    first_delays = [decorrelated_sequence(1, base, cap, s)[0]
                    for s in range(500)]
    assert statistics.pstdev(first_delays) > 0.1, "jitter is not dispersing"
    print("backoff bounds hold; jitter disperses the herd")


if __name__ == "__main__":
    test_backoff_sequence_bounds()

Run it with pytest -q or directly with python. The third assertion is the one that catches a subtly broken jitter implementation: a delay function that grows correctly but forgets to randomise will pass the bound checks yet still cause a thundering herd, and only the dispersion assertion exposes it. A worked, benchmarked version of this harness — with side-by-side timing of all three variants — lives in Implementing Exponential Backoff with Jitter in Python.

Troubleshooting

FAQ

Related

• Queue Management, Retries & Delivery Guarantees — the parent section covering how spatial event pipelines move payloads reliably from producer to consumer
• Cache-Backed Idempotency Checks — the receiver-side deduplication that makes every retry safe to redeliver
• Dead-Letter Queues for Spatial Payloads — where events land once the retry ladder and budget are exhausted
• Delivery Guarantees & Event Ordering — reconciling at-least-once retries with the ordering constraints of spatial state
• Implementing Exponential Backoff with Jitter in Python — a self-contained, benchmarked implementation of all three jitter variants
• Tuning Retry Budgets for Webhook Provider SLAs — sizing budget ratios and ceilings against published provider retry windows

Explore this section