Queue Management, Retries & Delivery Guarantees

Home → Queue Management, Retries & Delivery Guarantees

A geospatial webhook receiver that accepts an HTTP request, parses a GeoJSON Feature, and writes it straight to PostGIS looks correct in a demo and falls apart in production. The moment a downstream service is slow, a database connection pool is exhausted, or a burst of sensor telemetry arrives faster than geometries can be validated, synchronous processing turns backpressure into dropped events and 5xx responses. The webhook provider then retries, amplifying the load exactly when the system is least able to absorb it. Queue management is the discipline of decoupling acceptance from processing — durably buffering spatial events, retrying delivery intelligently, isolating the payloads that can never succeed, and reasoning precisely about which guarantees the pipeline actually offers. This guide is for platform engineers, GIS backend developers, and SaaS founders who have to keep a real-time spatial pipeline delivering under load.

This is the reliability and durability layer of the stack, and it is deliberately distinct from its neighbors. It is not about detecting duplicates — that is the job of Idempotency & Spatial Deduplication — nor about parsing and validating payloads, which belongs to Spatial Payload Routing & Parsing. It is about the mechanics of moving a spatial event from producer to consumer without losing it and without melting the broker in the process. Four concerns structure everything that follows: how to retry with Exponential Backoff & Jitter for Spatial Webhooks, where to send messages that will never succeed with Dead-Letter Queues for Spatial Payloads, how to pick and shard a queue in Broker Selection & Partitioning for Spatial Streams, and how to reason about ordering and semantics in Delivery Guarantees & Event Ordering. The architectural context that frames all of them lives in Core Event Fundamentals & Architecture.

Why Synchronous Delivery Breaks on Spatial Workloads

Non-spatial webhook queues are hard enough. Spatial workloads add three multipliers that make naive synchronous or best-effort delivery untenable.

Payloads are heavy and variable. A point-location ping is a few hundred bytes; a multipolygon land-cover change with tens of thousands of vertices is several megabytes. Processing time is dominated not by I/O but by CPU — coordinate transformation, topology repair, and validity checks. When a single message can take 40 ms of shapely work and another takes 2 ms, a synchronous handler blocks the event loop unpredictably, and tail latency balloons. The queue exists precisely to move that variable-cost work off the request path so acceptance stays fast and bounded.

Volume is bursty and geographically correlated. Sensor fleets, tile-update pipelines, and vehicle trackers do not emit a smooth stream. A weather front, a shift change, or a batch reprocessing job produces a spike concentrated in a region — which means the load is not just high, it is skewed toward a subset of partitions. A queue that spreads work evenly across workers is the only thing that keeps one hot region from starving the rest.

Failure is expensive and easy to get wrong. When delivery fails, the two naive responses are both wrong. Dropping the event silently corrupts the map state — a feature change is simply lost. Retrying immediately and forever hammers a struggling downstream and, without an idempotent consumer, multiplies a transient blip into duplicated geometry. The correct behavior sits in between: bounded, decorrelated retries for transient faults, and a durable side-channel for payloads that are structurally broken. Getting this wrong shows up as either data loss or a retry storm, and both are hard to diagnose after the fact.

The answer is an explicit delivery layer: producers publish to a durable broker, consumers pull work at their own pace, retries are bounded and randomized, and poison messages are quarantined with enough spatial context to be repaired and replayed.

Anatomy of a Durable Spatial Delivery Layer

The delivery layer has five moving parts arranged around a broker. A producer accepts the webhook and publishes; the broker durably holds the event on a partition; a consumer pulls and processes; failed processing enters a bounded retry loop; and messages that exhaust their budget are quarantined in a dead-letter queue. The diagram traces one spatial event through every path.

Each element is independently observable and independently scalable. The producer is stateless and fast because it only publishes. The broker owns durability and ordering per partition. The consumer pool scales horizontally to drain the queue. The retry loop and DLQ isolate failure so that one poison payload never blocks the shard behind it. The sections below build each element in Python.

Architectural Patterns for Reliable Delivery

Pattern 1 — Bounded Retry with Exponential Backoff and Jitter

The first line of defense against transient failure is a retry, and the single most important property of a production retry is that it is bounded and decorrelated. An unbounded retry is a denial-of-service attack on your own downstream; a fixed-interval retry across a fleet of workers produces synchronized thundering-herd spikes that re-crash a broker just as it recovers. The fix is exponential backoff — doubling the wait between attempts — combined with jitter, which randomizes each delay so that a hundred workers that failed at the same millisecond do not all retry at the same millisecond. The full derivation of jitter strategies and retry budgets is covered in Exponential Backoff & Jitter for Spatial Webhooks; the wrapper below is the reusable core.

import asyncio
import random
from typing import Awaitable, Callable, TypeVar

T = TypeVar("T")


class DeliveryExhausted(Exception):
    """Raised when every retry attempt fails; the caller routes to the DLQ."""


async def deliver_with_backoff(
    send: Callable[[], Awaitable[T]],
    *,
    max_attempts: int = 6,
    base_delay: float = 0.5,   # seconds
    max_delay: float = 30.0,
    jitter: str = "full",
) -> T:
    """
    Retry an async delivery callable with exponential backoff + jitter.

    'full' jitter (AWS-style) samples uniformly in [0, computed_backoff],
    which decorrelates retries across a fleet of workers all replaying the
    same spatial event after a shared broker or downstream outage.
    Only transient errors are retried; deterministic failures should be
    raised as a different exception type and skipped straight to the DLQ.
    """
    last_exc: Exception | None = None
    for attempt in range(max_attempts):
        try:
            return await send()
        except (ConnectionError, TimeoutError) as exc:
            last_exc = exc
            if attempt == max_attempts - 1:
                break
            ceiling = min(max_delay, base_delay * (2 ** attempt))
            if jitter == "full":
                delay = random.uniform(0, ceiling)
            elif jitter == "equal":
                delay = ceiling / 2 + random.uniform(0, ceiling / 2)
            else:  # no jitter — synchronized, avoid in a fleet
                delay = ceiling
            await asyncio.sleep(delay)
    raise DeliveryExhausted(f"failed after {max_attempts} attempts") from last_exc

The exception discipline is the subtle part. The wrapper retries only ConnectionError and TimeoutError — the transient class. A ValueError from an invalid geometry or an unknown CRS is deterministic: it will fail identically on every attempt, so it should never enter this loop. Retrying a structurally broken spatial payload wastes the entire budget and delays its arrival in the dead-letter queue, where a human or repair job can actually fix it.

Pattern 2 — Publish and Consume with Explicit Acknowledgment

The broker sits between a fast producer and a variable-speed consumer. The pattern that gives you at-least-once delivery is a consumer group with explicit acknowledgment: the consumer reads a batch, processes each entry, and acknowledges an entry only after the geometry is fully handled. If the worker crashes mid-batch, unacknowledged entries remain in the pending list and are redelivered to another consumer. The example uses Redis Streams under asyncio, keyed on an H3 cell so that events for a region share a shard — the partitioning rationale is developed in Broker Selection & Partitioning for Spatial Streams.

import json
from redis.asyncio import Redis
from shapely.geometry import shape
from shapely.validation import make_valid

redis = Redis(host="localhost", port=6379, decode_responses=True)
STREAM = "spatial.events"
GROUP = "geometry-workers"


async def publish_spatial_event(event: dict) -> str:
    """
    Publish a GeoJSON Feature to a Redis Stream. The H3 cell travels as a
    field so a downstream router (or a Kafka producer) can shard on it.
    Returns the entry ID — a monotonic '<ms>-<seq>' offset within the stream.
    """
    payload = {
        "h3_cell": event["h3_cell"],                  # partition key
        "device_id": event["device_id"],
        "geometry": json.dumps(event["geometry"]),    # GeoJSON, EPSG:4326
        "observed_at": event["observed_at"],
    }
    return await redis.xadd(STREAM, payload)


async def handle_geometry(fields: dict) -> None:
    """Validate and apply one event. Raises on a deterministic failure."""
    geom = make_valid(shape(json.loads(fields["geometry"])))
    if geom.is_empty or not geom.is_valid:
        raise ValueError("geometry invalid after repair")
    # ... apply an idempotent state mutation to PostGIS here ...


async def consume(consumer_name: str) -> None:
    # Create the group once; ignore the error if it already exists.
    try:
        await redis.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
    except Exception:
        pass
    while True:
        resp = await redis.xreadgroup(
            GROUP, consumer_name, {STREAM: ">"}, count=32, block=5000
        )
        for _stream, messages in resp or []:
            for entry_id, fields in messages:
                try:
                    await deliver_with_backoff(lambda f=fields: handle_geometry(f))
                    # XACK only after success — this is what makes it at-least-once
                    await redis.xack(STREAM, GROUP, entry_id)
                except (DeliveryExhausted, ValueError):
                    await route_to_dlq(entry_id, fields)
                    await redis.xack(STREAM, GROUP, entry_id)  # ack so PEL stays clean

The acknowledgment placement encodes the delivery guarantee. Acknowledge before processing and you get at-most-once — a crash loses the event. Acknowledge after processing and you get at-least-once — a crash redelivers it, which is exactly why the consumer must be idempotent. That idempotency contract, not the broker, is what ultimately protects against duplicate geometry, and it is the seam where this layer meets Idempotency & Spatial Deduplication.

Pattern 3 — Dead-Letter Routing that Preserves Geometry Context

A dead-letter queue is only useful if what lands in it can be repaired and replayed. For spatial payloads, that means the DLQ record must carry more than the raw bytes: it needs the declared CRS, the failure stage, the specific error, and any partition key already assigned, so an operator can transform, repair, or re-route the geometry without guessing. The design considerations for a durable spatial DLQ — including storage schema and safe replay — are covered in Dead-Letter Queues for Spatial Payloads.

import time


async def route_to_dlq(entry_id: str, fields: dict) -> None:
    """
    Quarantine a failed event with enough context to repair and replay it.
    The DLQ is a separate durable stream so its depth is monitored on its own.
    """
    record = {
        "original_entry_id": entry_id,
        "h3_cell": fields.get("h3_cell", ""),
        "device_id": fields.get("device_id", ""),
        "geometry": fields.get("geometry", ""),      # raw GeoJSON preserved verbatim
        "declared_crs": fields.get("crs", "EPSG:4326"),
        "failed_at": str(time.time()),
        "reason": "delivery_exhausted_or_invalid_geometry",
    }
    await redis.xadd("spatial.events.dlq", record)

Preserving the raw geometry verbatim is deliberate. A repair job replays the payload through the same normalization and validation path the live consumer uses, so the quarantined bytes must be byte-identical to what failed — never a re-serialized approximation that might mask or mutate the original defect.

Serialization and the Wire Format Tradeoff

The format a spatial event travels in directly shapes broker throughput, retention cost, and consumer parse latency. Serialization is not a free choice at high volume: a payload that is three times larger consumes three times the broker memory, network, and disk retention, and a format without schema enforcement pushes validation cost onto every consumer. The relevant axes for a delivery layer are on-wire size, decode speed at the consumer, and whether the broker can enforce a contract at the boundary.

For most stacks the pragmatic rule is: keep GeoJSON as the canonical internal representation the consumer normalizes to, but let the wire format on a high-frequency broker be Protobuf or Avro so the queue stays cheap and the contract is enforced at ingestion. The mapping between GeoJSON and a compact binary encoding is handled upstream in Spatial Payload Routing & Parsing, so the delivery layer only sees a decoded event.

Spatial-Specific Concerns in the Queue

Partitioning and Per-Key Ordering

Ordering is the concern that separates a spatial queue from a generic one. Brokers guarantee ordering only within a partition, never across the whole topic. That is fine — provided the events that must stay ordered relative to each other always land on the same partition. For spatial data the natural partition key is a discrete grid cell. Hashing on an H3 cell keeps every event for a region on one ordered shard, so a “feature created” always precedes the “feature updated” for the same parcel, while unrelated regions spread across the fleet for parallelism. Choosing the grid system and resolution is a real tradeoff between locality and skew, examined in H3 vs S2 vs Quadkey for Spatial Partitioning, and the ordering semantics that follow are detailed in Delivery Guarantees & Event Ordering.

import h3


def partition_key(lon: float, lat: float, resolution: int = 6) -> str:
    """
    Derive a partition key from an H3 cell so all events in a region share a
    shard and preserve per-region order. Coarse resolutions (5-7) balance
    locality against skew; too fine a resolution scatters related events,
    too coarse a resolution creates hot shards over dense metros.
    """
    return h3.latlng_to_cell(lat, lon, resolution)  # note: H3 takes (lat, lon)

The resolution choice is a load-balancing decision, not a precision one. Too coarse and a dense metro becomes a hot partition that one consumer cannot drain; too fine and events that should stay ordered scatter across shards. Resolution 5–7 is the usual starting band for routing, with skew watched as a first-class metric.

CRS and Geometry Validation Before the Queue Commits

A queue should never durably store a payload it cannot eventually process. Two spatial checks belong at the producer boundary, before the event is published. First, CRS: every geometry should be transformed to a single canonical projection — EPSG:4326 (WGS84) for global storage, or EPSG:3857 (Web Mercator) where tile alignment matters — so that consumers never have to reason about mixed projections mid-stream. Second, validity: an OGC Simple Features check catches self-intersecting rings, sub-four-point polygons, and NaN coordinates at the door. A payload that fails these is a deterministic failure — it goes straight to the DLQ, not the retry loop, because no amount of retrying repairs a broken ring. Catching it at the boundary lets the producer return a 422 Unprocessable Entity to the sender instead of durably queueing a message that is guaranteed to poison a consumer.

Production Hardening

Failure Modes and Delivery Guarantees

Choosing a guarantee. The honest target for a spatial webhook stack is at-least-once delivery from the broker paired with an idempotent consumer, which together approximate exactly-once effect without paying for distributed exactly-once transport. True end-to-end exactly-once requires transactional coupling between the broker and the datastore that few stacks can justify. At-least-once plus a deterministic idempotency key is cheaper, more robust, and degrades gracefully.

Poison-message loops. Without a retry cap and a DLQ, a single un-processable geometry is redelivered forever, permanently occupying a consumer slot and blocking the ordered messages behind it on its partition. A bounded retry budget plus dead-lettering is what breaks the loop.

Consumer crash mid-batch. Because acknowledgment follows processing, a crash leaves entries in the pending list, and a reclaim mechanism (Redis XAUTOCLAIM, or a Kafka rebalance) reassigns them. This is the mechanism that delivers at-least-once — and the reason the consumer must tolerate seeing the same event twice.

Backpressure and lag. When producers outrun consumers, lag grows on the hottest partitions first. The mitigation is horizontal consumer scaling bounded by the partition count — you cannot have more parallel consumers than partitions in a group — which is why partition count is a capacity decision made up front, not a knob turned later.

Geo-Specific Delivery Metrics

Standard queue dashboards track throughput and error rate. A spatial delivery layer needs a few more signals, and they belong in the broader picture drawn by Monitoring & Observability for Spatial Pipelines:

• Retry rate — retries per delivered event. A rising ratio is the earliest sign of a struggling downstream or an over-tight timeout, well before the DLQ starts filling.
• DLQ depth and arrival rate — the absolute count and inflow of quarantined messages. A sudden spike usually traces to one upstream firmware or schema change emitting invalid geometry, and the DLQ’s reason field pinpoints it.
• Per-partition / per-shard consumer lag — lag measured per shard, not just in aggregate. An aggregate that looks healthy can hide one saturated H3 partition over a hot region; per-shard lag is what surfaces partition skew.
• Geometry validation failure rate — the fraction of events rejected for invalid topology or CRS per window. Correlate it with producer deploys to catch a bad emitter early.
• Acknowledgment latency (P95/P99) — time from delivery to XACK, isolating slow geometry processing from broker or network delay so the right layer gets tuned.

Instrumenting these turns the delivery layer from a black box into a diagnosable system, and each metric maps to a concrete action — scale consumers, tune a timeout, or chase a bad producer — rather than a vague alert.

FAQ

Related

• Exponential Backoff & Jitter for Spatial Webhooks — retry budgets, jitter strategies, and provider-SLA-aware backoff tuning
• Dead-Letter Queues for Spatial Payloads — quarantine schemas that preserve geometry and CRS context, plus safe replay
• Broker Selection & Partitioning for Spatial Streams — Redis Streams vs Kafka vs RabbitMQ and H3-based topic partitioning
• Delivery Guarantees & Event Ordering — at-least-once vs exactly-once and per-key ordering for out-of-order spatial events
• Idempotency & Spatial Deduplication — the consumer-side contract that turns at-least-once delivery into exactly-once effect
• Monitoring & Observability for Spatial Pipelines — retry rate, DLQ depth, per-shard consumer lag, and geometry validation failure tracking

Explore this section