Dead-Letter Queues for Spatial Payloads

A spatial dead-letter queue is a durable holding area for geospatial events that failed processing, and it earns its place only when each record captures enough context — raw payload, failed stage, exact geometry error, CRS with EPSG code, idempotency key, and an offload pointer for oversized geometries — to make a later replay safe and exactly-once.

This topic is part of Queue Management, Retries & Delivery Guarantees, the discipline of moving geospatial webhook events through a broker without losing them, duplicating their side effects, or letting one poison payload stall an entire partition.

Prerequisites

Before wiring up a dead-letter path, confirm your stack meets the following baseline. Check off each item as you verify it:

Architecture Overview

A dead-letter queue is not a second inbox. It is a branch off the consumer that captures why an event could not be processed, in enough spatial detail that a human or an automated replay job can act on it later. The consumer attempts processing, and only when a failure is classified as terminal (or the retry budget is spent) does the event cross into the dead-letter branch with a fully populated envelope.

Spatial dead-letter branch off the consumer A broker delivers events to a consumer that runs parse, CRS transform, and topology validation stages. Transient failures loop back through retry with backoff; terminal failures branch down into a dead-letter envelope which writes to a durable store and offloads oversized geometry to object storage. A replay worker reads the store and re-injects events using the preserved idempotency key. Broker Kafka / Redis Consumer parse → CRS → topology → spatial join ack Processed DB · tiles transient → retry + backoff terminal / budget spent DL envelope stage · error · CRS idempotency key large geom Object store S3 blob + hash Dead-letter store DLT topic / PostGIS table Replay worker reuse idempotency key re-inject

Layer breakdown:

  1. Consumer with classified failure — each processing stage (parse, CRS transform, topology validation, spatial join) can fail. The consumer maps the raised exception to a failure class before deciding where the event goes.
  2. Retry branch (transient only) — broker timeouts, connection resets, and downstream 503s loop back through Exponential Backoff & Jitter for Spatial Webhooks. The dead-letter path is the destination those retries fall into once the budget is exhausted, not a substitute for them.
  3. Envelope construction and offload — terminal failures are wrapped in a dead-letter envelope that records the failed stage, the exact geometry error, the CRS with EPSG code, and the preserved idempotency key. Oversized geometries are written to object storage and replaced by a pointer.
  4. Durable dead-letter store — a Kafka *.DLT topic, a PostGIS table, or a Redis parked list holds the envelope until a replay worker or a human operator acts on it, reusing the idempotency key to keep the whole cycle exactly-once.

Step-by-Step Implementation

Step 1 — Classify the Failure

The single most important design decision in a dead-letter path is deciding what not to dead-letter automatically. Three failure classes behave completely differently on replay: a transient broker error will likely succeed on retry, an invalid geometry will never succeed until the data is corrected, and a schema violation is a contract breach that needs an upstream fix. Collapsing them into one queue produces either an infinite poison loop or a graveyard of retryable events nobody replays.

python 
from __future__ import annotations
from enum import StrEnum


class FailureClass(StrEnum):
    """How a failed spatial event should be treated downstream."""
    TRANSIENT = "transient"        # broker/network/downstream 5xx — retry, then DLQ
    INVALID_GEOMETRY = "invalid_geometry"  # deterministic — never auto-retry
    SCHEMA_VIOLATION = "schema_violation"  # contract breach — never auto-retry
    CRS_MISMATCH = "crs_mismatch"  # deterministic — needs reprojection or upstream fix
    UNKNOWN = "unknown"            # unclassified — DLQ and alert a human


class PipelineStage(StrEnum):
    PARSE = "parse"
    CRS_TRANSFORM = "crs_transform"
    TOPOLOGY_VALIDATION = "topology_validation"
    SPATIAL_JOIN = "spatial_join"
    PERSIST = "persist"


# Domain exceptions raised by each stage
class TransientBrokerError(Exception): ...
class GeometryValidityError(Exception): ...
class SchemaViolationError(Exception): ...
class CRSMismatchError(Exception):
    def __init__(self, message: str, epsg: int | None) -> None:
        super().__init__(message)
        self.epsg = epsg


def classify(exc: Exception) -> FailureClass:
    """Map a raised exception to a failure class. Order matters: check the
    deterministic, non-retryable classes before the catch-all."""
    match exc:
        case TransientBrokerError():
            return FailureClass.TRANSIENT
        case GeometryValidityError():
            return FailureClass.INVALID_GEOMETRY
        case CRSMismatchError():
            return FailureClass.CRS_MISMATCH
        case SchemaViolationError():
            return FailureClass.SCHEMA_VIOLATION
        case _:
            return FailureClass.UNKNOWN

Only TRANSIENT and UNKNOWN are candidates for automatic replay. The deterministic classes are parked for human or upstream correction. The upstream discipline that produces the GeometryValidityError in the first place is covered in Geometry Validation Pipelines; the dead-letter record simply preserves what that pipeline reported.

Step 2 — Define the Dead-Letter Envelope

The envelope is the contract between the failing consumer and the eventual replay. A normal DLQ record carries the payload and an error string; a spatial one carries the geometry-validity detail, the failed stage, the CRS with its EPSG code, the idempotency key, and — crucially — a GeometryRef that is either the inline geometry or a pointer to an offloaded blob.

python 
from datetime import datetime, timezone
from typing import Any, Self
from pydantic import BaseModel, Field, model_validator


class GeometryRef(BaseModel):
    """Either the geometry is inline, or it lives in object storage and we
    keep only a pointer. Exactly one of the two must be populated."""
    inline: dict[str, Any] | None = None          # GeoJSON per RFC 7946
    offload_uri: str | None = None                # e.g. s3://dlq-geom/2026/…
    byte_size: int = 0
    content_sha256: str | None = None             # integrity check for replay

    @model_validator(mode="after")
    def exactly_one_source(self) -> Self:
        if bool(self.inline) == bool(self.offload_uri):
            raise ValueError("set exactly one of inline / offload_uri")
        return self


class GeometryError(BaseModel):
    """The specific validity problem that failed the geometry."""
    reason: str                        # from shapely explain_validity()
    location: list[float] | None = None  # offending vertex [lon, lat]
    epsg: int | None = None            # populated on a CRS mismatch


class DeadLetterEnvelope(BaseModel):
    """Everything a replay job needs to act safely on a failed spatial event."""
    schema_version: str = "dlq.v1"
    event_id: str
    idempotency_key: str               # preserved verbatim from ingestion
    failure_class: FailureClass
    failed_stage: PipelineStage
    error_message: str
    geometry_error: GeometryError | None = None
    source_crs: str = "EPSG:4326"      # always carry the EPSG code
    geometry: GeometryRef
    raw_payload: dict[str, Any]        # untouched original, minus offloaded geom
    attempts: int = Field(ge=1)        # how many times it was tried before DLQ
    first_seen_at: datetime
    dead_lettered_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc)
    )

Keeping raw_payload untouched (aside from swapping the geometry for a pointer) matters: replay must reconstruct the exact bytes the consumer originally saw, or the idempotency key derived from it will no longer match.

Step 3 — Offload Oversized Geometries

A single continent-scale MultiPolygon or a densely sampled sensor track can exceed Kafka’s default 1 MB message limit and will be rejected outright — meaning your dead-letter write itself fails, and the event is lost. Offload anything above a threshold to object storage and store only the pointer.

python 
import hashlib
import json

OFFLOAD_THRESHOLD_BYTES = 256 * 1024  # keep DLQ messages small and scannable


def build_geometry_ref(geometry: dict, object_store, key_prefix: str) -> GeometryRef:
    """Inline small geometries; offload large ones to object storage and
    return a pointer with a content hash for later integrity verification."""
    encoded = json.dumps(geometry, sort_keys=True).encode("utf-8")
    size = len(encoded)
    digest = hashlib.sha256(encoded).hexdigest()

    if size <= OFFLOAD_THRESHOLD_BYTES:
        return GeometryRef(inline=geometry, byte_size=size, content_sha256=digest)

    # Large geometry: write the blob, keep only a pointer in the envelope
    object_key = f"{key_prefix}/{digest}.geojson"
    object_store.put_object(Key=object_key, Body=encoded)  # e.g. boto3 client
    return GeometryRef(
        offload_uri=f"s3://dlq-geom/{object_key}",
        byte_size=size,
        content_sha256=digest,
    )

Using the content hash as the object key deduplicates identical failing geometries automatically and gives replay a way to detect a truncated or corrupted blob before it is re-injected.

Step 4 — Route a Failed Event to the Dead-Letter Store

This is the function the consumer calls when a stage raises. It classifies the error, extracts the geometry-specific detail, builds the envelope, and publishes it to the correct destination — a retryable dead-letter for transient failures, a poison store for deterministic ones.

python 
import logging
from shapely.geometry import shape
from shapely.validation import explain_validity

logger = logging.getLogger(__name__)


def _extract_geometry_error(
    exc: Exception, raw_geometry: dict | None
) -> GeometryError | None:
    """Pull a precise, human-readable validity error out of the exception."""
    if isinstance(exc, CRSMismatchError):
        return GeometryError(reason=str(exc), epsg=exc.epsg)
    if isinstance(exc, GeometryValidityError) and raw_geometry is not None:
        geom = shape(raw_geometry)
        # explain_validity() returns e.g. "Self-intersection[12.5 41.9]"
        reason = explain_validity(geom)
        return GeometryError(reason=reason)
    return None


def route_to_dead_letter(
    *,
    raw_payload: dict,
    idempotency_key: str,
    failed_stage: PipelineStage,
    exc: Exception,
    attempts: int,
    first_seen_at: datetime,
    object_store,
    dlq_publisher,
) -> DeadLetterEnvelope:
    """Wrap a failed spatial event with full context and publish it."""
    failure_class = classify(exc)
    raw_geometry = raw_payload.get("geometry")

    geom_ref = build_geometry_ref(
        raw_geometry or {"type": "GeometryCollection", "geometries": []},
        object_store,
        key_prefix="2026/dlq",
    )

    # The raw payload we persist must not double-store a large offloaded geom
    persisted_payload = dict(raw_payload)
    if geom_ref.offload_uri is not None:
        persisted_payload["geometry"] = {"$ref": geom_ref.offload_uri}

    envelope = DeadLetterEnvelope(
        event_id=raw_payload.get("event_id", "unknown"),
        idempotency_key=idempotency_key,
        failure_class=failure_class,
        failed_stage=failed_stage,
        error_message=str(exc),
        geometry_error=_extract_geometry_error(exc, raw_geometry),
        source_crs=raw_payload.get("crs", "EPSG:4326"),
        geometry=geom_ref,
        raw_payload=persisted_payload,
        attempts=attempts,
        first_seen_at=first_seen_at,
    )

    # Deterministic failures never auto-retry; they go to a poison store + alert
    destination = (
        "spatial.events.retry.dlt"
        if failure_class in {FailureClass.TRANSIENT, FailureClass.UNKNOWN}
        else "spatial.events.poison.dlt"
    )
    dlq_publisher.publish(destination, envelope.model_dump_json())
    logger.warning(
        "dead-lettered event %s at stage=%s class=%s -> %s",
        envelope.event_id, failed_stage, failure_class, destination,
    )
    return envelope

Because the idempotency key is assigned at ingestion and carried through unchanged, the dead-letter record inherits the exactly-once guarantee described in Idempotency & Spatial Deduplication — replay does not have to reinvent it.

Spatial Validation and Error Handling

The envelope is only as trustworthy as the error detail it records. Two failure modes deserve explicit handling: a geometry that is structurally valid JSON but topologically broken, and a payload whose declared CRS does not match what the consumer expected.

python 
from pyproj import CRS


def validate_before_dead_letter(raw_geometry: dict, declared_crs: str,
                                expected_epsg: int = 4326) -> None:
    """Raise the precise, classifiable exception the DLQ envelope expects.
    Call this inside the consumer's topology/CRS stages."""
    # --- CRS check: a mismatch must surface the EPSG code, not a vague string
    try:
        crs = CRS.from_user_input(declared_crs)
    except Exception as exc:
        raise CRSMismatchError(f"unparseable CRS {declared_crs!r}: {exc}", epsg=None)
    if crs.to_epsg() != expected_epsg:
        raise CRSMismatchError(
            f"payload CRS is EPSG:{crs.to_epsg()} but pipeline expects "
            f"EPSG:{expected_epsg} (WGS84)",
            epsg=crs.to_epsg(),
        )

    # --- Topology check: NaN coords and self-intersections both fail here
    geom = shape(raw_geometry)
    if geom.is_empty:
        raise GeometryValidityError("geometry is empty")
    if not geom.is_valid:
        # explain_validity pinpoints e.g. "Self-intersection[12.500 41.900]"
        raise GeometryValidityError(explain_validity(geom))

A crs_mismatch such as a payload arriving in EPSG:3857 (Web Mercator) when the join layer is EPSG:4326 (WGS84) is deterministic: it will fail identically on every replay until the event is reprojected or the upstream provider is corrected. Recording the EPSG code in GeometryError.epsg lets a replay job decide whether it can auto-reproject or must escalate. Coordinates that arrive as NaN — common when a sensor drops a fix mid-stream — fail the topology check and are captured with the same precision.

Retry, Backoff, and Delivery Guarantees

A dead-letter queue is the terminus of the retry path, not a parallel one. Transient failures should be retried in place with exponential backoff and jitter; the event only crosses into the dead-letter store when the retry budget is spent. Conflating the two — dead-lettering on the first transient blip — floods the store with events that would have succeeded on the second attempt.

python 
import asyncio
import random


async def process_with_retry_then_dead_letter(
    *, raw_payload, idempotency_key, first_seen_at,
    consumer, object_store, dlq_publisher,
    max_attempts: int = 5, base_delay: float = 0.5,
) -> None:
    """Retry transient failures with full jitter; on budget exhaustion or a
    deterministic failure, hand the event to the dead-letter store."""
    for attempt in range(1, max_attempts + 1):
        try:
            await consumer.handle(raw_payload)  # raises a classified exception
            return
        except Exception as exc:
            failure_class = classify(exc)
            deterministic = failure_class not in {
                FailureClass.TRANSIENT, FailureClass.UNKNOWN
            }
            budget_spent = attempt == max_attempts

            if deterministic or budget_spent:
                route_to_dead_letter(
                    raw_payload=raw_payload,
                    idempotency_key=idempotency_key,
                    failed_stage=getattr(exc, "stage", PipelineStage.SPATIAL_JOIN),
                    exc=exc,
                    attempts=attempt,
                    first_seen_at=first_seen_at,
                    object_store=object_store,
                    dlq_publisher=dlq_publisher,
                )
                return

            # Transient with budget remaining: full jitter over [0, base·2^n]
            delay = random.uniform(0, base_delay * (2 ** attempt))
            await asyncio.sleep(delay)

How the guarantee holds: the consumer performs its atomic idempotency claim before any spatial side effect, so a replayed event whose work already completed short-circuits. Because the dead-letter envelope carries the original idempotency_key, replay is effectively-once end-to-end. The backoff-and-jitter mechanics that feed this path are detailed in Exponential Backoff & Jitter for Spatial Webhooks, and the safe-replay procedure itself is covered in Replaying Dead-Letter Spatial Events Safely.

Storage Backends

The dead-letter store choice trades queryability against operational simplicity.

Backend Strength Weakness Best for
Kafka *.DLT topic Same infra as the main stream; ordered; replay is a re-produce Poor ad-hoc querying; retention windows can drop old failures High-throughput pipelines already on Kafka
PostGIS table Query by geometry, EPSG code, or failure class; inspect the broken shape directly Another system to operate; write throughput lower than a broker Analyst-driven triage of invalid geometries
Redis Streams parked list Sub-millisecond writes; easy TTL Not durable enough as a system of record without AOF Short-lived transient failures awaiting fast replay
Object storage + index Cheap for very large geometries; effectively unbounded retention No native querying without a separate index Archival of oversized offloaded payloads

A PostGIS table is the most operator-friendly choice because you can SELECT failed events by failure_class, filter by source_crs, or even map the invalid geometry to see the self-intersection. That end-to-end pattern — a PostGIS dead-letter table drained by a Celery replay task — is walked through in Building a Spatial DLQ with PostGIS and Celery.

Verification

Confirm the envelope captures the right context and that offload round-trips cleanly. Run with pytest -v.

python 
import json
from datetime import datetime, timezone


class _FakeStore:
    """In-memory stand-in for an S3 client."""
    def __init__(self) -> None:
        self.objects: dict[str, bytes] = {}
    def put_object(self, Key: str, Body: bytes) -> None:
        self.objects[Key] = Body


class _FakePublisher:
    def __init__(self) -> None:
        self.sent: list[tuple[str, str]] = []
    def publish(self, destination: str, body: str) -> None:
        self.sent.append((destination, body))


BAD_GEOM = {  # bowtie polygon: self-intersecting
    "type": "Polygon",
    "coordinates": [[[0, 0], [1, 1], [1, 0], [0, 1], [0, 0]]],
}
PAYLOAD = {"event_id": "evt-9", "crs": "EPSG:4326", "geometry": BAD_GEOM}


def test_invalid_geometry_routes_to_poison_and_records_reason():
    pub, store = _FakePublisher(), _FakeStore()
    route_to_dead_letter(
        raw_payload=PAYLOAD,
        idempotency_key="idem:v1:abc",
        failed_stage=PipelineStage.TOPOLOGY_VALIDATION,
        exc=GeometryValidityError("Self-intersection[0.5 0.5]"),
        attempts=1,
        first_seen_at=datetime.now(timezone.utc),
        object_store=store,
        dlq_publisher=pub,
    )
    destination, body = pub.sent[0]
    env = json.loads(body)
    # Deterministic failure must NOT land in the retryable topic
    assert destination == "spatial.events.poison.dlt"
    assert env["failure_class"] == "invalid_geometry"
    assert env["failed_stage"] == "topology_validation"
    assert "Self-intersection" in env["geometry_error"]["reason"]
    # The idempotency key survives verbatim for safe replay
    assert env["idempotency_key"] == "idem:v1:abc"


def test_large_geometry_is_offloaded_not_inlined():
    pub, store = _FakePublisher(), _FakeStore()
    big = {"type": "MultiPoint",
           "coordinates": [[i * 1e-6, i * 1e-6] for i in range(60_000)]}
    ref = build_geometry_ref(big, store, key_prefix="test")
    assert ref.offload_uri is not None and ref.inline is None
    assert len(store.objects) == 1  # blob written exactly once
    # content hash lets replay verify integrity before re-injecting
    assert ref.content_sha256 is not None


def test_crs_mismatch_captures_epsg_code():
    err = GeometryError(reason="EPSG:3857 not EPSG:4326", epsg=3857)
    assert err.epsg == 3857

The first test is the load-bearing one: it proves a deterministic geometry failure is diverted away from the auto-retry topic and that the idempotency key is preserved for a later exactly-once replay.

Troubleshooting

Symptom Likely spatial cause Fix
Poison loop: same event dead-lettered thousands of times Invalid geometry routed to a retryable topic and auto-replayed Classify GeometryValidityError as deterministic; send it to the poison store, not the retry DLT
Dead-letter write itself fails with a message-size error Oversized MultiPolygon exceeds the broker’s 1 MB limit Offload geometries above the threshold to object storage; store only the pointer + hash
Replay reprocesses events that already succeeded Idempotency key regenerated at replay from a re-serialised payload Reuse the stored idempotency_key verbatim; never re-derive it during replay
Envelope error says “invalid geometry” but no location explain_validity result discarded in favour of str(exc) Store the full explain_validity string in geometry_error.reason
CRS failures can’t be auto-fixed on replay Mismatch recorded as a generic string without the EPSG code Populate GeometryError.epsg from CRS.to_epsg() so replay can decide to reproject
Offloaded blob is truncated on replay No integrity check between write and read Compare content_sha256 before re-injecting; discard and alert on mismatch
Transient failures flood the dead-letter store Dead-lettering on the first attempt instead of after the retry budget Only route to DLQ once attempts == max_attempts for transient classes

FAQ

What should a spatial dead-letter record capture beyond a normal DLQ entry?

Beyond the raw payload and error message that any dead-letter record holds, a spatial record must capture the pipeline stage that failed (parse, CRS transform, topology validation, or spatial join), the precise geometry-validity error including the offending vertex or ring, the CRS with its EPSG code when a mismatch caused the failure, the assigned idempotency key so replay stays exactly-once, and a size or offload pointer when the geometry is too large to store inline.

How do I stop a dead-letter queue from replaying the same event twice?

Carry the idempotency key assigned at ingestion into the dead-letter envelope and reuse it verbatim on replay. The consumer performs its normal atomic claim on that key before doing any spatial work, so a replayed event that was in fact already processed short-circuits harmlessly. Never regenerate the key at replay time, because a re-serialised payload can hash differently and defeat the guarantee.

Should invalid geometries and transient broker errors go to the same dead-letter queue?

Route them to separate destinations. Transient broker errors are worth retrying with exponential backoff and usually recover on their own, so they should only reach the dead-letter store after the retry budget is exhausted. Invalid geometry and schema violations are deterministic and will never succeed on replay without human or upstream correction, so they belong in a poison queue that triggers an alert rather than an automatic retry loop.

How do I dead-letter a 40 MB MultiPolygon without bloating the queue?

Offload the raw geometry to object storage such as S3 or a large-object column, and store only a pointer (bucket, key, byte size, and content hash) inside the dead-letter envelope. The broker message stays small and fast to scan, while the full payload remains recoverable for replay. Record the content hash so you can verify the offloaded blob was not truncated or corrupted before you re-inject it.

Explore this section

Building a Spatial DLQ with PostGIS and Celery

Read more

Replaying Dead-Letter Spatial Events Safely

Read more