Idempotency & Spatial Deduplication

Home → Idempotency & Spatial Deduplication

Modern geospatial platforms increasingly rely on event-driven architectures to ingest real-time telemetry, IoT sensor payloads, and third-party webhook notifications. While this paradigm delivers horizontal scalability and service decoupling, it introduces a fundamental reliability challenge: duplicate event delivery. Webhook providers retry on HTTP timeouts, message brokers redeliver on consumer crashes, and network partitions produce ambiguous acknowledgments. In traditional CRUD systems, idempotency is typically solved by hashing request payloads and tracking processed keys. In geospatial systems, the problem compounds significantly. Two webhook payloads may differ in coordinate precision, projection metadata, or attribute ordering while representing the exact same geographic feature.

Idempotency and spatial deduplication together form the architectural discipline of guaranteeing that repeated or overlapping spatial events produce a single, deterministic state mutation. For platform engineers, GIS backend developers, and SaaS founders building real-time spatial applications, mastering this intersection is non-negotiable. It prevents phantom asset duplication, eliminates cascading billing errors, and ensures spatial analytics remain mathematically sound across distributed systems. The same event-driven foundations that govern this domain are covered in Core Event Fundamentals & Architecture, where delivery guarantees and broker selection are treated in depth. Every technique in this guide connects to one of four implementation concerns: Event Key Generation for Spatial Data, Cache-Backed Idempotency Checks, Spatial Overlap Deduplication, and Conflict Resolution Strategies.

Why Standard Idempotency Breaks on Spatial Data

Standard idempotency patterns assume byte-for-byte payload equivalence. Geospatial data routinely violates this assumption across three axes:

Coordinate noise. A GPS tracker might report [-122.4194, 37.7749] in one webhook and [-122.4194001, 37.7749002] in the next due to floating-point drift or hardware jitter. Two hashes, one real-world location.

CRS and serialization variance. A municipal GIS system might send the same polygon with vertices reordered, or with a different Coordinate Reference System (CRS) tag — for instance EPSG:4326 vs urn:ogc:def:crs:OGC:1.3:CRS84 — while the footprint is topologically identical. JSON key ordering differences further break naive MD5 or SHA-256 hashing. The strategies for normalizing these CRS discrepancies at ingestion time are detailed in CRS Normalization Strategies.

Topological equivalence without coordinate equality. Two geometries representing the same land parcel might be stored differently by separate upstream systems — one as a Polygon, another as a MultiPolygon with a single ring — yet they describe the same feature. No string comparison detects this.

When webhooks retry and deduplication fails, systems typically exhibit one of three failure modes:

1. Overprocessing — creates duplicate records, corrupting spatial joins, inflating storage, and triggering redundant downstream workflows.
2. Underprocessing — drops legitimate updates because a rigid hash mismatch prevents ingestion, leaving map states stale.
3. Topological conflicts — merges overlapping geometries incorrectly, producing self-intersections, sliver polygons, or silent data loss that poisons downstream analytics.

The solution requires a layered approach: deterministic key generation, stateful caching, spatial topology evaluation, and explicit conflict resolution.

Anatomy of a Spatial Idempotency Pipeline

A production-grade spatial webhook pipeline must separate ingestion, idempotency validation, spatial evaluation, and persistence into distinct, independently observable stages. The following diagram shows the data path from raw webhook arrival to committed state.

The labeled components map to four architectural layers: a normalization and fingerprinting step at ingestion, an atomic cache gate, a spatial database query for topology evaluation, and a conflict-aware persistence write. Each layer is independently testable and observable.

Architectural Patterns for Spatial Idempotency

Pattern 1 — Normalized Fingerprint + Atomic Cache Gate

This is the foundation pattern and handles the majority of real-world duplicates: retried webhooks delivering the same payload to the same endpoint within a short window.

Payload normalization is the step that makes hashing work on spatial data. Before computing any fingerprint, the system must:

• Round coordinates to a fixed decimal precision (6–8 decimals covers sub-metre accuracy for EPSG:4326).
• Transform all incoming geometries to a single canonical CRS — EPSG:4326 is the standard for global storage; EPSG:3857 is appropriate when pixel-level tile alignment matters.
• Sort JSON keys, strip null values, and normalize timestamps to UTC ISO 8601.
• Resolve geometry type aliases (Polygon vs single-ring MultiPolygon) to a canonical form.

After normalization, a composite key is formed by concatenating a business identifier (e.g., device_id, parcel_id) with a SHA-256 hash of the canonical geometry and attribute blob. This key is then written to Redis with SET NX EX, which atomically sets the key only if it does not already exist, with a TTL matching your webhook provider’s retry window (typically 24–72 hours). See Cache-Backed Idempotency Checks for the full Redis implementation pattern, including how to handle the key using GETSET for idempotency token responses.

import hashlib
import json
from redis.asyncio import Redis

redis = Redis(host="localhost", port=6379, decode_responses=True)

def build_idempotency_key(device_id: str, normalized_payload: dict) -> str:
    canonical = json.dumps(normalized_payload, sort_keys=True, separators=(",", ":"))
    fingerprint = hashlib.sha256(canonical.encode()).hexdigest()
    return f"idem:{device_id}:{fingerprint}"

async def admit_or_reject(key: str) -> bool:
    """Returns True if this event is new and should be processed."""
    # SET NX EX is a single atomic command — never replace with GET + SET
    admitted = await redis.set(key, "processing", nx=True, ex=172800)
    return bool(admitted)

The critical constraint is that EXISTS followed by SET is not atomic and must never be used. Between those two operations, a second worker thread processing the same webhook retry can slip through, causing a double-write.

Pattern 2 — Tolerance-Based Spatial Overlap Matching

Cache validation handles exact or near-exact duplicates efficiently, but it cannot detect semantically identical features with differing geometries due to measurement variance. A delivery route polygon submitted with slightly shifted vertices due to GPS sampling variance will produce a different fingerprint, bypassing the cache gate entirely.

At this stage, the pipeline queries PostGIS using tolerance-based spatial matching. ST_DWithin checks whether an incoming feature falls within a domain-calibrated distance of any existing stored feature, while ST_Equals or ST_Within verifies topological containment after the proximity filter. GiST indexes on geometry columns reduce these queries to logarithmic time across millions of records. The full implementation approach — including index creation DDL, the EXPLAIN ANALYZE patterns for verifying index usage, and how to combine ST_DWithin with ST_Equals in a single pass — is detailed in Spatial Overlap Deduplication.

Before any PostGIS query runs, geometries must pass the same OGC Simple Features validity checks described in Geometry Validation Pipelines. An invalid geometry — a self-intersecting ring, a coordinate sequence with NaN values — will cause PostGIS to raise an exception at constraint enforcement time.

Tolerance thresholds must be calibrated to your domain:

A tolerance that is too loose silently drops legitimate feature updates; one that is too tight allows duplicates to persist for high-jitter sensor streams.

import asyncpg
from shapely.geometry import shape
from shapely.validation import make_valid

async def find_spatial_duplicate(
    pool: asyncpg.Pool,
    geojson_geometry: dict,
    tolerance_degrees: float = 0.00009
) -> asyncpg.Record | None:
    """
    Returns the closest existing feature within tolerance, or None.
    Geometry is validated before the query to satisfy PostGIS topology rules.
    """
    geom = make_valid(shape(geojson_geometry))
    wkt = geom.wkt

    async with pool.acquire() as conn:
        return await conn.fetchrow("""
            SELECT id, updated_at, ST_AsGeoJSON(geom) AS geom_json
            FROM spatial_features
            WHERE ST_DWithin(
                geom,
                ST_GeomFromText($1, 4326),
                $2
            )
            ORDER BY ST_Distance(geom, ST_GeomFromText($1, 4326))
            LIMIT 1
        """, wkt, tolerance_degrees)

Calling make_valid before the query is not optional. Self-intersecting rings and improperly wound coordinate sequences will fail PostGIS geometry constraints, turning what should be a clean duplicate-detection query into a 500 error.

Pattern 3 — Confidence-Scored Conflict Resolution

When the spatial evaluator identifies an overlapping feature, the pipeline must decide what to do: discard the incoming event, replace the stored feature, merge attributes, or retain the higher-precision geometry regardless of timestamp. Simple systems default to last-write-wins, which is correct only when all producers share the same measurement quality. Production platforms require explicit strategies.

Implementing robust Conflict Resolution Strategies requires attaching a confidence score or data quality indicator to every event at ingestion time. Common scoring axes include:

• Positional accuracy class — RTK-GPS (< 2 cm), GNSS consumer grade (~3 m), cellular triangulation (~30–300 m).
• Geometry freshness — prefer the feature with the more recent observation_time, not ingestion_time, which can be skewed by queue delay.
• Source authority rank — authoritative cadastral survey data outranks sensor-derived approximations regardless of timestamp.

The conflict resolver compares the incoming score against the stored feature’s score and applies the higher-quality geometry, while merging non-conflicting attributes from both versions. An immutable append log records both the winning and losing versions for audit and replay.

from dataclasses import dataclass
from datetime import datetime

@dataclass
class SpatialFeatureVersion:
    feature_id: str
    geometry_wkt: str
    accuracy_metres: float   # lower is better
    observed_at: datetime
    source_rank: int          # lower is higher authority (1 = survey, 3 = sensor)

def resolve_conflict(
    stored: SpatialFeatureVersion,
    incoming: SpatialFeatureVersion
) -> SpatialFeatureVersion:
    """
    Returns the version that should become the canonical record.
    Prefers lower source_rank, then lower accuracy_metres, then newer observed_at.
    """
    if incoming.source_rank < stored.source_rank:
        return incoming
    if incoming.source_rank == stored.source_rank:
        if incoming.accuracy_metres < stored.accuracy_metres:
            return incoming
        if incoming.accuracy_metres == stored.accuracy_metres:
            if incoming.observed_at > stored.observed_at:
                return incoming
    return stored

Python Implementation: The Full Pipeline

Combining the three patterns into a coherent request handler requires careful sequencing. The normalization step must precede key generation; the cache check must precede the spatial query (cache is orders of magnitude cheaper than a PostGIS query); and the database upsert must be atomic with any post-commit cleanup of the Redis key on failure.

Payload Normalization

The normalization module is the entry point for all incoming spatial payloads. It must be deterministic across all worker processes, meaning no random UUIDs or timestamp-derived values may enter the canonical representation. For payloads already using Protobuf or MessagePack on the wire — as covered in GeoJSON-to-Protobuf Mapping — deserialize to a GeoJSON dict first so the normalization code path stays uniform across transport encodings.

from __future__ import annotations
import json
import hashlib
from typing import Any
import pyproj
from shapely.geometry import shape, mapping
from shapely.ops import transform
from shapely.validation import make_valid

_TRANSFORM_TO_WGS84: dict[str, pyproj.Transformer] = {}

def _get_transformer(source_epsg: int) -> pyproj.Transformer:
    key = str(source_epsg)
    if key not in _TRANSFORM_TO_WGS84:
        _TRANSFORM_TO_WGS84[key] = pyproj.Transformer.from_crs(
            source_epsg, 4326, always_xy=True
        )
    return _TRANSFORM_TO_WGS84[key]

def normalize_geometry(geojson: dict, source_epsg: int = 4326) -> dict:
    """
    Transforms geometry to EPSG:4326, validates topology, and rounds
    coordinates to 7 decimal places (~1.1 cm precision).
    """
    geom = make_valid(shape(geojson))
    if source_epsg != 4326:
        t = _get_transformer(source_epsg)
        geom = transform(t.transform, geom)
    # Round to 7 decimals for stable fingerprinting
    rounded = json.loads(
        json.dumps(mapping(geom),
        default=lambda x: round(x, 7) if isinstance(x, float) else x)
    )
    return rounded

def canonical_payload(event: dict[str, Any]) -> str:
    """
    Produces a stable, sorted JSON string for hashing.
    Strips null values and normalizes the geometry field.
    """
    clean = {k: v for k, v in event.items() if v is not None}
    if "geometry" in clean:
        clean["geometry"] = normalize_geometry(
            clean["geometry"],
            source_epsg=clean.get("crs_epsg", 4326)
        )
    return json.dumps(clean, sort_keys=True, separators=(",", ":"))

FastAPI Middleware Integration

The idempotency gate runs as FastAPI middleware, meaning it intercepts every POST request before it reaches any route handler. This placement avoids leaking duplicate-check logic into business logic and allows the gate to short-circuit with a 200 OK before the request body is parsed by application code. The async patterns for handling heavy geometry payloads — where parsing alone can take tens of milliseconds — are explored further in Async Processing for Heavy Geometries.

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import json

app = FastAPI()

@app.middleware("http")
async def spatial_idempotency_gate(request: Request, call_next):
    if request.method != "POST":
        return await call_next(request)

    raw = await request.body()
    try:
        event = json.loads(raw)
    except ValueError:
        return await call_next(request)

    device_id = event.get("device_id", "unknown")
    canonical = canonical_payload(event)
    key = f"idem:{device_id}:{hashlib.sha256(canonical.encode()).hexdigest()}"

    admitted = await redis.set(key, "processing", nx=True, ex=172800)
    if not admitted:
        return JSONResponse(
            status_code=200,
            content={"status": "duplicate_ignored", "idempotency_key": key}
        )

    return await call_next(request)

Spatial Serialization: JSON vs Protobuf vs MessagePack

For low-frequency administrative webhooks (< 100 events/second), GeoJSON over HTTP is the right default — it is human-readable, schema-flexible, and aligns with RFC 7946. At high frequency (> 1,000 events/second), serialization overhead becomes measurable:

For the normalization pipeline described above, GeoJSON remains the canonical internal format even when the wire format is Protobuf or MessagePack. Deserializing to GeoJSON before fingerprinting ensures a single normalization code path regardless of transport encoding.

Spatial-Specific Concerns

CRS Normalization Before Key Generation

CRS variance is the most common source of false duplicate misses in production. A payload tagged EPSG:32637 (UTM zone 37N, units: metres) and a second payload tagged EPSG:4326 (WGS 84, units: degrees) describing the same polygon will differ in both coordinate values and their JSON representation. Transforming all incoming geometries to a single canonical CRS — EPSG:4326 for global systems — before normalization closes this gap. The pyproj.Transformer instance should be cached per source EPSG as shown above; constructing a new Transformer on every request adds ~2 ms per call. Mixed-CRS event streams require the runtime detection approach described in Handling Mixed CRS Payloads in Python Event Handlers.

Spatial Indexing: H3, S2, and Quadkey for Composite Keys

Discrete global grid systems are powerful components of a spatial idempotency key for high-frequency sensor streams. Rather than hashing the full geometry, you can include the H3 cell (at resolution 9, ~0.1 km²) or S2 cell containing the feature centroid as part of the idempotency key. This bounds the spatial search space dramatically for the PostGIS overlap query: instead of ST_DWithin over the entire table, a pre-filter on h3_cell = $1 reduces the candidate set before the expensive geometry comparison.

import h3

def h3_component(lon: float, lat: float, resolution: int = 9) -> str:
    """Returns the H3 cell ID containing this coordinate at the given resolution."""
    return h3.latlng_to_cell(lat, lon, resolution)

def build_composite_key(device_id: str, lon: float, lat: float,
                         canonical_json: str) -> str:
    cell = h3_component(lon, lat, resolution=9)
    payload_hash = hashlib.sha256(canonical_json.encode()).hexdigest()[:16]
    return f"idem:{device_id}:{cell}:{payload_hash}"

H3 resolution 9 covers roughly 105 m × 105 m, which is appropriate for vehicle tracking. For precision agriculture or cadastral work, use resolution 11 (~25 m × 25 m) or resolution 13 (~4 m × 4 m).

Geometry Validation Before Dispatch

Every geometry entering the pipeline should pass OGC Simple Features validity checks before fingerprinting or database insertion. An invalid geometry — a self-intersecting ring, a polygon with fewer than four points, a coordinate sequence with NaN values — will cause PostGIS to raise an exception at constraint enforcement time. Catching invalidity at the ingestion boundary, before the idempotency key is committed to Redis, allows the system to return an informative 422 Unprocessable Entity to the webhook sender rather than silently eating the event or returning a misleading 500.

from shapely.validation import explain_validity

def validate_geojson_geometry(geojson: dict) -> tuple[bool, str]:
    """Returns (is_valid, reason). reason is empty string when valid."""
    try:
        geom = shape(geojson)
    except Exception as e:
        return False, f"shape() failed: {e}"
    if geom.is_empty:
        return False, "geometry is empty"
    if not geom.is_valid:
        return False, explain_validity(geom)
    return True, ""

Production Hardening

Failure Modes and Mitigations

Partial pipeline failure. If the PostGIS upsert succeeds but the Redis key is not transitioned from processing to committed, a subsequent retry will be rejected by the idempotency gate despite the fact that the event was never fully processed. Mitigate this by using a two-phase key state: write processing on admission, then overwrite with committed after the database commit confirms. On a rollback, explicitly delete the Redis key to allow the retry through.

Redis unavailability. If Redis is unreachable, the idempotency gate must fail open (admit the event) or fail closed (reject the event). The correct choice depends on your consistency requirements. Fail open is appropriate when duplicate writes are detectable and reconcilable downstream; fail closed is appropriate for billing or compliance events where a duplicate is more harmful than a dropped event.

Large geometry payloads. Multi-polygon features with tens of thousands of vertices can exceed the Redis string storage limit practically — more relevantly, storing a 2 MB WKT string in Redis wastes memory and bloats serialization time. Store large geometries in object storage (S3/MinIO/R2) and cache only the SHA-256 hash reference in Redis. The strategies for offloading heavy geometry processing without blocking the HTTP response are examined in Async Processing for Heavy Geometries.

Timezone and temporal drift. Systems that track temporal validity (valid_from, valid_to) must normalize all timestamps to UTC before they enter the canonical representation. A timestamp stored as 2025-03-15T14:00:00+05:30 and the same instant stored as 2025-03-15T08:30:00Z will produce different fingerprints.

Observability Metrics for Geo Workloads

Every webhook event should be logged with its idempotency key, spatial fingerprint, and resolution outcome. The following metrics are specific to spatial deduplication pipelines and should be tracked in addition to standard HTTP and queue metrics:

• Cache hit ratio — percentage of events rejected as duplicates at the Redis gate. A sudden spike indicates a webhook provider is stuck in a retry loop.
• Spatial overlap detection rate — percentage of cache-miss events that nonetheless match an existing stored feature via ST_DWithin. This rate should be low; a high rate indicates your coordinate normalization is insufficient.
• Conflict resolution distribution — breakdown of discard, merge, and replace outcomes. Unexpected spikes in discard may indicate a source system sending stale data.
• Geometry validation failure rate — count of events rejected for invalid topology per time window. Correlated with firmware updates on IoT devices.
• P95 latency per pipeline stage — Redis gate, PostGIS overlap query, and persistence write should each be instrumented independently to isolate bottlenecks.
• PostGIS GiST index scan ratio — confirm that the spatial evaluator is hitting the index, not performing sequential scans. A low ratio after schema migrations indicates a missing ANALYZE.

Dead-Letter Queue Design for Spatial Payloads

Events that fail geometry validation, exceed size limits, or trigger unhandled conflict outcomes must be routed to a dead-letter queue (DLQ) rather than silently dropped. The DLQ record must include the raw payload, the normalization stage at which the failure occurred, the specific error, and the idempotency key if one was assigned before the failure. This enables a human operator or automated repair job to correct the geometry and replay the event through the full pipeline using the same idempotency key, ensuring exactly-once semantics are preserved on replay. The relationship between retry strategy and spatial payload design is examined in Sensor Data Routing Patterns, where at-least-once vs exactly-once delivery tradeoffs for high-volume sensor streams are treated in depth.

FAQ

Related

• Event Key Generation for Spatial Data — deterministic fingerprinting of GeoJSON payloads for collision-resistant idempotency keys
• Cache-Backed Idempotency Checks — Redis patterns for atomic gate implementation and TTL management
• Spatial Overlap Deduplication — PostGIS tolerance-based matching and GiST index strategies
• Conflict Resolution Strategies — last-write-wins, confidence scoring, and immutable append log patterns
• Core Event Fundamentals & Architecture — delivery guarantees, event schema design, and broker selection for spatial workloads
• CRS Normalization Strategies — transforming mixed-CRS payloads to a canonical projection before fingerprinting
• Geometry Validation Pipelines — OGC topology checks and repair patterns that protect your PostGIS upsert from invalid geometries
• All geospatial webhook topics — return to the full architecture index for delivery, routing, and security sections

Explore this section