Monitoring & Observability for Spatial Pipelines

Home → Monitoring & Observability for Spatial Pipelines

Every other section of this site builds a spatial pipeline — receiving webhooks, deduplicating events, routing and parsing geometries, guaranteeing delivery. This section measures them. Observability is the discipline of being able to answer, from telemetry alone, why a running geospatial system is behaving the way it is: which geometries are failing validation, which geographic shard is lagging, whether your deduplication layer is actually catching duplicates, and where a single event spent its time as it crossed a fleet of async handlers. For platform engineers, GIS backend developers, and SaaS founders running real-time spatial systems, generic application monitoring is necessary but nowhere near sufficient. Request rate, p95 latency, and HTTP status codes describe the transport. They are blind to the shape of the data, and in spatial systems the shape of the data is where almost every interesting failure lives.

This is the observability layer for the architectures the rest of the site teaches, so it deliberately does not re-explain how to build them. It measures and diagnoses them. The material divides into three implementation concerns, each with its own dedicated guide: Geo-Specific Metrics & Instrumentation for the numbers only geo workloads produce, Consumer Lag & Partition Skew Monitoring for keeping sharded spatial streams balanced, and Structured Logging & Tracing for Spatial Events for reconstructing the path of a single event through async geometry handlers. Together they form the three planes of observability — metrics, logs, and traces — wrapped around a spatial pipeline.

Why Generic Observability Misses the Spatial Failures

A conventional monitoring stack watches a webhook receiver and reports that it is returning 200 OK in 12 milliseconds at 800 requests per second. Everything looks healthy. Meanwhile, a firmware update on a fleet of IoT sensors has started emitting self-intersecting polygons, 4 percent of incoming events are being quietly routed to a dead-letter queue, and a downstream map is going stale in three cities. None of that is visible in the transport metrics, because the transport is working perfectly. The failure is entirely in the geometry.

Spatial pipelines fail along axes that generic dashboards cannot see:

The failure is in the payload, not the protocol. A geometry can be perfectly valid JSON and a perfectly formed HTTP body while being an invalid polygon under the OGC Simple Features rules. HTTP-level monitoring sees a well-formed request; only geometry-aware instrumentation sees the topology error. You need a metric that increments specifically when shape() succeeds but is_valid returns false.

The distribution matters more than the average. A payload vertex-count histogram with a healthy median can hide a long tail of million-vertex multipolygons that each block an event loop for hundreds of milliseconds. An average latency number smears that tail into invisibility. Spatial workloads are defined by their outliers — the one enormous geometry, the one hot shard, the one CRS nobody expected — so percentiles and full distributions matter far more than means.

Aggregate throughput hides geographic imbalance. When you partition a stream by a spatial key — an H3 cell, a geohash prefix, a Quadkey — total throughput can look balanced while a single dense urban shard is drowning. This is partition skew, and it is unique to spatially partitioned streams because population density, not a hash function, decides how events distribute. A round-robin partitioner produces even load by construction; a spatial partitioner produces load shaped like a population map.

Correctness is invisible without a dedicated signal. Whether your idempotency layer is actually catching duplicates is not something you can infer from success rates. If spatial deduplication silently stops matching — because a producer changed coordinate precision, say — every duplicate now sails through and creates a phantom record, and the only symptom is a slow-rising storage bill. The dedup hit ratio has to be an explicit, watched metric or the regression is undetectable until it is a data-quality incident.

The remedy is to instrument the pipeline at the level of the data it carries. That means three coordinated planes: geo-aware metrics for aggregate health and alerting, structured logs that carry spatial context on every line, and distributed traces that follow one event across async boundaries. The rest of this guide covers how they fit together and how to build each in Python.

The Three Instrumentation Planes Around a Spatial Pipeline

Metrics, logs, and traces answer different questions and you need all three. Metrics answer “is the system healthy, and is it getting worse?” — they are cheap, aggregate, and drive alerts. Logs answer “what exactly happened to this event?” — they are detailed, per-event, and drive forensic investigation. Traces answer “where did this event spend its time, across every service and handler it touched?” — they connect causally-related work across the async boundaries that make spatial pipelines hard to reason about. The diagram below shows all three planes wrapped around the same pipeline, each tapping the stages it needs.

A practical implication of this layering is that each plane has a different cost profile and therefore a different sampling strategy. Metrics are always-on and aggregate, so you keep them all. Logs are per-event and can be voluminous, so you emit them structured but may sample the successful path and always keep failures. Traces are the most expensive per event, so you sample them — but you bias sampling toward the interesting: always trace an event that failed validation or hit a hot shard, sample the boring successes at a low rate. Getting this right is what keeps observability affordable at spatial-webhook volumes.

The Metrics That Only Matter for Geo Workloads

A generic service exposes request count, error count, and a latency histogram. A spatial pipeline needs those too, but its real health lives in a second set of metrics that describe the geometry flowing through it. The table below is the core catalogue — the metrics you should be exposing from a production geospatial webhook system, with the Prometheus metric type, why it earns its place, and guidance on where to set an alert. Full instrumentation for each lives in Geo-Specific Metrics & Instrumentation; the point here is the shape of the catalogue.

Two of these deserve pointers to their home sections. Consumer lag per geographic shard is where observability meets delivery: the pipeline that produces lag — brokers, retries, redelivery windows — is built in Queue Management, Retries & Delivery Guarantees, and this section watches the lag that pipeline generates. The dedup hit ratio measures the effectiveness of Idempotency & Spatial Deduplication; a healthy ratio is the proof that the deduplication layer described there is still doing its job in production.

A note that governs every row: keep label cardinality bounded. A Prometheus metric creates one time series per unique combination of label values, so labelling by raw coordinate, full-resolution H3 cell, or geometry hash will produce millions of series and take down your monitoring backend before it ever helps you. Label with the EPSG code, the geometry type, a coarse shard identifier, and a bounded outcome enum — dimensions with tens of values, not millions.

Instrumenting Geometry Validation Outcomes with Prometheus

The single most valuable geo-specific metric is validation outcome, because it converts an invisible data-quality problem into a graph you can alert on. Here is a self-contained instrumentation of the validation boundary using the Prometheus Python client. A Counter tracks outcomes labelled by result and geometry type; a Histogram records the vertex count so the heavy-geometry tail is visible. The dedicated walkthrough is Tracking Geometry Validation Failure Rate with Prometheus.

from prometheus_client import Counter, Histogram
from shapely.geometry import shape
from shapely.validation import explain_validity

# Low-cardinality labels only: outcome enum + geometry type.
# Never label with coordinates, hashes, or full-resolution cell IDs.
GEOM_VALIDATION = Counter(
    "geom_validation_total",
    "Geometry validation outcomes at the ingestion boundary",
    ["outcome", "geom_type"],  # outcome in {valid, invalid, unparseable, empty}
)

# Exponential buckets span the vertex range from trivial to event-loop-blocking.
GEOM_VERTICES = Histogram(
    "geom_vertex_count",
    "Distribution of vertices per incoming geometry",
    ["geom_type"],
    buckets=(4, 16, 64, 256, 1024, 4096, 16384, 65536),
)

def _vertex_count(geom) -> int:
    # Works for Polygon/MultiPolygon/LineString without materializing all coords twice.
    return sum(len(g.exterior.coords) if g.geom_type == "Polygon"
               else len(g.coords) for g in getattr(geom, "geoms", [geom]))

def validate_and_record(geojson: dict) -> bool:
    """Validate one geometry, emit metrics, and return whether it is admissible."""
    try:
        geom = shape(geojson)
    except Exception:
        GEOM_VALIDATION.labels(outcome="unparseable", geom_type="unknown").inc()
        return False

    gtype = geom.geom_type
    if geom.is_empty:
        GEOM_VALIDATION.labels(outcome="empty", geom_type=gtype).inc()
        return False

    GEOM_VERTICES.labels(geom_type=gtype).observe(_vertex_count(geom))

    if not geom.is_valid:
        # explain_validity() text (e.g. "Self-intersection") is a great log field,
        # but must NOT become a metric label — it is high-cardinality free text.
        GEOM_VALIDATION.labels(outcome="invalid", geom_type=gtype).inc()
        return False

    GEOM_VALIDATION.labels(outcome="valid", geom_type=gtype).inc()
    return True

The validation failure rate is then a PromQL expression over the counter — sum(rate(geom_validation_total{outcome="invalid"}[5m])) / sum(rate(geom_validation_total[5m])) — which you alert on as a ratio so the threshold holds regardless of traffic volume. The explain_validity() string, which tells you why a geometry is invalid, is deliberately kept out of the metric and pushed into the structured log instead, exactly because it is unbounded text.

Structured Logs That Carry Spatial Context

Metrics tell you the validation failure rate rose to 3 percent; they cannot tell you that the failures are all self-intersecting polygons from device fleet 7 inside one bounding box. Logs answer that, but only if they carry spatial context as structured, queryable fields. A log line that interpolates coordinates into a message string — "rejected geometry at 12.4,55.6" — forces a slow full-text scan and cannot be filtered by range. The same information as JSON fields lets you query “all invalid geometries where epsg != 4326 inside this bbox” directly in your log store.

The essential spatial fields to attach to every event log are the bounding box (a bounded, diffable summary of where the geometry is), the H3 cell (which shard it belongs to, enabling correlation with partition skew), the EPSG code (the projection it arrived in), and the geometry type and vertex count. The example below builds a structured logger and shows the fields in place. The complete pattern, including trace-ID correlation, is in Structured Logging & Tracing for Spatial Events.

import json
import logging
import h3
from shapely.geometry import shape

logger = logging.getLogger("spatial.pipeline")

def spatial_log_context(geojson: dict, epsg: int, device_id: str) -> dict:
    """Build a machine-parseable spatial context for one event."""
    geom = shape(geojson)
    minx, miny, maxx, maxy = geom.bounds
    centroid = geom.centroid
    return {
        "device_id": device_id,
        "epsg": epsg,                       # e.g. 4326 (WGS84) or 3857 (Web Mercator)
        "geom_type": geom.geom_type,
        # Round the bbox so identical footprints log identical values (stable, diffable).
        "bbox": [round(minx, 6), round(miny, 6), round(maxx, 6), round(maxy, 6)],
        # Coarse H3 cell (resolution 3) groups events into shards without exploding cardinality.
        "h3_cell": h3.latlng_to_cell(centroid.y, centroid.x, 3),
        "vertex_count": len(geom.exterior.coords) if geom.geom_type == "Polygon" else None,
    }

def log_validation_failure(geojson: dict, epsg: int, device_id: str, reason: str) -> None:
    ctx = spatial_log_context(geojson, epsg, device_id)
    ctx.update({"event": "geometry_validation_failed", "reason": reason, "level": "warning"})
    # One line of JSON: every field is filterable in the log store.
    logger.warning(json.dumps(ctx, separators=(",", ":")))

Two details make these logs durable. First, round the bounding box to a fixed precision so the same real-world footprint always produces the same logged bbox — this makes duplicate footprints groupable and keeps the field diffable across replays. Second, always log the EPSG code even when it is the expected EPSG:4326 (WGS84); the value of the field is precisely that it lets you find the events where it was not what you expected, such as an unannounced EPSG:3857 (Web Mercator) payload. Follow the CRS-notation convention everywhere — the code number is what makes the log queryable, and it is what a future incident responder will grep for.

Tracing an Event Across Async Geometry Handlers

The hardest spatial pipelines to debug are the asynchronous ones, where a single webhook fans out through a broker into a set of consumers that parse, validate, transform, and persist a geometry on different workers, possibly on different machines. Metrics and logs describe each hop in isolation; only a distributed trace stitches them into one causal story that says “this event spent 4 ms parsing, 180 ms validating a 90,000-vertex multipolygon, and 6 ms persisting.” OpenTelemetry is the vendor-neutral way to produce that story in Python.

The model is one span per meaningful stage, with spatial attributes on each span so the trace is self-explaining, and trace context propagated through the broker so the consumer’s spans link to the producer’s. The snippet below shows a producer starting a span and injecting context into message headers, and a consumer extracting it to continue the trace. The full treatment, including instrumenting asyncio gather fan-out and handling context across await boundaries, is in Adding OpenTelemetry Spans to Async Geometry Handlers.

from opentelemetry import trace, propagate
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("spatial.pipeline")

async def produce_geometry_event(geom, epsg: int, h3_cell: str, publish) -> None:
    """Start a span, tag it with spatial attributes, and inject context into headers."""
    with tracer.start_as_current_span("geometry.publish", kind=SpanKind.PRODUCER) as span:
        span.set_attribute("geo.epsg", epsg)
        span.set_attribute("geo.geom_type", geom.geom_type)
        span.set_attribute("geo.vertex_count", len(geom.exterior.coords))
        span.set_attribute("geo.h3_cell", h3_cell)  # links trace to partition/shard

        headers: dict[str, str] = {}
        propagate.inject(headers)  # W3C traceparent goes into the broker message headers
        await publish(payload=geom.wkt, headers=headers)

async def consume_geometry_event(message) -> None:
    """Continue the producer's trace on the consumer side across the async boundary."""
    ctx = propagate.extract(message.headers)  # rebuild parent context from headers
    with tracer.start_as_current_span("geometry.validate", context=ctx,
                                      kind=SpanKind.CONSUMER) as span:
        span.set_attribute("geo.h3_cell", message.headers.get("h3_cell", ""))
        # ... run validation / CRS transform here; exceptions are recorded on the span ...
        span.add_event("validation.complete")

Because each span carries the H3 cell as an attribute, traces double as a diagnostic for partition skew: filter your traces to the hot shard identified by the metrics plane and you immediately see the geometries that are making it hot. This is the payoff of coordinating the three planes — a metric raises the alarm, a trace localizes the cause to a shard, and the structured logs on that shard name the exact devices and bounding boxes responsible. The partitioning scheme that assigns events to shards in the first place, and the trade-offs between grid systems, are covered where they belong in H3 vs S2 vs Quadkey for Spatial Partitioning; here we only observe the shards it produces.

Consumer Lag and Partition Skew: Watching Sharded Spatial Streams

When a stream is partitioned by a spatial key, the health of the system is not one lag number but a distribution of lag across shards. A round-robin partitioner spreads load by construction; a spatial partitioner spreads it according to where events happen in the world, which is to say wildly unevenly. Ship geohash-prefix or H3-cell partitioning to a mapping product and the shard covering a metropolitan area will carry orders of magnitude more traffic than the shard covering an ocean. The result is a hot consumer that lags while its peers sit idle, and total throughput that looks entirely healthy the whole time.

Observing this requires per-shard metrics, not aggregates. Expose consumer lag as a gauge labelled by shard, and compute a skew ratio — the maximum shard load divided by the median — as a first-class signal. A skew ratio near 1 means balanced load; a ratio of 5 means your hottest shard is doing five times the median work and is a redistribution candidate. The remedies (finer partition keys, salting hot cells, dynamic reassignment) belong to stream design; detecting when you need them belongs here, and is developed fully in Consumer Lag & Partition Skew Monitoring and its deep dive Detecting Partition Skew in H3-Sharded Streams.

The alerting principle throughout is to fire on rates and ratios rather than absolute counts. Absolute event counts grow with your business and will either page you constantly or never; a validation failure rate, a skew ratio, and a per-shard lag measured against your redelivery window all stay meaningful as the system scales. This is also why the metrics catalogue above expresses nearly every threshold as a proportion or a comparison to a baseline rather than a fixed number.

From Metrics to Dashboards and Alerts

The three planes become operationally useful when they converge on a small number of screens and pages that a responder can read under pressure. A geospatial webhook health dashboard should lead with the geo-specific signals — validation failure rate, CRS-mismatch rate by EPSG code, partition skew, dedup hit ratio, and the vertex-count distribution — above the standard request-rate and latency panels, because in a spatial incident the geo panels are where the story is. A worked build of exactly this layout is in A Grafana Dashboard for Geospatial Webhook Health.

Alerts should map to the metrics catalogue but stay deliberately few. A validation failure rate above roughly 1 percent sustained, any CRS-mismatch traffic for a projection outside your allowlist, a partition skew ratio past 3 to 5, per-shard consumer lag exceeding the redelivery window, and a dedup hit ratio that suddenly collapses are the signals worth waking someone for. Each of these is a change in the shape of the data, which is the recurring theme of spatial observability: the transport can be flawless while the geometry quietly goes wrong, and the whole point of this instrumentation layer is to make that going-wrong visible the moment it starts.

Finally, treat observability as part of the pipeline’s contract, not an afterthought bolted on during an incident. Every stage the other sections teach you to build — the receiver, the deduplication gate, the parser and router, the retry and delivery layer — should emit its metrics, structured logs, and spans as it is written. Instrumentation added later never quite covers the paths that matter, and the paths that matter in spatial systems are the rare, heavy, malformed geometries that only observability can catch in the act.

FAQ

Related

• Geo-Specific Metrics & Instrumentation — the full catalogue of spatial metrics and how to expose them with the Prometheus Python client
• Consumer Lag & Partition Skew Monitoring — per-shard lag gauges and skew-ratio detection for spatially partitioned streams
• Structured Logging & Tracing for Spatial Events — JSON logs with bbox and H3 context, and OpenTelemetry spans across async handlers
• Queue Management, Retries & Delivery Guarantees — the broker, retry, and redelivery layer whose consumer lag this section measures
• Idempotency & Spatial Deduplication — the deduplication layer whose effectiveness the dedup hit ratio metric confirms in production
• H3 vs S2 vs Quadkey for Spatial Partitioning — the grid-system trade-offs behind the shards this section monitors for skew

Explore this section