Geo-Specific Metrics & Instrumentation

Instrumenting a geospatial webhook pipeline means exporting the handful of metrics that reveal spatial-specific failure — geometry validity, CRS drift, vertex volume, dedup effectiveness, per-stage latency, and dead-letter depth — as Prometheus time series, each modelled with the correct instrument type so the numbers stay meaningful under aggregation.

This topic sits under Monitoring & Observability for Spatial Pipelines, the broader practice of making a real-time geospatial event system legible in production. Generic HTTP dashboards tell you the endpoint returned 200; they do not tell you that 4% of incoming polygons are self-intersecting or that a sensor firmware update started shipping EPSG:27700 into a pipeline that assumes EPSG:4326 (WGS84). This page defines the metric set that closes that gap and shows how to wire it with the prometheus_client library.

Prerequisites

Before instrumenting, confirm your stack meets this baseline. Check off each item as you verify it:

• Python 3.11+ — required for the typing and contextlib timing helpers used below
• prometheus-client 0.20+ — the official Python instrumentation library (Counter, Gauge, Histogram, make_asgi_app)
• fastapi 0.110+ / starlette — the async webhook handler the instruments hook into
• shapely 2.0+ — used to count vertices and evaluate geometry validity for the metrics
• A running Prometheus server — or prometheus in Docker, configured to scrape your service’s /metrics
• A single-value-set understanding of your CRS inputs — know which EPSG codes actually arrive, so CRS labels stay bounded
• pytest + httpx — for the scrape-the-endpoint verification test at the end

Architecture

Instrumentation in the Prometheus model is pull-based, and that shapes the whole design. Your handler never pushes numbers anywhere. Instead it mutates in-process metric objects that live in a local registry, and a separate Prometheus server periodically scrapes a text-format /metrics endpoint your app exposes. The pipeline is four layers:

1. Handler instrumentation — inside the FastAPI geometry handler, each processing stage calls Counter.inc(), Gauge.set(), or Histogram.observe() on module-level metric objects. These are cheap in-memory atomic updates with no network I/O.
2. Metric registry — the prometheus_client default CollectorRegistry holds every metric and its label-keyed child series in memory. This is the single source of truth the exporter reads.
3. /metrics exposition — an ASGI sub-app (or the make_asgi_app() helper) renders the registry to the Prometheus text exposition format on demand, when scraped.
4. Prometheus scrape — the Prometheus server pulls /metrics on its configured interval (commonly 15s), stores the samples, and makes them queryable in PromQL and Grafana.

The core geo metric set

Choosing the instrument type is the decision that makes or breaks the metric. A Counter only ever goes up and is read through rate(); use it for events you tally. A Gauge holds a single current value that can rise or fall; use it for levels you sample. A Histogram records a distribution into buckets plus a sum and count; use it for per-event measurements where percentiles matter. The following table fixes the choice for each spatial signal.

Two of these deserve emphasis. The dedup hit ratio is not stored as a ratio: Prometheus cannot aggregate a pre-divided Gauge across replicas, so you export raw counts and let PromQL divide. The dedup layer this instruments is described in Idempotency & Spatial Deduplication; a low hit ratio there is a signal that upstream retries are not being caught. DLQ depth is a Gauge because it is a level, not an event count — it rises when events fail past their retry budget and falls when an operator drains the queue.

The reason the Counter-versus-Gauge distinction is not cosmetic is that Prometheus treats the two completely differently at query time. A Counter is expected to only increase and to reset to zero on process restart; the rate() and increase() functions are built to detect those resets and compute a correct per-second rate across them. Feed a Gauge into rate() and you get nonsense, because a Gauge’s decreases look like counter resets. Conversely, applying rate() to something that should be a level — like DLQ depth — erases exactly the information you care about. Getting the type right at declaration time is therefore what makes every downstream query, alert, and dashboard panel correct; a mistyped metric cannot be salvaged in PromQL. When in doubt, ask whether the number can ever legitimately go down during normal operation: if yes, it is a Gauge; if it only accumulates, it is a Counter; if you need its distribution rather than a single value, it is a Histogram.

Step-by-step implementation

Step 1 — Declare the metrics at module scope

Metric objects must be created exactly once per process. Declaring them at module scope guarantees a single instance in the default registry; creating them inside the request handler raises a duplicate-timeseries error on the second request. Every histogram gets explicit buckets tuned to spatial reality — default buckets are latency-shaped and useless for vertex counts.

from prometheus_client import Counter, Gauge, Histogram

# Counter: geometry validation failures, sliced by why and by geometry type.
GEOMETRY_VALIDATION_FAILURES = Counter(
    "geo_geometry_validation_failures_total",
    "Count of geometries rejected by the validation stage.",
    ["reason", "geometry_type"],
)

# Counter: CRS mismatches requiring reprojection. EPSG codes are a bounded set.
CRS_REPROJECTIONS = Counter(
    "geo_crs_reprojections_total",
    "Count of payloads reprojected from a source CRS to the pipeline CRS.",
    ["source_epsg", "target_epsg"],
)

# Histogram: vertices per geometry. Buckets span point to dense-polygon scale.
GEOMETRY_VERTEX_COUNT = Histogram(
    "geo_geometry_vertex_count",
    "Distribution of coordinate/vertex counts per incoming geometry.",
    ["geometry_type"],
    buckets=(1, 10, 50, 100, 500, 1_000, 5_000, 10_000, 100_000),
)

# Counter pair for dedup: export raw counts, derive the ratio in PromQL.
DEDUP_LOOKUPS = Counter(
    "geo_dedup_cache_lookups_total",
    "Total idempotency cache lookups.",
    ["outcome"],  # values: "hit" | "miss"
)

# Histogram: per-stage latency in seconds, SLO-aligned buckets.
STAGE_LATENCY = Histogram(
    "geo_stage_latency_seconds",
    "Wall-clock latency of each pipeline stage.",
    ["stage"],  # values: "validate" | "dedup" | "reproject" | "route"
    buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)

# Gauge: current dead-letter queue depth.
DLQ_DEPTH = Gauge(
    "geo_dlq_depth",
    "Current number of events sitting in the dead-letter queue.",
    ["queue"],
)

Note the naming convention: Counters end in _total, Histograms measuring time end in _seconds, and Gauges use a plain noun. Prometheus tooling relies on these suffixes.

Step 2 — Wire the instruments into the FastAPI geometry handler

Each stage updates its metric as the event flows through. Use a small timing context manager so the latency histogram wraps each stage without repetition. The vertex count is computed from the parsed geometry, and the dedup outcome is recorded on the branch that decides it.

import time
from contextlib import contextmanager

from fastapi import FastAPI, Request, Response
from shapely.geometry import shape

app = FastAPI()


@contextmanager
def observe_stage(stage: str):
    """Record wall-clock duration of a pipeline stage into STAGE_LATENCY."""
    start = time.perf_counter()
    try:
        yield
    finally:
        STAGE_LATENCY.labels(stage=stage).observe(time.perf_counter() - start)


def count_vertices(geom) -> int:
    """Total coordinate count across all rings/parts of a shapely geometry."""
    if geom.geom_type == "Point":
        return 1
    if hasattr(geom, "geoms"):  # multi-part geometry collection
        return sum(count_vertices(g) for g in geom.geoms)
    if geom.geom_type == "Polygon":
        return len(geom.exterior.coords) + sum(len(r.coords) for r in geom.interiors)
    return len(geom.coords)  # LineString, LinearRing


@app.post("/webhook/spatial")
async def receive(request: Request) -> Response:
    body = await request.json()
    geometry_type = body.get("geometry", {}).get("type", "unknown")

    # --- validate stage ---
    with observe_stage("validate"):
        try:
            geom = shape(body["geometry"])
            if not geom.is_valid:
                GEOMETRY_VALIDATION_FAILURES.labels(
                    reason="invalid_topology", geometry_type=geometry_type
                ).inc()
                return Response(status_code=422, content="invalid geometry")
        except (KeyError, ValueError, TypeError):
            GEOMETRY_VALIDATION_FAILURES.labels(
                reason="malformed", geometry_type=geometry_type
            ).inc()
            return Response(status_code=400, content="malformed payload")

    # Record geometry size for the distribution.
    GEOMETRY_VERTEX_COUNT.labels(geometry_type=geometry_type).observe(count_vertices(geom))

    # --- dedup stage ---
    with observe_stage("dedup"):
        is_duplicate = await already_seen(body["event_id"])  # your cache lookup
        DEDUP_LOOKUPS.labels(outcome="hit" if is_duplicate else "miss").inc()
        if is_duplicate:
            return Response(status_code=200, content="duplicate")

    # --- reproject stage (CRS alignment) ---
    source_epsg = body.get("crs", "EPSG:4326")
    if source_epsg != "EPSG:4326":  # pipeline canonical CRS per RFC 7946
        with observe_stage("reproject"):
            geom = reproject_to_wgs84(geom, source_epsg)  # your pyproj transform
            CRS_REPROJECTIONS.labels(
                source_epsg=source_epsg, target_epsg="EPSG:4326"
            ).inc()

    # --- route stage ---
    with observe_stage("route"):
        await route_event(body, geom)

    return Response(status_code=202, content="accepted")

The validation failures counter is the raw material behind Tracking Geometry Validation Failure Rate with Prometheus, which turns geo_geometry_validation_failures_total into an alerting rule. The validation logic itself belongs in a dedicated stage; see Geometry Validation Pipelines for the full topology, ring-orientation, and CRS checks that precede instrumentation.

Step 3 — Keep the DLQ gauge fresh

A Gauge only reflects reality if something sets it. Update it wherever an event enters or leaves the dead-letter queue, or refresh it periodically from the queue’s own length. Prefer setting the absolute value from the source of truth over inc()/dec(), which drift if a process restarts.

async def refresh_dlq_depth(dlq_client, queue_name: str = "spatial-dlq") -> None:
    """Sample the queue's true length and publish it as the current gauge value."""
    depth = await dlq_client.llen(queue_name)  # e.g. Redis list length
    DLQ_DEPTH.labels(queue=queue_name).set(depth)

Call refresh_dlq_depth on a short interval from a background task. A steadily climbing geo_dlq_depth is the earliest sign that a downstream dependency is failing every retry — pair it with the consumer-side signals in Consumer Lag & Partition Skew Monitoring to distinguish a stuck consumer from a genuine burst of poison messages.

Label cardinality: the one way to break your metrics

Prometheus creates a distinct time series for every unique combination of label values. Bounded labels are free; unbounded labels are a production incident. The rule is absolute: only attach a label whose value set is small and stable.

# SAFE — source_epsg draws from a handful of known EPSG codes.
CRS_REPROJECTIONS.labels(source_epsg="EPSG:27700", target_epsg="EPSG:4326").inc()

# DANGEROUS — every event has a unique ID, so this creates one series per event.
# The local registry grows without bound and the scrape payload explodes.
# GEOMETRY_VALIDATION_FAILURES.labels(reason=event_id, geometry_type=gt).inc()

# DANGEROUS — raw coordinates are effectively infinite-cardinality.
# GEOMETRY_VERTEX_COUNT.labels(geometry_type=f"{lat},{lon}").observe(n)

High-cardinality geo labels to never use: geometry ID, feature ID, event ID, raw coordinates, bounding-box strings, user ID, and full H3/S2 cell indices at fine resolution. If you need per-feature detail, that belongs in traces and logs, not metrics — see Structured Logging & Tracing for Spatial Events for where unbounded spatial identifiers legitimately live. A coarse H3 resolution (say, resolution 3, a few thousand cells globally) can be an acceptable label; resolution 9 (millions of cells) cannot.

A practical guardrail is to bound the value at the call site by mapping to a category:

def epsg_label(raw_crs: str) -> str:
    """Collapse unrecognised CRS strings into a single 'other' bucket
    so a malformed or exotic CRS cannot explode label cardinality."""
    known = {"EPSG:4326", "EPSG:3857", "EPSG:27700", "EPSG:4269", "CRS84"}
    return raw_crs if raw_crs in known else "other"

Expose the /metrics endpoint

Mount the exposition endpoint as an ASGI sub-app so scrapes are served by the same process that holds the registry. This is the surface Prometheus pulls.

from prometheus_client import make_asgi_app

# Serves the default registry in Prometheus text exposition format.
app.mount("/metrics", make_asgi_app())

Then point Prometheus at it:

scrape_configs:
  - job_name: "geo-webhook"
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ["geo-webhook.internal:8000"]

A note on multi-process servers: under Gunicorn/Uvicorn with several workers, each worker holds its own registry, and a plain scrape hits only one at random. For that topology use prometheus_client’s multiprocess mode (PROMETHEUS_MULTIPROC_DIR) so counters aggregate across workers.

Verification

Do not trust that a metric is exported until you have scraped it. This test drives the handler, then pulls /metrics through the ASGI app and asserts the expected series and label values appear in the exposition text.

import pytest
from httpx import AsyncClient, ASGITransport

from app import app  # the FastAPI app defined above


@pytest.mark.asyncio
async def test_metrics_endpoint_exports_geo_series():
    transport = ASGITransport(app=app)
    async with AsyncClient(transport=transport, base_url="http://test") as client:
        # Drive one malformed event so a validation-failure series is created.
        await client.post("/webhook/spatial", json={"event_id": "e1", "geometry": {}})

        # Scrape the exposition endpoint the way Prometheus would.
        resp = await client.get("/metrics")
        assert resp.status_code == 200
        body = resp.text

        # The counter and its labels must be present in the text format.
        assert "geo_geometry_validation_failures_total" in body
        assert 'reason="malformed"' in body

        # Histograms export _bucket, _sum, and _count series.
        assert "geo_stage_latency_seconds_bucket" in body
        assert 'stage="validate"' in body


@pytest.mark.asyncio
async def test_dedup_counter_increments_by_outcome():
    from app import DEDUP_LOOKUPS

    before = DEDUP_LOOKUPS.labels(outcome="miss")._value.get()
    DEDUP_LOOKUPS.labels(outcome="miss").inc()
    after = DEDUP_LOOKUPS.labels(outcome="miss")._value.get()
    assert after == before + 1

Run with pytest -v --asyncio-mode=auto. The first test proves the full instrument-to-exposition path works; the second confirms a labelled counter actually moves. Once the series are confirmed present, they can be graphed — the panels are laid out in A Grafana Dashboard for Geospatial Webhook Health, which turns each metric here into a dashboard row.

Troubleshooting

FAQ

Related

• Monitoring & Observability for Spatial Pipelines — the parent section covering metrics, logging, tracing, and lag monitoring for geospatial event systems
• Tracking Geometry Validation Failure Rate with Prometheus — turning the validation-failure counter into an alerting rule with PromQL
• A Grafana Dashboard for Geospatial Webhook Health — the panels that visualise every metric defined here
• Consumer Lag & Partition Skew Monitoring — pairing DLQ depth with consumer-side signals to locate stalls
• Structured Logging & Tracing for Spatial Events — where high-cardinality spatial identifiers belong instead of metric labels

Explore this section