Defining a Freshness SLO for a Spatial Pipeline

Start the clock at the source mutation, stop it at commit rather than at parse, write the objective against the worst shard, and pick the target from what a reader would notice — an objective derived from current performance is satisfied by construction and tells you nothing.

This guide sits under SLOs & Alerting for Spatial Webhook Pipelines, within Monitoring & Observability for Spatial Pipelines. That topic covers the indicator set; this one covers writing the freshness objective itself.

When to use this pattern

  • The pipeline has no objective, or one whose target nobody can explain.
  • Latency dashboards look healthy while consumers report stale data, which usually means the clock starts in the wrong place.
  • A new stream is being onboarded and the objective is being written before the traffic exists, which is the right time.

Three decisions, and the clock is the first

Where the clock runs decides what the objective is about One feature edit is traced from source to reader across five segments: the producer batching the change for up to four minutes before sending, the network delivery taking under a second, the broker queue holding it for two minutes behind a backlog, the consumer parsing it in forty milliseconds, and the write plus tile invalidation taking three seconds. A clock started at receipt and stopped at parse covers only the fourth segment and reports forty milliseconds — a number that is accurate, meaningless, and reliably green. Started at receipt and stopped at commit it covers the broker queue and the write, reporting just over two minutes, which is better but still excludes the producer's batching. Started at the source mutation and stopped at commit it covers everything the reader waits for, reporting about six minutes. Only the third measures the property the pipeline exists to provide, and the difference between it and the first is two orders of magnitude — which is why a pipeline can have a perfect latency dashboard and stale maps at the same time. one feature edit, from source mutation to visible in a tile producer batching — up to 4 min net broker queue — 2 min parse write + tile — 3 s receipt → parse: 40 ms · accurate, meaningless, reliably green receipt → commit: 2 min source → commit: ≈ 6 min Only the third measures what the reader waits for The first and the third differ by two orders of magnitude, which is how a pipeline holds a perfect latency dashboard and stale maps at the same time. Excluding the producer's batching excludes what is often the largest term.
Figure 1. Each candidate clock is a defensible measurement of something. Only one of them is a measurement of the promise the pipeline makes.

Complete runnable implementation

python 
import time
from dataclasses import dataclass
from datetime import datetime, UTC

from prometheus_client import Counter, Histogram

# Buckets MUST straddle the objective, or the ratio below interpolates a
# number that means nothing. The 60 s boundary is not decorative.
FRESHNESS = Histogram(
    "spatial_event_freshness_seconds",
    "Source mutation to consumer commit",
    labelnames=("shard", "stream"),
    buckets=(1, 5, 15, 30, 60, 120, 300, 900, 3600, float("inf")),
)
CLOCK_SKEW = Counter("spatial_event_future_timestamps_total",
                     "Events stamped in the future", ("shard",))


@dataclass(frozen=True, slots=True)
class FreshnessObjective:
    stream: str
    target_seconds: float      # what a reader notices
    ratio: float               # fraction that must be inside the target
    window_days: int           # over which the ratio is evaluated
    aggregation: str = "worst-shard"


OBJECTIVES = {
    # A dispatcher watching vehicles notices tens of seconds.
    "vehicle.position": FreshnessObjective("vehicle.position", 60.0, 0.99, 30),
    # A planner reviewing parcel edits does not notice tens of minutes.
    "feature.boundary_edit": FreshnessObjective("feature.boundary_edit",
                                                900.0, 0.995, 30),
}


def observe_commit(event: dict, shard: str, stream: str) -> None:
    """Record freshness at COMMIT, from the SOURCE timestamp.

    Both halves matter: parse-time observation reports the fastest stage as
    the whole, and receipt-time origin excludes the producer's batching.
    """
    occurred = datetime.fromisoformat(event["occurred_at"]).astimezone(UTC)
    elapsed = time.time() - occurred.timestamp()

    if elapsed < 0:
        # A producer clock ahead of ours. Clamping keeps the event in the
        # denominator; discarding it would silently improve the ratio.
        CLOCK_SKEW.labels(shard=shard).inc()
        elapsed = 0.0

    FRESHNESS.labels(shard=shard, stream=stream).observe(elapsed)


def recording_rules(objective: FreshnessObjective) -> str:
    """Per-shard ratio, then the worst shard. Never avg()."""
    return f"""
- record: shard:freshness_ratio:rate5m
  expr: |
    sum by (shard) (
      rate(spatial_event_freshness_seconds_bucket{{
        le="{int(objective.target_seconds)}", stream="{objective.stream}"}}[5m])
    )
    /
    sum by (shard) (
      rate(spatial_event_freshness_seconds_count{{stream="{objective.stream}"}}[5m])
    )
    # Quiet shards divide by ~0 and produce NaN, which poisons min().
    and on (shard) (
      sum by (shard) (
        rate(spatial_event_freshness_seconds_count{{stream="{objective.stream}"}}[5m])
      ) > 0.01
    )

- record: fleet:freshness_ratio:rate5m
  expr: min(shard:freshness_ratio:rate5m)
"""


def achievable(objective: FreshnessObjective, observed_ratio: float) -> str:
    """Say plainly whether the target is reachable rather than moving it."""
    if observed_ratio >= objective.ratio:
        return "met"
    if observed_ratio >= objective.ratio - 0.02:
        return "reachable with current architecture"
    return ("not reachable — the objective is correct and the pipeline is not; "
            "record the gap rather than lowering the target")

The achievable function exists to make one specific outcome sayable. An objective that the pipeline cannot meet is uncomfortable, and the usual response is to lower it until it is green, which converts a known problem into an unknown one.

Two ways to pick a number, and only one can be missed Two approaches to choosing the freshness target are contrasted. Deriving it from current performance means measuring the ninety-ninth percentile the pipeline achieves today and writing that down as the objective. The result is green from the day it ships, cannot be missed except by a regression, and encodes whatever the pipeline happens to do — including the parts nobody designed — as the standard. It also gives the error budget no meaning, because a budget only measures distance from a target that was chosen independently. Deriving it from what a reader notices means starting with the use: a dispatcher watching vehicles reacts to tens of seconds, so sixty seconds is a defensible target whether or not the pipeline currently achieves it. That objective can be missed, which is the point — a missable objective is the only kind that carries information. If the pipeline cannot reach it, the honest response is to record the gap as a known shortfall with an owner, not to move the target until the dashboard turns green. derived from current performance measure today's p99, write it down green from the day it ships can only be missed by a regression encodes whatever the pipeline happens to do — including the parts nobody designed the error budget measures nothing derived from what a reader notices a dispatcher reacts to tens of seconds → 60 s a planner does not notice tens of minutes → 15 min can be missed — which is the point if the pipeline cannot reach it, record the gap as a known shortfall with an owner the budget now measures a real distance A missable objective is the only kind that carries information. Moving the target until the dashboard turns green converts a known problem into an unknown one, and the pipeline is no faster afterwards.
Figure 2. The uncomfortable objective is the useful one. Its discomfort is the information.

Parameter reference

Name Type Spatial constraint Default
Clock start — Source mutation timestamp, set by the producer —
Clock stop — Commit, or tile invalidation — when the reader can see it —
target_seconds float From the use, then checked for achievability per stream
ratio float Fraction inside the target over the window 0.99
Histogram buckets tuple Must include an exact boundary at the target —
aggregation str worst-shard; avg hides a saturated metro partition worst-shard

Gotchas and spatial edge cases

  1. Retune the buckets whenever the target moves. A target of 60 seconds against buckets of 30 and 300 cannot be evaluated at all, and Prometheus will happily return an interpolated ratio that looks like a number. Every target change is also a bucket change, and they must ship together.

  2. A dead shard leaves the aggregate and improves it. When a consumer stops emitting, its series goes stale and min() stops seeing it, so the fleet ratio rises at the moment a region went dark. Pair the objective with an absent() alert per known shard, or the SLO reports its best figure during an outage.

  3. Clamp future timestamps rather than dropping them. A producer clock two seconds ahead yields a negative measurement that Prometheus rejects, removing the event from the denominator and silently improving the ratio. Clamp, count the occurrence, and if skew approaches the target the objective is measuring the clocks.

  4. The stop point differs by consumer. For a tile pipeline the reader sees the change when the tile is invalidated, not when the message is committed to a database, and the gap between those can be minutes — see Scoping Tile Invalidation to the Zoom Levels That Changed. One stream can need two objectives if it feeds two kinds of reader.

  5. Deliberate delay spends the budget. Debouncing, batching and windowing all trade freshness for efficiency, so their maximum wait has to fit inside the target with room left for the actual work — a five-second debounce against a sixty-second objective is comfortable, sixty seconds against sixty is not.

  6. One objective per stream, not one per pipeline. Vehicle telemetry and cadastral edits have targets an order of magnitude apart, and a single objective covering both is either far too loose for the first or unreachable for the second.

A target with no bucket is a ratio Prometheus will invent The same freshness histogram is shown with two bucket layouts against a sixty-second objective. In the first, an explicit boundary sits at sixty, so the count of observations at or below sixty seconds is a stored number and the ratio is exact. In the second, the nearest boundaries are thirty and three hundred, so no stored count corresponds to the objective; a query for the fraction under sixty seconds has to interpolate within a bucket that spans nine tenths of a decade, and it returns a number that looks like a measurement and is a guess whose error depends on how the events are distributed inside that bucket. The failure is silent in both directions: the ratio can read comfortably above the objective while the true value is below it, or the reverse. Because the target and the buckets live in different files — one in the objective definition, one in the instrumentation — a change to either alone produces this, which is why they have to ship together. freshness histogram against a 60-second objective boundary at the objective 60 s the count at or below 60 s is a stored number · the ratio is exact nearest boundaries 30 and 300 30 s 300 s 60 s no stored count matches the objective · the query interpolates across nine tenths of a decade Silent in both directions, and the target and the buckets live in different files — which is why they have to ship together.
Figure 3. An interpolated ratio looks exactly like a measured one on a dashboard, so this failure is only ever found by checking that a boundary exists.

Verification

python 
import pytest
from datetime import datetime, timedelta, UTC

NOW = datetime(2026, 8, 8, 12, 0, tzinfo=UTC)


def test_clock_starts_at_the_source_not_receipt():
    """The measurement must include the producer's batching delay."""
    event = {"occurred_at": (NOW - timedelta(minutes=6)).isoformat(),
             "received_at": (NOW - timedelta(minutes=2)).isoformat()}
    observed = _observed_value(event, now=NOW)
    assert observed > 300, "the producer's 4-minute batching was excluded"


def test_future_timestamp_is_clamped_and_counted():
    """Dropping it would silently improve the ratio."""
    event = {"occurred_at": (NOW + timedelta(seconds=2)).isoformat()}
    before = _skew_count()
    assert _observed_value(event, now=NOW) == 0.0
    assert _skew_count() == before + 1


def test_buckets_include_an_exact_boundary_at_every_target():
    """A target with no matching bucket cannot be evaluated."""
    boundaries = {1, 5, 15, 30, 60, 120, 300, 900, 3600}
    for objective in OBJECTIVES.values():
        assert objective.target_seconds in boundaries


def test_unachievable_target_is_reported_not_lowered():
    """The uncomfortable outcome must be sayable."""
    objective = OBJECTIVES["vehicle.position"]
    verdict = achievable(objective, observed_ratio=0.86)
    assert "not reachable" in verdict and "lowering" in verdict

The third test is cheap and catches a real failure: it fails the moment somebody changes a target without changing the histogram, which is a two-line edit in two different files that nothing else connects.