Alert Rules That Survive a Bursty Spatial Stream

Alert on how fast the error budget is being consumed, over a short and a long window that must both be burning — and add an absent() rule per shard, because a consumer that dies stops emitting entirely and improves the fleet aggregate on its way out.

This guide sits under SLOs & Alerting for Spatial Webhook Pipelines, within Monitoring & Observability for Spatial Pipelines. It turns the objective from Defining a Freshness SLO for a Spatial Pipeline into pages worth answering.

When to use this pattern

  • Alerts on this pipeline are muted, routed to a channel nobody reads, or answered with “it cleared on its own”.
  • Traffic has predictable bursts — shift changes, weather events, scheduled imports — that are not incidents.
  • An objective exists, because burn rate is defined against a budget and there is no budget without one.

One signal, two very different events

The threshold cannot tell these apart; the two windows can Two incidents are plotted on the same freshness indicator. The first is a burst: a shift change pushes freshness above the objective for about ninety seconds and it recovers without intervention. The second is a regression: a bad deploy pushes freshness above the objective and it stays there. A threshold alert crosses on both and pages on both, so an operator answers the burst, finds it already resolved, and learns that this page does not need answering — which is the response that mutes the alert before the regression arrives. The five-minute burn-rate window also rises on both, because it responds quickly by design. The one-hour window barely moves during the burst, since ninety seconds of elevated error contributes little to an hour of measurement, but climbs steadily during the regression. Requiring both windows to be burning therefore fires on the second and stays silent through the first, and the delay this costs is about two minutes — the time the long window needs to accumulate enough of a sustained regression to cross. a 90-second burst — a shift change objective threshold alert: FIRES · 5 m burn: rises · 1 h burn: barely moves two-window rule: silent — correctly a sustained regression — a bad deploy threshold alert: FIRES · 5 m burn: rises · 1 h burn: climbs steadily two-window rule: FIRES, within about 2 minutes what the threshold alert costs the operator answers the burst, finds it resolved, and learns this page does not need answering the alert is muted before the regression arrives — which is how a pipeline ends up with silent pagers what the two-window rule costs about two minutes of extra latency on a genuine regression, while the long window accumulates in exchange for a page that is always worth answering — which is the only property that keeps it unmuted
Figure 1. The two-minute delay is the entire cost, and it buys the property that decides whether an alert still exists in six months.

Complete runnable implementation

yaml 
# 99% freshness over 30 days → a 1% error budget.
# Burn rate = observed error ratio ÷ budget. A rate of 1 exhausts the budget
# exactly at the end of the window; 14.4 exhausts it in about 50 hours.
groups:
  - name: spatial-freshness-alerts
    interval: 30s
    rules:
      # ---- fast burn: page. Short window makes it quick, long makes it stick.
      - alert: SpatialFreshnessBudgetBurningFast
        expr: |
          (1 - fleet:freshness_ratio:rate5m) > (14.4 * 0.01)
          and
          (1 - fleet:freshness_ratio:rate1h) > (14.4 * 0.01)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Freshness budget burning 14x — worst shard is behind"
          runbook: "Check per-shard lag before touching the consumer fleet"

      # ---- slow burn: ticket. Catches the regression nobody notices.
      - alert: SpatialFreshnessBudgetBurningSlow
        expr: |
          (1 - fleet:freshness_ratio:rate6h) > (3 * 0.01)
          and
          (1 - fleet:freshness_ratio:rate1d) > (3 * 0.01)
        for: 15m
        labels:
          severity: ticket

      # ---- a shard that stops emitting IMPROVES the aggregate. No burn-rate
      # rule can see that, because nothing is burning — the events simply
      # stopped being counted.
      - alert: SpatialShardSilent
        expr: |
          absent_over_time(
            spatial_event_freshness_seconds_count{shard=~"metro-.*"}[10m]
          )
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Shard {{ $labels.shard }} has emitted nothing for 10 minutes"

      # ---- one shard failing while the fleet looks fine. min() catches this
      # in the aggregate, but naming the shard is what makes the page actionable.
      - alert: SpatialShardBehind
        expr: |
          shard:freshness_ratio:rate30m < 0.90
          and
          shard:freshness_ratio:rate6h < 0.95
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "Shard {{ $labels.shard }} below objective for 6 hours"

The burn-rate multipliers are not arbitrary. A rate of 14.4 against a 30-day window exhausts the budget in about 50 hours, which is short enough to warrant waking someone; a rate of 3 exhausts it in about 10 days, which warrants a ticket. Choosing them by how long the budget would last is what makes the severity defensible.

The aggregate improves when a region goes dark Twelve shards are aggregated with min, and the fleet freshness ratio is plotted. For the first period the metro shard is struggling at a ratio of zero point eight two while the others sit near zero point nine nine, so min reports zero point eight two and the burn-rate alerts are firing correctly. The metro consumer then crashes hard enough to stop emitting metrics at all. Its series goes stale, min no longer sees it, and the fleet ratio jumps to zero point nine nine — the number improves at the exact moment the region stopped being served. Every burn-rate alert resolves, the incident channel goes quiet, and the dashboard shows a recovery. Nothing in the burn-rate family can detect this, because from their point of view there is no error: the events are not late, they are absent. The absent_over_time rule per known shard is the only thing that fires, and it has to enumerate the shards explicitly, because a rule that only watches series that exist cannot notice one that does not. fleet freshness ratio = min(shards) 1.00 0.80 objective 0.99 metro consumer stops emitting metro struggling at 0.82 — burn-rate alerts firing correctly min() no longer sees it · ratio jumps to 0.99 · every alert resolves the dashboard shows a recovery, and the region is not being served at all No burn-rate rule can see this — the events are not late, they are absent absent_over_time per known shard is the only rule that fires, and it must enumerate the shards: a rule watching series that exist cannot notice one that does not.
Figure 2. The recovery on this chart is the incident. Every alert resolving simultaneously is itself a signal, and only an explicitly enumerated absence check catches it.

Parameter reference

Name Type Spatial constraint Default
Fast burn multiplier float Budget exhausted in ~50 h; page-worthy 14.4
Slow burn multiplier float Budget exhausted in ~10 days; ticket-worthy 3
Short window duration Fast enough to catch a regression in minutes 5m / 6h
Long window duration Long enough that a burst does not move it 1h / 1d
for duration Additional damping; small, since the windows already damp 2m / 15m
Shard list in absent regex Must enumerate known shards explicitly metro-.*

Gotchas and spatial edge cases

  1. absent_over_time needs the shards named. A rule matching whatever series exist cannot notice a series that stopped existing, so the shard set has to come from configuration and be updated when a region is added. A newly onboarded region with no absence rule is a region that can go dark silently.

  2. Quiet shards produce NaN and poison min(). A shard with no traffic in the evaluation window divides by a near-zero rate, and NaN propagates differently across Prometheus versions. Gate the per-shard recording rule on a minimum event rate, which also stops overnight quiet periods from firing the shard-behind alert.

  3. Every alert resolving at once is a signal. A pipeline whose entire alert set clears simultaneously has usually lost its metrics rather than fixed its problem. A meta-alert on the count of series reporting is cheap and catches an exporter outage that otherwise reads as a clean bill of health.

  4. Burn-rate thresholds must be recomputed when the objective changes. The multipliers are relative to the budget, so tightening from 99% to 99.5% halves the budget and doubles the effective burn for the same error rate. Shipping a new target without new alert thresholds either floods or silences the pager.

  5. Test the rules against recorded bursts, not synthetic ones. promtool test rules accepts input series, and a real shift-change burst has a shape that a hand-written ramp does not — in particular it usually comes with a simultaneous traffic increase, which changes the denominator as well as the numerator.

  6. A per-shard page needs the shard in the annotation. “Freshness budget burning” sends an operator to a dashboard; “shard metro-3 below objective for 6 hours” sends them to a consumer. The label is available and omitting it costs the first ten minutes of every incident.

The multiplier is a deadline, which is what makes the severity arguable Four burn rates are mapped to how long a thirty-day error budget survives at each. At one times the budget rate the budget lasts exactly the thirty-day window, which is by definition acceptable and warrants nothing. At three times it is exhausted in about ten days, which is slow enough to fix during working hours and fast enough that ignoring it loses the month — a ticket. At six times it lasts five days. At fourteen point four times it is gone in about fifty hours, which is short enough that waiting until morning means the budget is spent before anyone looks, so it earns a page. Choosing the multipliers this way makes the severity defensible in a way that a threshold picked from a dashboard never is: the question "why does this page and that one not" has an answer in hours rather than in preference, and when somebody proposes tightening the objective the multipliers can be recomputed rather than re-argued. burn rate → how long a 30-day budget survives → severity 1× — budget lasts the full 30 days · acceptable by definition · no alert 3× — gone in ~10 days · fixable in working hours, but ignoring it loses the month · ticket 6× — gone in ~5 days · the middle ground most fleets skip 14.4× — gone in ~50 hours · waiting until morning spends it before anyone looks · page "Why does this page and that one not" then has an answer in hours rather than in preference — and tightening the objective becomes a recomputation rather than an argument.
Figure 3. Deriving the multiplier from a deadline is what lets the severity survive a review. A number chosen from a dashboard cannot answer why it is the number.

Verification

python 
import subprocess
import textwrap


def promtool(cases: str) -> None:
    open("/tmp/tests.yml", "w").write(cases)
    subprocess.run(["promtool", "test", "rules", "/tmp/tests.yml"], check=True)


def test_ninety_second_burst_does_not_page():
    """The case that mutes a threshold alert."""
    promtool(textwrap.dedent("""
        rule_files: [/etc/prometheus/spatial-freshness.yml]
        tests:
          - interval: 30s
            input_series:
              - series: 'fleet:freshness_ratio:rate5m'
                values: '1+0x8 0.2+0x3 1+0x40'
              - series: 'fleet:freshness_ratio:rate1h'
                values: '1+0x8 0.98+0x3 1+0x40'
            alert_rule_test:
              - eval_time: 6m
                alertname: SpatialFreshnessBudgetBurningFast
                exp_alerts: []
    """))


def test_sustained_regression_pages_within_five_minutes():
    promtool(textwrap.dedent("""
        rule_files: [/etc/prometheus/spatial-freshness.yml]
        tests:
          - interval: 30s
            input_series:
              - series: 'fleet:freshness_ratio:rate5m'
                values: '1+0x4 0.5+0x60'
              - series: 'fleet:freshness_ratio:rate1h'
                values: '1+0x4 0.5+0x60'
            alert_rule_test:
              - eval_time: 5m
                alertname: SpatialFreshnessBudgetBurningFast
                exp_alerts:
                  - exp_labels: {severity: page}
    """))


def test_silent_shard_pages_even_though_the_ratio_improved():
    """The failure no burn-rate rule can see."""
    promtool(textwrap.dedent("""
        rule_files: [/etc/prometheus/spatial-freshness.yml]
        tests:
          - interval: 30s
            input_series:
              - series: 'spatial_event_freshness_seconds_count{shard="metro-3"}'
                values: '0+10x10 _x40'
            alert_rule_test:
              - eval_time: 16m
                alertname: SpatialShardSilent
                exp_alerts:
                  - exp_labels: {severity: page, shard: metro-3}
    """))

The first test is the one that justifies the whole design, and it is worth running in CI rather than once at review time — a later edit that drops the long-window clause from the expression passes every other test and reintroduces the threshold alert exactly.