An Error-Budget Policy for Tile Pipelines

Name a consequence that happens automatically, an owner who can enact it without escalating, and the exemptions — all before the budget is spent, because an exemption invented during an incident is indistinguishable from ignoring the policy.

This guide sits under SLOs & Alerting for Spatial Webhook Pipelines, within Monitoring & Observability for Spatial Pipelines. It is the step after the slow-burn ticket from Alert Rules That Survive a Bursty Spatial Stream has been filed and ignored twice.

When to use this pattern

  • An objective exists and is missed regularly with no consequence, which is the state that makes objectives decorative.
  • The tile pipeline competes for capacity with feature work, and the competition is settled by whoever asks most recently.
  • Overload currently produces an undirected backlog, so which tiles go stale is decided by queue order rather than by anyone.

A policy is a decision made early

What separates a policy from a sentence Two error-budget policies are compared clause by clause. The first says that when the budget is exhausted the team will review priorities and decide on appropriate action. It names no consequence, so nothing happens automatically; no owner, so enacting it requires a meeting; and no exemptions, so any change can be argued as necessary. It describes what would have happened anyway, and gives nobody a reason to protect the budget during the month. The second says that when the budget is exhausted, non-urgent changes to the tile pipeline are frozen until it recovers above twenty-five per cent; that the on-call engineer for the pipeline can declare the freeze without escalation; that security fixes, data-loss fixes and reliability work that would restore the budget are exempt; and that a freeze lasting more than ten working days escalates to a named engineering manager who can either extend it or accept the risk explicitly. Every clause answers a question that would otherwise be answered during an incident by whoever is most insistent. "the team will review priorities" consequence: none — nothing happens automatically owner: none — enacting it requires a meeting exemptions: none — so any change can be argued as necessary describes what would have happened anyway gives nobody a reason to protect the budget a policy that decides things consequence: non-urgent pipeline changes frozen until the budget recovers above 25% owner: the pipeline's on-call engineer declares it, without escalation exempt: security fixes, data-loss fixes, and the reliability work that would restore the budget way out: a freeze past 10 working days escalates to a named manager Every clause on the right answers a question that would otherwise be settled during an incident by whoever is most insistent.
Figure 1. The right-hand policy is not stricter — it is decided. Its value is that nobody has to make these calls while the pager is going.

Complete runnable implementation

The policy is a document, and the code is what makes it self-enforcing rather than aspirational.

python 
from dataclasses import dataclass
from enum import IntEnum


class BudgetState(IntEnum):
    HEALTHY = 0        # > 50% remaining
    WATCH = 1          # 25-50% — reliability work is prioritised
    FROZEN = 2         # < 25% — non-urgent pipeline changes blocked
    EXHAUSTED = 3      # 0% — freeze plus mandatory review


# Written in advance. An exemption invented during an incident is
# indistinguishable from ignoring the policy.
EXEMPT_CHANGE_TYPES = frozenset({
    "security-fix",
    "data-loss-fix",
    "reliability",        # the work that would restore the budget
    "rollback",
})

FREEZE_ESCALATION_DAYS = 10


@dataclass(frozen=True, slots=True)
class Policy:
    service: str
    owner_role: str                 # who can declare it, without escalating
    escalates_to: str               # who decides after FREEZE_ESCALATION_DAYS
    recover_to: float = 0.25        # freeze lifts here, NOT at 0% — otherwise
                                    # the pipeline oscillates in and out of freeze


TILE_POLICY = Policy(
    service="tile-pipeline",
    owner_role="tile-pipeline on-call",
    escalates_to="platform engineering manager",
)


def state_for(budget_remaining: float) -> BudgetState:
    if budget_remaining > 0.50:
        return BudgetState.HEALTHY
    if budget_remaining > 0.25:
        return BudgetState.WATCH
    if budget_remaining > 0.0:
        return BudgetState.FROZEN
    return BudgetState.EXHAUSTED


def may_deploy(change_type: str, budget_remaining: float,
               frozen_for_days: float = 0.0) -> tuple[bool, str]:
    """Called by CI. A policy nothing enforces is a preference."""
    state = state_for(budget_remaining)

    if state <= BudgetState.WATCH:
        return True, "budget healthy"

    if change_type in EXEMPT_CHANGE_TYPES:
        return True, f"{change_type} is exempt under the policy"

    if frozen_for_days > FREEZE_ESCALATION_DAYS:
        return False, (
            f"frozen {frozen_for_days:.0f} days — {TILE_POLICY.escalates_to} must "
            "either extend the freeze or accept the risk explicitly"
        )

    return False, (
        f"tile-pipeline budget at {budget_remaining:.0%}; non-urgent changes are "
        f"frozen until it recovers above {TILE_POLICY.recover_to:.0%}"
    )

Lifting the freeze at 25% rather than at 0% is the same hysteresis that keeps a paused partition from oscillating: recovering to exactly the threshold means the next bad hour re-freezes, and a freeze that toggles daily stops being taken seriously.

The degradation ladder

The other half of the policy decides what the pipeline does under overload, rather than leaving it to queue order.

Which tiles are allowed to go stale, decided in advance Tile zoom levels are ordered by how much of the audience each serves and how expensive each is to regenerate. Zoom levels zero to nine cover whole regions, so each tile is requested by every viewer looking anywhere in that area, and regenerating one means processing every feature it contains; these must stay current, and they are the last to be allowed to lag. Levels ten to fourteen are neighbourhood scale, shared by many viewers and moderately expensive; they are allowed to lag by minutes under pressure. Levels fifteen and deeper cover a street or a building, are requested by very few viewers each, and are cheap enough to render on demand when someone actually asks; they may lag by hours. Under overload the pipeline therefore sheds rebuild work from the bottom of this ladder upwards, and the effect on a reader is that fine detail in a rarely-viewed street is briefly out of date while every overview anyone looks at stays correct. Without the ladder, queue order decides, which means the tiles that go stale are whichever happened to be enqueued last — frequently the shallow ones, because a large feature edit invalidates them last. which tiles may lag, in what order, decided before the overload z0–z9 · region overviews — must stay current, shed last every viewer looking anywhere in the area requests these · regenerating one processes every feature it contains target lag: under the freshness objective, always z10–z14 · neighbourhood scale — may lag by minutes shared by many viewers, moderately expensive to regenerate target lag under pressure: 15 minutes z15+ · street and building scale — may lag by hours, shed first very few viewers each · cheap enough to render on demand when somebody actually asks target lag under pressure: 4 hours, or on demand Without the ladder, queue order decides — and a large feature edit enqueues shallow tiles last, so those are the ones that go stale.
Figure 2. The ladder inverts the default. Left to the queue, overload degrades exactly the tiles that the most people are looking at.
python 
# Rebuild priority under pressure. Lower value = rebuilt first.
ZOOM_PRIORITY = {z: (0 if z <= 9 else 1 if z <= 14 else 2) for z in range(0, 23)}

# Maximum acceptable lag per band while the pipeline is under pressure.
DEGRADED_LAG_SECONDS = {0: 60, 1: 900, 2: 14_400}


def rebuild_order(tiles):
    """Shed from the bottom of the ladder upwards.

    Sorting by (band, age) means a shallow tile is always rebuilt before a
    deep one, and within a band the oldest goes first.
    """
    return sorted(tiles, key=lambda t: (ZOOM_PRIORITY[t.z], -t.age_seconds))

Parameter reference

Name Type Spatial constraint Default
recover_to float Freeze lifts here, not at 0%, or it toggles daily 0.25
EXEMPT_CHANGE_TYPES frozenset Written in advance; additions are a policy change, not a judgement call 4 types
FREEZE_ESCALATION_DAYS int After this, a named person must extend or accept the risk 10
ZOOM_PRIORITY dict Shallow zooms first — the inverse of what queue order produces 3 bands
DEGRADED_LAG_SECONDS dict Per band, agreed with whoever consumes the tiles —
owner_role str Someone who can declare the freeze without a meeting on-call

Gotchas and spatial edge cases

  1. A policy nothing enforces is a preference. The may_deploy check has to run in CI and block the pipeline, not print a warning. A policy enforced by everyone remembering it is enforced during calm periods and forgotten during the ones it exists for.

  2. The exemption list must be short and hard to extend. Every plausible change can be argued into a category, and a list with a “business critical” entry has no entries at all. Adding one is a policy change with the same review as any other, which is what stops it being a judgement call at the moment of maximum pressure.

  3. The degradation ladder must match how tiles are actually invalidated. If invalidation is scoped by zoom as Scoping Tile Invalidation to the Zoom Levels That Changed describes, most small edits never enqueue shallow tiles at all, and the ladder mainly governs the large edits — which is the right target.

  4. On-demand rendering for deep zooms needs to exist before it is relied on. A ladder that allows z15 to lag four hours assumes something renders it when asked. If nothing does, the policy has quietly promised a capability the pipeline does not have.

  5. The budget and the shed rate are the same currency. Load shedding, described in Shedding Spatial Load by Geographic Priority, spends the completeness budget. A pipeline shedding routinely while its freshness budget looks healthy has moved the problem rather than solved it.

  6. Review the policy when it does not fire. A budget that has never been exhausted in a year means the objective is too loose, and the policy has been costless because it has never applied. That is worth noticing, because a policy that cannot bite provides the same information as no policy at all.

Freezing at zero and lifting at zero makes the policy a toggle Budget remaining is traced across a quarter. It falls steadily through a bad month, crosses twenty-five per cent and triggers the freeze. If the freeze lifted the moment the budget returned above zero, the pipeline would spend the following weeks oscillating: a good day pushes the budget just positive, the freeze lifts, a normal day's errors push it negative again, and the freeze returns — several times a week, each transition sending a notification that changes somebody's plans. A policy that toggles that often is one people route around, and the first person to argue that a particular change should not be blocked wins, because everyone already suspects the state is arbitrary. Lifting at twenty-five per cent instead means the freeze ends only once the pipeline has genuinely recovered, so it happens once, is visible, and is worth acting on. The hysteresis is the same mechanism that keeps a paused partition from oscillating, applied to an organisational decision rather than a broker one. error budget remaining, across a quarter 100% 0% lift at 25% freeze band freeze declared, once lifted, once Lifting at 0% instead would toggle several times a week, and a policy that toggles is one people route around — the first person to argue an exception wins.
Figure 3. The same hysteresis that keeps a paused partition from oscillating, applied to an organisational decision rather than a broker one.

Verification

python 
import pytest


def test_frozen_budget_blocks_a_feature_change():
    allowed, reason = may_deploy("feature", budget_remaining=0.10)
    assert not allowed and "frozen" in reason


def test_exempt_changes_pass_during_a_freeze():
    for change_type in ("security-fix", "data-loss-fix", "reliability", "rollback"):
        allowed, _ = may_deploy(change_type, budget_remaining=0.0)
        assert allowed, f"{change_type} must remain deployable"


def test_freeze_lifts_with_hysteresis():
    """Lifting at 0% would re-freeze on the next bad hour."""
    assert not may_deploy("feature", budget_remaining=0.20)[0]
    assert may_deploy("feature", budget_remaining=0.30)[0]


def test_long_freeze_escalates_to_a_named_person():
    _, reason = may_deploy("feature", budget_remaining=0.10, frozen_for_days=12)
    assert "platform engineering manager" in reason


def test_shallow_tiles_are_rebuilt_before_deep_ones():
    """The inverse of what queue order produces."""
    tiles = [_tile(z=18, age=3600), _tile(z=6, age=30), _tile(z=12, age=600)]
    assert [t.z for t in rebuild_order(tiles)] == [6, 12, 18]

The last test is the one that catches a regression in the thing readers actually experience: the deep tile is by far the oldest, so any ordering by age alone puts it first, and that ordering is exactly what the ladder exists to override.