Adding a Field to a Live Spatial Event Schema

Add the field to the idempotency hash’s exclusion list before you add it to the schema — otherwise the same logical event hashes differently on either side of the rollout, deduplication stops matching, and every redelivery during the changeover is written twice.

This guide sits under Schema Evolution & Versioning for Spatial Events, within Core Event Fundamentals & Architecture. It assumes the deterministic key scheme described in Event Key Generation for Spatial Data.

When to use this pattern

  • You are adding an optional field — a confidence score, a source identifier, a sensor model — to a stream that is already in production and already deduplicated.
  • Your idempotency or content hash is currently computed over the serialised payload rather than over a named subset of it.
  • Consumers exist that you do not control, so the change has to be safe for readers that will never be updated.

If the stream has no deduplication and no content hashing, adding an optional field really is trivial, and this guide is about the case where it is not.

Why the hash is the problem, not the field

The field itself is harmless. Old consumers ignore it, new consumers read it, and every payload validates on both sides. What breaks is anything downstream that treated the payload’s bytes as the event’s identity.

Where an added field does and does not reach One logical event for feature 4471 is delivered before and after producers begin sending a confidence field. Hashing a canonical serialisation of the whole payload includes the new field, so the two deliveries produce different digests and the deduplication store, holding the earlier one, does not recognise the later one; the event is written a second time and every redelivery during the rollout window does the same. Hashing over an explicit identity list — feature id, normalised geometry, occurred-at — excludes the new field by construction, so both deliveries produce the same digest and the second is correctly recognised as a duplicate. The distinction is not about which fields are important but about which fields constitute identity: confidence describes the observation, it does not say which observation this is. hash(canonical(whole payload)) {feature_id, geometry, occurred_at} before → 9f2c1a… {feature_id, geometry, occurred_at, confidence} after → 41ba6d… — no match written twice for every redelivery in the window hash over an explicit identity list IDENTITY = (feature_id, geometry, occurred_at) confidence is not in the list, so it cannot move the digest — 9f2c1a… both times recognised as a duplicate written once, correctly The question is not which fields matter — it is which fields constitute identity. Confidence describes the observation. It does not say which observation this is, so it has no business in the key.
Figure 1. Moving to an explicit identity list is the change that has to land first — it makes this addition safe and every future one safe as well.

Complete runnable implementation

python 
import hashlib
import json
import math
from typing import Any

from shapely.geometry import mapping, shape
from shapely.ops import transform


# The identity of a spatial event: which feature, what shape, when observed.
# Anything describing the observation rather than naming it stays out.
IDENTITY_FIELDS = ("feature_id", "geometry", "occurred_at")

PRECISION = 6  # decimal places; part of the schema, see the parent topic


def _round_geometry(geometry: dict[str, Any]) -> dict[str, Any]:
    """Round coordinates so float noise cannot move the digest."""
    geom = shape(geometry)
    rounded = transform(
        lambda *coords: tuple(round(c, PRECISION) for c in coords), geom
    )
    return mapping(rounded)


def identity_key(event: dict[str, Any]) -> str:
    """Derive the idempotency key from named fields only.

    Adding, removing or reordering any field outside IDENTITY_FIELDS cannot
    change this value — which is exactly the property that makes a schema
    addition safe to deploy against a live deduplication store.
    """
    identity = {}
    for name in IDENTITY_FIELDS:
        value = event[name]
        identity[name] = _round_geometry(value) if name == "geometry" else value

    canonical = json.dumps(
        identity, sort_keys=True, separators=(",", ":"), ensure_ascii=True
    )
    return hashlib.sha256(canonical.encode()).hexdigest()[:32]


def confidence_of(event: dict[str, Any]) -> float:
    """Read the new field, preserving the difference between absent and zero.

    An event produced before the field existed did not measure confidence.
    Returning 0.0 would assert the reading is certainly wrong, which is a
    different — and false — statement.
    """
    if "confidence" not in event:
        return math.nan
    return float(event["confidence"])

Parameter reference

Name Type Spatial constraint Default
IDENTITY_FIELDS tuple[str, ...] Must name identity, not description; changing it is itself a breaking change ("feature_id", "geometry", "occurred_at")
PRECISION int Decimal places in EPSG:4326; 6 ≈ 11 cm. Changing it moves every key 6
sort_keys bool Must be True, or key insertion order leaks into the digest True
separators tuple[str, str] Must be (",", ":") so whitespace cannot vary —
ensure_ascii bool True keeps non-ASCII place names encoded identically across producers True

Gotchas and spatial edge cases

  1. Changing IDENTITY_FIELDS is a breaking change, not a refactor. Every key in the deduplication store was computed under the old list. Adding or removing an identity field invalidates all of them at once, so it needs the same overlap treatment as any other schema version bump — and, unlike a payload field addition, it cannot be made invisible.

  2. The geometry must be normalised before it enters the key. If the added field’s rollout coincides with a producer that emits different ring winding or a different CRS, you will see the deduplication failure and blame the new field. Normalise projection, winding and precision inside identity_key so the key depends on the shape rather than on its encoding.

  3. NaN does not compare equal to itself. Using math.nan to mean “not measured” is correct for storage and wrong for equality tests: nan == nan is False, so a naive comparison treats two unmeasured events as different. Test with math.isnan(), or carry a separate confidence_measured: bool if the value flows into comparison logic.

  4. A field added inside geometry is not optional at all. GeoJSON geometry objects have a fixed member set; adding a vendor field inside one produces a payload many strict parsers reject, and it will be included by any geometry-aware hashing. Extra fields belong in the envelope, alongside the routing keys.

  5. Backfilling the new field changes historical keys. If you run a job to populate confidence on stored events, and anything recomputes keys from stored records, those records now hash differently from the live stream. Backfill the value but never recompute the key — the key was assigned at ingest and is part of the event’s identity, as Replaying Dead-Letter Spatial Events Safely describes.

Three deploys, and only the first one is optional to notice The change lands in three deploys. First the identity list is introduced on its own, with no schema change at all: the key function stops hashing the whole payload and starts hashing named fields, and because no field has been added yet the digests it produces are identical to the ones already in the deduplication store, so the deploy is a no-op on the wire and can be verified against live traffic before anything depends on it. Second, consumers gain the new field and the version that carries it, while producers still emit the old version. Third, producers begin emitting the field. Only after the identity list is in place is the third deploy safe, which is why it has to be separated out rather than bundled — bundling means the deduplication break and the field addition ship together, and the first is diagnosed as the second. 1 · identity list only no schema change · no new field digests are identical to those already in the store — a verifiable no-op 2 · readers gain the field consumers understand the new version producers still emit the old one still nothing on the wire 3 · writers emit it safe only because step 1 landed the key cannot see the new field, so dedup keeps matching throughout What bundling steps 1 and 3 costs you The deduplication break and the field addition ship in the same release, so the symptom — duplicate writes appearing across the fleet — is attributed to the field, which is the one part of the change that was never the problem. Separating them makes step 1 independently verifiable against live traffic, which is the only point in the sequence where that is possible.
Figure 2. Landing the identity list as its own deploy is what makes it verifiable: with no schema change alongside it, any digest that moves is a bug in the key function rather than an expected consequence of the field.
Absent is not zero, and a filter can tell A downstream consumer filters out readings whose confidence is below 0.2. If events produced before the field existed are defaulted to 0.0, every one of them fails that filter and the historical record disappears from the consumer's view — silently, because a filter that excludes data raises nothing. If they carry NaN instead, the comparison is false in both directions, so the consumer must handle them explicitly and the choice becomes visible in code review. If they carry a separate measured flag, the consumer can branch on whether a measurement exists before comparing values at all. Only the first option loses information, and it loses it in the direction that is hardest to notice. Downstream filter: keep readings where confidence >= 0.2 default 0.0 0.0 >= 0.2 is False every pre-rollout event is filtered out and a filter that excludes data raises nothing at all NaN nan >= 0.2 is False, and so is nan < 0.2 the consumer must handle it explicitly so the decision surfaces in review rather than in a dashboard, later separate measured flag branch on existence before comparing no value can be mistaken for a measurement costs one boolean per event, and makes the ambiguity impossible to express Only the first loses information, and it loses it in the direction hardest to notice Nothing errors, no count drops to zero, and the events are still in the store. They have simply stopped reaching one consumer, which is the kind of failure that is normally found months later by someone asking why a historical chart starts on the day of a deploy.
Figure 3. The default value you pick for old events is an assertion about them. Zero asserts a measurement that was never taken; only the third option makes that assertion impossible to write by accident.

Verification

python 
import math

import pytest


def test_added_field_does_not_move_the_key():
    """The property the whole change depends on."""
    before = {"feature_id": 4471, "occurred_at": "2026-08-08T09:14:00Z",
              "geometry": {"type": "Point", "coordinates": [13.4049547, 52.5200087]}}
    after = before | {"confidence": 0.87}

    assert identity_key(before) == identity_key(after)


def test_absent_confidence_is_not_zero():
    """A pre-rollout event must not claim a measurement it never had."""
    assert math.isnan(confidence_of({"feature_id": 1}))
    assert confidence_of({"feature_id": 1, "confidence": 0.0}) == 0.0


def test_identity_field_change_is_detected():
    """The gate's negative case: the key must be sensitive to identity.

    A key that ignored everything would satisfy the first test perfectly, so
    this asserts the digest still moves when the geometry genuinely changes.
    """
    a = {"feature_id": 4471, "occurred_at": "2026-08-08T09:14:00Z",
         "geometry": {"type": "Point", "coordinates": [13.4049547, 52.5200087]}}
    b = a | {"geometry": {"type": "Point", "coordinates": [13.5, 52.5200087]}}
    assert identity_key(a) != identity_key(b)

The third test is the one that earns its place. Without it, a key function that hashed a constant would pass the first two and destroy deduplication entirely.

FAQ

Why does adding an optional field break deduplication?

Only if the idempotency key is derived from the whole payload. A canonical serialisation of the full event includes the new field, so the same logical event hashes differently before and after the producer starts sending it. The deduplication store holds the old hashes, the new events produce new ones, nothing matches, and every redelivery across the rollout window is treated as novel and written again. Hashing over an explicit list of identity fields makes the addition invisible to the key.

Should a new numeric field default to zero for old events?

Almost never. Zero is a measurement; absence is the lack of one, and collapsing the second into the first invents data. A confidence of 0.0 asserts the reading is certainly wrong, while a missing confidence says nobody measured it — a consumer filtering on confidence below 0.2 will discard every historical event if you default to zero. Use NaN, None, or a separate presence flag so the difference survives.

Do I need a new schema version just to add a field?

Yes, even though the change is structurally compatible. The version is what lets a consumer know which contract it received, so it can tell a genuinely absent value from one the producer simply had not started sending yet. Without it, an old event and a new event that happens to omit an optional field are indistinguishable, and any logic that treats absence as meaningful has no way to interpret it.