Handling Out-of-Order Pings from Intermittent Devices

Order every decision on the device’s own clock, reject any state transition carrying an observation older than the state it would replace, and hold a track open until a measured watermark has passed — a device leaving a tunnel flushes its buffer in whatever order its firmware happens to use, so arrival order makes the reconstructed path a function of connectivity.

This guide sits under Sensor Data Routing Patterns, within Core Event Fundamentals & Architecture. It is the failure mode that breaks the stateful router in Routing Telemetry by Geofence Membership, which assumes each ping is newer than the last.

When to use this pattern

  • Devices buffer locally when they lose connectivity — vehicles in tunnels, vessels out of range, sensors on intermittent power.
  • Downstream logic is stateful: geofence membership, trip reconstruction, distance accumulation, arrival detection.
  • The device supplies its own timestamp, which is the precondition for any of this working.

What a buffered flush does to stateful logic

The path the network produces is not the path the vehicle drove A vehicle drives into a tunnel at nine minutes past, losing connectivity for eleven minutes while continuing to record positions every ten seconds. On emerging it flushes sixty-six buffered pings within about two seconds, and its firmware sends the newest first. Downstream, a geofence router that trusts arrival order sees the vehicle apparently teleport to its current position, then walk backwards through the tunnel, then jump forward again when live reporting resumes — producing an exit from the destination zone, a re-entry into the origin zone, and a second exit, none of which happened. A billing rule keyed on zone entries charges twice; an arrival notification fires, then un-fires, then fires again. Ordering on the device clock instead reconstructs the drive exactly as it happened, and the only cost is that the router must hold its decision until the flush has been sorted, which is what the watermark is for. 11-minute tunnel · 66 buffered pings flushed in ~2 s, newest first device clock (what happened) 66 pings recorded underground arrival order (what the network delivered) all 66, reversed a router trusting arrival order sees: teleport forward · walk backwards · jump forward again what downstream emits exit destination zone · re-enter origin zone · exit again billing charges twice · arrival fires, un-fires, fires ordering on the device clock reconstructs the drive exactly as it happened costs only the delay of holding the decision — the watermark
Figure 1. Whether the oldest or the newest buffered ping arrives first is a firmware detail that can change in an update, which is why arrival order cannot be the basis of anything.

Complete runnable implementation

Two mechanisms, and they solve different halves. The monotonic guard protects state from going backwards, cheaply and immediately. The watermark buffer produces a correctly ordered track, at the cost of latency.

python 
import heapq
from dataclasses import dataclass, field
from datetime import datetime, timedelta, UTC

from prometheus_client import Counter, Histogram

LATE = Counter("ping_late_total", "Pings older than the current state", ("asset",))
TOO_LATE = Counter("ping_after_watermark_total", "Pings arriving after close")
SKEW = Histogram(
    "ping_arrival_skew_seconds", "Arrival minus device timestamp",
    buckets=(1, 5, 30, 120, 600, 1800, 3600, float("inf")),
)

# Measured, not chosen: the 99th percentile of observed arrival skew.
WATERMARK_LAG = timedelta(minutes=15)
# A device claiming to be further in the future than this has a broken clock.
MAX_FUTURE_SKEW = timedelta(minutes=2)


class ClockImplausible(Exception):
    """The device timestamp cannot be true, so nothing derived from it can be."""


@dataclass(slots=True)
class AssetState:
    latest_observed: datetime | None = None
    buffer: list = field(default_factory=list)     # min-heap on device time


class OrderedIngest:
    def __init__(self, lag: timedelta = WATERMARK_LAG) -> None:
        self._lag = lag
        self._state: dict[str, AssetState] = {}

    def accept(self, ping: dict, now: datetime | None = None) -> bool:
        """Buffer one ping. Returns False if it is too late to be ordered."""
        now = now or datetime.now(UTC)
        observed = datetime.fromisoformat(ping["occurred_at"]).astimezone(UTC)
        SKEW.observe((now - observed).total_seconds())

        # A future timestamp is a broken clock, not a late ping, and the two
        # need opposite handling: one is quarantined, the other is waited for.
        if observed > now + MAX_FUTURE_SKEW:
            raise ClockImplausible(f"{ping['asset_id']} claims {observed.isoformat()}")

        state = self._state.setdefault(ping["asset_id"], AssetState())
        if observed < now - self._lag:
            TOO_LATE.inc()
            return False

        heapq.heappush(state.buffer, (observed, ping["ping_id"], ping))
        return True

    def release(self, now: datetime | None = None):
        """Yield every ping whose device time is older than the watermark.

        Anything still in the buffer could still be overtaken by a later
        arrival, so releasing it early is exactly the bug this fixes.
        """
        now = now or datetime.now(UTC)
        watermark = now - self._lag

        for asset_id, state in self._state.items():
            while state.buffer and state.buffer[0][0] <= watermark:
                observed, _, ping = heapq.heappop(state.buffer)
                yield ping

    def is_regression(self, asset_id: str, observed: datetime) -> bool:
        """The cheap guard: does this ping predate the state it would change?

        Used by stateful consumers that must act immediately and cannot wait
        for the watermark — a geofence router, for instance.
        """
        state = self._state.setdefault(asset_id, AssetState())
        if state.latest_observed is not None and observed <= state.latest_observed:
            LATE.labels(asset=asset_id).inc()
            return True
        state.latest_observed = observed
        return False

The two are used together. A geofence router calls is_regression and skips the transition without waiting; a trip reconstructor consumes from release and gets a correctly ordered track fifteen minutes behind real time.

Immediate and approximate, or delayed and correct Two consumers of the same ping stream are contrasted. The monotonic guard acts immediately: it compares each ping's device timestamp against the newest already seen for that asset and drops anything older, so state can never go backwards and the decision latency is zero. What it cannot do is use the dropped pings — a geofence crossing that happened inside the tunnel is discarded rather than replayed, so the router's membership is right at the current instant and its history has a hole. The watermark buffer waits fifteen minutes, sorts everything that arrived in that period by device time, and releases a correctly ordered track including the tunnel positions, at the cost of every consumer downstream of it being fifteen minutes behind. Neither is the right answer for both consumers, which is why a pipeline usually runs both: live alerting takes the guard, and anything that reconstructs a path, computes a distance or issues a bill takes the buffer. monotonic guard — decide now compare against the newest device time seen drop anything older · zero added latency + state can never go backwards + costs one comparison per ping − the tunnel positions are discarded, not replayed − membership is right now; history has a hole for live alerting and geofence routing watermark buffer — decide later hold 15 minutes, sort by device time, release a min-heap per asset, popped against the watermark + the track is the drive, including the tunnel + distances and durations are computable − every consumer behind it is 15 minutes late − memory grows with fleet size × lag for trip reconstruction, distance and billing Neither is right for both consumers, so a pipeline normally runs both off one stream — the guard in front of anything that alerts, the buffer in front of anything that bills.
Figure 2. Running both is not redundancy. They answer different questions, and a pipeline that picks one has silently decided that either its alerts are late or its bills are wrong.

Parameter reference

Name Type Spatial constraint Default
WATERMARK_LAG timedelta ~99th percentile of measured arrival skew, not the maximum 15 min
MAX_FUTURE_SKEW timedelta Above NTP drift, below any plausible buffering delay 2 min
Heap key (datetime, str) Ping id breaks ties so two identical timestamps sort deterministically —
is_regression bool Uses <=, so a repeated timestamp is a duplicate, not an update —
Buffer memory — Grows as fleet size × ping rate × lag; bound it explicitly —
Late path — Emit corrections; do not silently rewrite a closed track —

Gotchas and spatial edge cases

  1. A wrong clock is not a late ping and needs the opposite handling. A device whose clock is a year fast has every ping released immediately and permanently blocks the monotonic guard — nothing newer can ever arrive. A device a year slow has every ping discarded as too late. Both look like data loss; only the ClockImplausible check distinguishes them from a genuine outage, and the correct response is to quarantine the device, not the stream.

  2. Interpolating across a gap invents positions. It is tempting to draw a straight line across the tunnel, and the line will cross geofences the vehicle never entered. If the reconstructed track is used for anything consequential, mark the gap explicitly and let each consumer decide, rather than producing a track that cannot be distinguished from observed data.

  3. Distance accumulated in arrival order is roughly double. Summing consecutive distances over a reversed burst counts every leg twice — once forwards and once backwards — so an odometer built naively over an intermittent fleet overstates by the length of every outage. This is usually the first symptom anyone notices.

  4. Deduplication and ordering are different problems with the same smell. A device that retries its flush sends the same pings again; that is Time-Windowed Deduplication for Moving Assets, and it must run before the heap or the buffer fills with duplicates that all sort to the same position.

  5. The watermark must advance on wall-clock time, not on arrivals. If it advances only when a ping arrives, an asset that goes quiet never releases its buffer, and the last pings before an outage sit in memory until the device comes back — which for a vessel can be weeks. Drive release from a timer.

  6. Per-asset buffers make this stateful, so partition by asset. The same constraint as the geofence router: two instances each holding half an asset’s pings each produce half a track.

The odometer doubles, and only for the vehicles that lose signal A vehicle drives four kilometres through a tunnel while buffering, then flushes in reverse. A distance accumulator that sums consecutive positions in arrival order walks the route backwards and then forwards again, counting every leg twice, so the trip is reported as roughly eight kilometres. Accumulated in device-clock order the same pings give four. The error is not random noise that averages out across a fleet: it is a systematic overstatement proportional to the length of every connectivity gap, so it lands entirely on the vehicles that drive through tunnels, car parks and rural dead zones, and not at all on the ones that stay connected. Any figure derived from the odometer inherits that bias — fuel efficiency per vehicle, distance-based billing, maintenance intervals — and the affected vehicles are exactly the ones whose data is hardest to sanity-check, because their tracks legitimately have gaps in them. one 4 km tunnel transit, accumulated two ways arrival order every leg counted twice → ≈ 8 km reported device-clock order 4 km — the distance the vehicle drove Not noise that averages out: a systematic overstatement proportional to every connectivity gap, landing entirely on the vehicles that drive through tunnels and dead zones — and their tracks legitimately have gaps, so it is hard to sanity-check.
Figure 3. This is usually the first symptom anyone notices, and it points at the ordering rather than at the odometer — which is where the investigation normally starts.

Verification

python 
from datetime import datetime, timedelta, UTC
import pytest

T0 = datetime(2026, 8, 8, 12, 0, tzinfo=UTC)


def ping(asset: str, offset_s: int, n: int) -> dict:
    return {"asset_id": asset, "ping_id": f"p{n}",
            "occurred_at": (T0 + timedelta(seconds=offset_s)).isoformat()}


def test_reversed_burst_is_released_in_device_order():
    """The property the whole design exists for."""
    ingest = OrderedIngest(lag=timedelta(minutes=5))
    now = T0 + timedelta(minutes=20)
    for n, offset in enumerate(reversed(range(0, 600, 10))):     # newest first
        ingest.accept(ping("veh-1", offset, n), now=now)

    released = [p["occurred_at"] for p in ingest.release(now=now)]
    assert released == sorted(released)


def test_guard_rejects_a_ping_older_than_current_state():
    ingest = OrderedIngest()
    assert ingest.is_regression("veh-2", T0 + timedelta(seconds=60)) is False
    assert ingest.is_regression("veh-2", T0 + timedelta(seconds=30)) is True


def test_future_clock_is_quarantined_not_buffered():
    """A year-fast device would otherwise block the guard forever."""
    ingest = OrderedIngest()
    with pytest.raises(ClockImplausible):
        ingest.accept(ping("veh-3", 365 * 24 * 3600, 0), now=T0)

The first test is the one to run against a real recorded burst rather than a synthetic reversal. Firmware rarely reverses cleanly — a partial flush interleaved with live reporting produces an order that is neither forwards nor backwards, and a heap handles it while any “detect and reverse” shortcut does not.