Sizing a Deduplication Window from Report Intervals

Measure the inter-arrival distribution per device class and take a low percentile of it — the vendor’s nominal interval describes one mode of a multi-modal distribution, and a window sized from the mean is dragged above every real interval by a handful of multi-hour outage gaps.

This guide sits under Time-Windowed Deduplication for Moving Assets, within Idempotency & Spatial Deduplication. It produces the MIN_SECONDS threshold used by Deduplicating Vehicle Pings in a Sliding Window.

When to use this pattern

  • The window was chosen from a configuration document, a vendor datasheet or a round number, and nobody has checked it against traffic.
  • The fleet is heterogeneous — different tracker models, different firmware, different power profiles.
  • The suppression rate is either surprisingly low, which means the window is doing nothing, or surprisingly high, which means it is discarding data someone downstream is waiting for.

The distribution is not a number

One nominal interval, four modes Inter-arrival gaps are plotted for a fleet of trackers whose datasheet says ten seconds. There are four distinct clusters. A tall spike near one to two seconds is the motion trigger firing during acceleration and cornering, which is a large share of all events. A second, taller spike sits at ten seconds, the configured idle cadence. A third cluster around thirty seconds comes from devices in power-saving mode. A long thin tail runs from several minutes to several hours and is the reconnection gaps after tunnels, garages and dead zones. The mean of this distribution lands near ninety seconds, above every mode, because the tail drags it there — a window sized on the mean suppresses nearly the entire stream. The median lands at ten seconds, which suppresses roughly half of all normal reporting. The tenth percentile lands near two seconds, which collapses the motion-trigger burst while leaving the idle cadence intact, and that is what the window is for. inter-arrival gaps · nominal interval 10 s · log scale 1–2 s motion trigger 10 s idle cadence 30 s power save minutes to hours — reconnection after tunnels and garages p10 ≈ 2 s — the window median 10 s — suppresses half of normal reporting mean ≈ 90 s — above every mode The mean is not merely imprecise here — it is outside the data A handful of multi-hour reconnection gaps pull it above every real interval, so a window sized on it discards nearly the whole stream.
Figure 1. No single number summarises this. The window is a threshold between the modes you want to collapse and the ones you want to keep, which makes it a percentile question rather than an average.

Complete runnable implementation

python 
import collections
import statistics
from dataclasses import dataclass

# The window sits above the gaps to collapse and below the gaps to keep.
# A low percentile puts it there; the median puts it in the middle of the
# idle cadence, suppressing about half of all normal reporting.
WINDOW_PERCENTILE = 10

# Reconnection bursts are not reporting cadence and must not shape the window.
BURST_FLOOR_SECONDS = 0.4
# Gaps above this are outages, and they are what drags the mean off the chart.
OUTAGE_CEILING_SECONDS = 600.0


@dataclass(frozen=True, slots=True)
class WindowRecommendation:
    device_class: str
    samples: int
    p10: float
    median: float
    mean: float
    window_seconds: float
    suppression_estimate: float


def inter_arrival_gaps(pings: list[tuple[str, float]]) -> dict[str, list[float]]:
    """Gaps per asset, then pooled per device class.

    Sorting per asset matters: pooling raw timestamps across a fleet produces
    gaps between different vehicles, which is not an interval at all.
    """
    by_asset: dict[str, list[float]] = collections.defaultdict(list)
    for asset_id, epoch in pings:
        by_asset[asset_id].append(epoch)

    gaps: list[float] = []
    for timestamps in by_asset.values():
        timestamps.sort()
        gaps += [b - a for a, b in zip(timestamps, timestamps[1:])]
    return gaps


def recommend(device_class: str, gaps: list[float]) -> WindowRecommendation:
    """Derive a window for one device class from its own gaps."""
    usable = [g for g in gaps
              if BURST_FLOOR_SECONDS <= g <= OUTAGE_CEILING_SECONDS]
    if len(usable) < 1000:
        raise ValueError(
            f"{device_class}: {len(usable)} usable gaps — too few to size a "
            "window; a recommendation from a small sample is a guess with a "
            "decimal point on it"
        )

    usable.sort()
    quantiles = statistics.quantiles(usable, n=100, method="inclusive")
    p10 = quantiles[WINDOW_PERCENTILE - 1]

    # What fraction of real reports this window would suppress. This is the
    # number to take to the consumer, because it is the one they feel.
    suppressed = sum(1 for g in usable if g < p10) / len(usable)

    return WindowRecommendation(
        device_class=device_class,
        samples=len(usable),
        p10=p10,
        median=statistics.median(usable),
        mean=statistics.fmean(usable),
        window_seconds=round(p10, 1),
        suppression_estimate=round(suppressed, 4),
    )


def window_for(device_class: str, table: dict[str, float],
               default: float = 60.0) -> float:
    """Look the window up per event; do not configure one number per fleet."""
    return table.get(device_class, default)

The ValueError on a small sample is not defensive padding. A window derived from two hundred gaps looks exactly like one derived from two million, and the difference only shows up as an unexplained change in suppression rate weeks later.

One number cannot fit three cadences Three device classes share a pipeline: vehicle trackers reporting about every ten seconds, trailer tags reporting about every five minutes, and container seals reporting about every hour. A single fleet-wide window of sixty seconds is applied to all three. For the vehicle trackers it suppresses most reports, which may be intended. For the trailer tags it suppresses nothing at all, because their gaps are already five times the window, so the deduplication stage is pure overhead on that class. For the container seals it is even more irrelevant. Worse, tuning the single window to do something useful for the trailer tags would require raising it to several minutes, which would discard almost every vehicle ping. The classes are not close enough for a compromise to exist, so the window has to be a lookup keyed on a class carried in the event envelope — which also means a new device model arriving without a class falls back to a documented default rather than silently inheriting a number chosen for something else. one 60 s window across three device classes 60 s window vehicle tracker · ~10 s most reports suppressed — possibly intended trailer tag · ~5 min nothing suppressed — pure overhead container seal · ~1 h even less relevant No compromise exists, so the window is a lookup keyed on a class in the envelope A new model arriving without a class then falls back to a documented default, rather than silently inheriting a number chosen for something else.
Figure 2. Raising the window to help the trailer tags would discard almost every vehicle ping. The classes are too far apart for one number, which is a measurement result rather than a preference.

Parameter reference

Name Type Spatial constraint Default
WINDOW_PERCENTILE int Low — above the burst mode, below the idle cadence 10
BURST_FLOOR_SECONDS float Excludes reconnection floods, which are not cadence 0.4
OUTAGE_CEILING_SECONDS float Excludes tunnels and garages; these are what wreck the mean 600.0
Minimum sample int Below this, raise rather than recommend 1000
Class table dict[str, float] One entry per device class differing by more than ~2× —
Re-measurement schedule Quarterly, and after any firmware rollout —

Gotchas and spatial edge cases

  1. Pool gaps per asset, never across the fleet. Sorting every timestamp in the stream and differencing produces gaps between different vehicles, which for a large fleet are milliseconds apart and would recommend a window of essentially zero. The bug is easy to write and the output looks plausible.

  2. A firmware rollout changes the distribution overnight. Devices that gain a motion trigger start producing a whole new mode, and the window sized last quarter now sits in the middle of it. Alert on a shift in the tenth percentile rather than waiting for someone to notice the suppression rate moved.

  3. The suppression estimate is what the consumer cares about, not the window. Take “this discards 38% of reports for this class” to the team consuming the stream, because that is the number they can evaluate. A window in seconds means nothing to someone computing dwell time.

  4. Devices in a depot dominate the sample. Parked vehicles report indefinitely and moving ones do not, so a naive sample is mostly stationary reporting. If the window is meant to govern moving assets, filter the sample by displacement first — which links this measurement to the distance threshold in the sliding window.

  5. A class with too few devices cannot be sized, only defaulted. Rather than computing a confident-looking number from three hundred gaps, fall back to the fleet default and record that the class is unsized. The ValueError above is what makes that visible.

  6. The window is not the TTL. They are independent numbers and conflating them is the failure described in Expiring Deduplication Keys Without Losing Late Retries — this measurement sizes only the first.

A firmware rollout moves the window under your feet The tenth percentile of the inter-arrival distribution is tracked across a firmware rollout. Before the rollout the fleet reports on a fixed ten-second cadence, so the tenth percentile sits near nine seconds and a window sized just under it suppresses almost nothing — which is correct, because there is almost nothing to suppress. Over the rollout week, devices gain a motion trigger that fires during acceleration and cornering, adding a whole new mode of one-to-two-second gaps to the distribution. The tenth percentile falls to about two seconds while the configured window stays at nine, so the window now sits above the entire new mode and suppresses every motion-triggered report — which is roughly a third of the fleet's events, discarded silently, including the ones describing the sharpest manoeuvres. Nothing errors and no alert fires, because a deduplication window doing more work looks identical to one doing the right amount. Alerting on a shift in the measured percentile catches it in days rather than at the next audit. 10th percentile of inter-arrival gaps, across a firmware rollout 10 s 0 configured window — unchanged fixed 10 s cadence firmware rollout adds a motion trigger p10 falls to ~2 s · the window now sits above the whole new mode Roughly a third of the fleet's events are now suppressed — including the reports describing the sharpest manoeuvres. Nothing errors: a window doing too much work looks exactly like one doing the right amount. Alert on the percentile, not the outcome.
Figure 3. The window did not change and the fleet did. Watching the measured percentile rather than the configured value is what turns that into a days-long problem instead of an audit finding.

Verification

python 
import random
import pytest


def synthetic_gaps(n: int = 50_000) -> list[float]:
    """Four modes, matching a real tracker: burst, idle, power-save, outage."""
    gaps = []
    gaps += [random.gauss(1.6, 0.4) for _ in range(int(n * 0.30))]
    gaps += [random.gauss(10.0, 1.2) for _ in range(int(n * 0.45))]
    gaps += [random.gauss(30.0, 4.0) for _ in range(int(n * 0.20))]
    gaps += [random.uniform(600, 7200) for _ in range(int(n * 0.05))]
    return [g for g in gaps if g > 0]


def test_window_sits_between_the_burst_and_idle_modes():
    rec = recommend("tracker-v3", synthetic_gaps())
    assert 0.8 < rec.window_seconds < 8.0


def test_mean_is_outside_every_mode():
    """The reason the mean must not be used."""
    rec = recommend("tracker-v3", synthetic_gaps())
    assert rec.mean > 40.0                     # above idle AND power-save
    assert rec.window_seconds < rec.median


def test_small_sample_raises_rather_than_guessing():
    with pytest.raises(ValueError, match="too few"):
        recommend("new-model", [10.0] * 50)


def test_gaps_are_computed_per_asset():
    """Two vehicles reporting alternately must not produce tiny gaps."""
    pings = []
    for i in range(100):
        pings.append(("veh-a", i * 10.0))
        pings.append(("veh-b", i * 10.0 + 0.05))   # interleaved, 50 ms apart
    gaps = inter_arrival_gaps(pings)
    assert min(gaps) > 5.0, "gaps were pooled across assets"

The last test is the one that catches the most common implementation mistake, and it fails loudly: a fleet-pooled computation on that input returns gaps of fifty milliseconds and would recommend a window that suppresses nothing at all.