Tuning an IoU Threshold for Sensor Coverage

Sweep the threshold across labelled pairs and pick the point where the cost of a false merge equals the cost of a missed one, then band it by footprint size — a fixed positional error costs far more IoU on a small footprint than on a large one, so one threshold treats small overlaps as distinct and large ones as duplicates.

This guide sits under Spatial Overlap Deduplication, within Idempotency & Spatial Deduplication. That topic covers the four-layer pipeline; this guide covers only the number at its centre.

When to use this pattern

  • Overlap deduplication is already running with a threshold nobody can justify.
  • The two errors have visibly different costs — merged observations that should not have been, or redundant records that should have collapsed.
  • Footprints vary in size by more than about a factor of five, which is where a single threshold stops working.

IoU is not scale-free

The same error, two very different scores The same ten-metre registration offset is applied to two footprints. On the left, a fifty-metre sensor cone shifted ten metres retains an intersection-over-union of about zero point six seven — well below a threshold of zero point eight, so the two readings are classified as distinct observations even though they are the same sensor seeing the same thing with ordinary positional error. On the right, a five-kilometre swath shifted by the identical ten metres scores about zero point nine nine six, comfortably above the threshold, so those two are merged. Both classifications come from one number applied to one measure, and both are wrong in opposite directions: the small pair should have merged and the large pair should perhaps not have, since ten metres of drift on a five-kilometre swath tells you nothing about whether the two passes observed the same event. The correct response is to band the threshold by footprint size, lowering it for small geometries where a fixed positional error dominates the score and raising it for large ones where it barely registers. one 10 m registration offset · one threshold of 0.80 50 m sensor cone, offset 10 m IoU ≈ 0.67 below 0.80 → classified DISTINCT but it is the same sensor seeing the same thing, with ordinary error a fixed offset dominates the score at this scale 5 km swath, offset the same 10 m IoU ≈ 0.996 → classified DUPLICATE 10 m of drift on 5 km says nothing about whether the passes saw the same event Band the threshold by footprint size: lower for small geometries where a fixed offset dominates, higher for large ones where it barely registers.
Figure 1. Both classifications are wrong, in opposite directions, from one number applied uniformly. Scale is not a detail of the measure; it is the measure's main sensitivity.

Complete runnable implementation

python 
from dataclasses import dataclass

from shapely.geometry import shape
from shapely.validation import make_valid

# Footprint area bands in square metres, each with its own threshold. The
# bands come from the fleet's sensor types, not from round numbers.
SIZE_BANDS = (
    (0.0, 1e4, "small"),        # < 1 hectare — a fixed offset dominates IoU
    (1e4, 1e7, "medium"),
    (1e7, float("inf"), "large"),
)


def iou(a, b) -> float:
    """Intersection over union. Zero for disjoint, one for identical."""
    a, b = make_valid(a), make_valid(b)
    union = a.union(b).area
    return 0.0 if union == 0 else a.intersection(b).area / union


def containment(a, b) -> float:
    """Intersection over the SMALLER area — catches the nested case.

    Two readings where one covers a quarter of the other cap out at an IoU
    of 0.25 however well they agree over the shared ground, so a narrow pass
    inside a wide swath scores as distinct on IoU alone.
    """
    smaller = min(a.area, b.area)
    return 0.0 if smaller == 0 else a.intersection(b).area / smaller


def band_of(a, b) -> str:
    mean_area = (a.area + b.area) / 2.0
    for low, high, name in SIZE_BANDS:
        if low <= mean_area < high:
            return name
    return "large"


@dataclass(frozen=True, slots=True)
class Operating:
    threshold: float
    precision: float
    recall: float
    cost: float


def sweep(labelled: list[tuple[object, object, bool]],
          false_merge_cost: float, missed_merge_cost: float,
          steps: int = 40) -> list[Operating]:
    """Score every candidate threshold against labelled pairs.

    `labelled` is (geometry_a, geometry_b, is_duplicate) from real data —
    a synthetic set of translated squares produces a clean curve and a
    threshold that does not survive contact with sensor footprints.
    """
    scores = [(iou(a, b), truth) for a, b, truth in labelled]
    results = []

    for step in range(1, steps + 1):
        threshold = step / steps
        tp = sum(1 for s, t in scores if s >= threshold and t)
        fp = sum(1 for s, t in scores if s >= threshold and not t)
        fn = sum(1 for s, t in scores if s < threshold and t)

        precision = tp / (tp + fp) if tp + fp else 1.0
        recall = tp / (tp + fn) if tp + fn else 1.0
        # Weighted cost, NOT F1: F1 assumes the two errors are equally bad,
        # which for a sensor archive they are emphatically not.
        results.append(Operating(
            threshold=threshold,
            precision=precision,
            recall=recall,
            cost=fp * false_merge_cost + fn * missed_merge_cost,
        ))
    return results


def choose(results: list[Operating]) -> Operating:
    return min(results, key=lambda r: (r.cost, -r.threshold))


def is_duplicate(a_geojson: dict, b_geojson: dict,
                 thresholds: dict[str, float]) -> bool:
    """Apply the tuned threshold for this pair's size band."""
    a, b = make_valid(shape(a_geojson)), make_valid(shape(b_geojson))
    threshold = thresholds[band_of(a, b)]
    # Either measure passing is enough: IoU for similar sizes, containment
    # for the nested case it cannot represent.
    return iou(a, b) >= threshold or containment(a, b) >= 0.95

Minimising a weighted cost rather than maximising F1 is the decision that makes this tuning rather than curve-fitting. F1 asserts the two errors are equally expensive, which is almost never true and is never checked.

The cost minimum is not where F1 puts it Precision and recall are plotted against the intersection-over-union threshold across a labelled set of sensor footprint pairs. Recall falls as the threshold rises, because fewer genuine duplicates clear the bar; precision rises, because fewer distinct pairs are wrongly merged. The F1 optimum sits where the two curves cross, near a threshold of zero point six five. The weighted-cost minimum sits considerably higher, near zero point eight two, because in this archive a false merge destroys an observation that will never be repeated while a missed merge merely leaves a redundant record that a later pass can still collapse — the first error is weighted twenty times the second. Choosing the crossing point would have merged roughly three times as many distinct observations, and none of those merges would have produced an error, a log line or a metric: the destroyed reading simply would not be in the archive, and nothing in the system knows it should have been. precision and recall against IoU threshold · labelled sensor pairs 0.0 0.5 1.0 recall precision F1 optimum ≈ 0.65 — where the curves cross weighted-cost minimum ≈ 0.82 error weights false merge ×20 destroys a reading nothing repeats missed merge ×1 a redundant record a later pass collapses The crossing point would merge ~3× as many distinct observations — with no error, no log line and no metric: the reading simply is not there.
Figure 2. F1 is the right objective only when the two errors cost the same. Writing the weights down is what turns "0.8 seems fine" into a decision somebody can argue with.

Parameter reference

Name Type Spatial constraint Default
SIZE_BANDS tuple Areas in m²; derived from sensor types, not round numbers 3 bands
false_merge_cost float Relative cost of destroying a distinct observation 20.0
missed_merge_cost float Relative cost of a redundant record a later pass can collapse 1.0
Containment threshold float For the nested case IoU cannot express 0.95
Labelled pairs list Real footprints; a few hundred per band minimum —
Area CRS — Equal-area projection; areas in EPSG:4326 are not comparable —

Gotchas and spatial edge cases

  1. Areas computed in EPSG:4326 are in square degrees and are not comparable across latitudes. A footprint at 60° north has roughly half the ground area of an identically-shaped one at the equator, so a size band expressed in degrees puts them in different bands. Reproject to an equal-area CRS before measuring anything the bands depend on.

  2. Invalid geometries make intersection unreliable rather than failing. A self-intersecting footprint can produce an intersection area larger than either input, so IoU exceeds one. make_valid on both inputs is not optional, and the repair itself is covered in Repairing Self-Intersecting Polygons Without Losing Area.

  3. A synthetic labelled set produces a threshold that does not survive real data. Translated squares give a clean monotone curve; real sensor footprints have ragged edges, partial cloud masks and nested passes, and the curve has a shoulder rather than a crossing. Label real pairs, even if only a few hundred.

  4. The nested case needs containment, not a lower IoU threshold. Lowering the IoU threshold enough to catch a narrow pass inside a wide swath also merges genuinely different neighbouring footprints. The two measures answer different questions and should both be available.

  5. Re-tune after any change to the sensor fleet or its registration pipeline. A new satellite with better positioning shifts the whole IoU distribution upwards, and the old threshold now merges pairs it was never validated against.

  6. Record which threshold and which band produced each merge decision. Without it, a later investigation into a missing observation cannot tell whether the merge was correct under the rules in force at the time, and the tuning becomes unauditable.

A size band in square degrees is a different band at every latitude Two identical sensor footprints are measured, one over the equator and one at sixty degrees north. On the ground they cover the same area. In square degrees they do not: a degree of longitude at sixty degrees north spans half the distance it does at the equator, so the northern footprint's area in square degrees is about half the southern one's. A size band expressed in square degrees therefore puts the two in different bands and applies different thresholds to identical sensors — and because most fleets operate over a range of latitudes, the effect is a systematic drift in deduplication behaviour from south to north that no amount of threshold tuning will fix, because the bands themselves are moving. Reprojecting to an equal-area coordinate reference system before measuring makes the two footprints the same size, which is what the bands assume. two identical footprints, measured in square degrees at the equator band: medium at 60° north — same ground area band: small, and a different threshold why tuning cannot fix it a degree of longitude at 60° N spans half its equatorial distance, so the area halves the bands themselves move with latitude reproject to an equal-area CRS before measuring Over a fleet spanning latitudes this is a systematic drift in deduplication behaviour from south to north, and it presents as a threshold that "works in one region and not another" — which sends the investigation to the sensors rather than to the units.
Figure 3. The bands assume equal ground area. Measuring in degrees breaks that assumption quietly, and the symptom looks like a sensor problem.

Verification

python 
import pytest
from shapely.geometry import Point, box

THRESHOLDS = {"small": 0.55, "medium": 0.75, "large": 0.88}


def test_small_footprint_offset_still_merges():
    """The case a flat 0.8 threshold gets wrong."""
    a = Point(0, 0).buffer(25)          # 50 m cone
    b = Point(10, 0).buffer(25)         # same cone, 10 m registration error
    assert iou(a, b) < 0.8              # would fail a flat threshold
    assert is_duplicate(a.__geo_interface__, b.__geo_interface__, THRESHOLDS)


def test_large_footprint_offset_is_not_merged_by_accident():
    """The band must be strict where a fixed offset barely registers."""
    a, b = box(0, 0, 5000, 1000), box(3000, 0, 8000, 1000)
    assert not is_duplicate(a.__geo_interface__, b.__geo_interface__, THRESHOLDS)


def test_nested_pass_is_caught_by_containment():
    """IoU caps at 0.25 here however well the pair agrees."""
    wide, narrow = box(0, 0, 4000, 4000), box(1000, 1000, 3000, 3000)
    assert iou(wide, narrow) < 0.3
    assert is_duplicate(wide.__geo_interface__, narrow.__geo_interface__, THRESHOLDS)


def test_weighted_cost_picks_a_higher_threshold_than_f1():
    """The decision the sweep exists to make."""
    labelled = _labelled_pairs()                      # real, from the archive
    balanced = choose(sweep(labelled, 1.0, 1.0))
    asymmetric = choose(sweep(labelled, 20.0, 1.0))
    assert asymmetric.threshold > balanced.threshold

The last test is a property of the tuning process rather than of any particular number, so it keeps holding as the archive grows — and it fails immediately if someone replaces the weighted cost with F1 while tidying the code.