Chunking a Multipolygon Across Message Size Limits

Split only on part boundaries so every chunk is a valid polygon, pack by serialised byte size rather than part count, and give the reassembler a timeout that fails the group — a missing part with no timeout is an in-memory leak and an event that silently never happened.

This guide sits under Streaming & Chunking Large Geometry Payloads, within Spatial Payload Routing & Parsing. That topic argues that chunking is the third-best option; this guide is for the case where it is nonetheless the right one, because the parts are independently useful.

When to use this pattern

  • The geometry is genuinely composite — separate administrative units, distinct islands, a feature collection — so each part means something on its own.
  • Consumers can act on parts as they arrive rather than needing the whole assembled shape, which is where the pattern earns its cost.
  • Object storage is unavailable or undesirable, which rules out the claim check.

If none of those hold, use The Claim-Check Pattern for Oversized Spatial Payloads instead.

Pack by bytes, because parts are not comparable

Ten parts is not a size A national coastline multipolygon holds four hundred parts whose serialised sizes span five orders of magnitude: most are small offshore islands of a few hundred bytes, a handful are substantial islands of tens of kilobytes, and two are mainland coastlines of about twenty megabytes each. A packer that puts ten parts in each chunk produces chunks of about four kilobytes for most of the geometry, which wastes almost the entire message budget on overhead, and one chunk of over forty megabytes when it happens to reach the two mainland parts, which exceeds any broker limit and fails the publish. A byte-aware packer accumulates serialised parts until adding the next would exceed the budget, so chunks are uniformly close to the limit, the mainland parts each get a chunk to themselves, and no chunk is ever over. The count-based failure is particularly unpleasant because it depends on where the large parts fall in the input order, so it can pass in testing on a reordered sample and fail in production on the same data. 400 parts · sizes spanning five orders of magnitude · 1 MB message budget ten parts per chunk 40+ MB — the chunk that reached both mainland parts · publish fails most chunks ≈ 4 KB — the message budget is almost entirely wasted and which chunk overflows depends on input order, so a reordered sample passes in testing pack until the next part would exceed the budget one mainland part each many small parts packed together Serialise each part, accumulate, and start a new chunk before the budget is exceeded. Chunks land uniformly close to the limit and none is ever over.
Figure 1. The count-based failure depends on where the large parts fall in the input order, which is why it survives testing and appears in production on the same data.

Complete runnable implementation

python 
import json
import uuid
from dataclasses import dataclass

from shapely.geometry import mapping, shape
from shapely.ops import unary_union

# Leave room for the envelope, headers and the broker's own overhead.
CHUNK_BUDGET = 700 * 1024


class OversizedPart(Exception):
    """One part alone exceeds the budget — chunking cannot help."""


@dataclass(frozen=True, slots=True)
class Chunk:
    group_id: str
    index: int
    total: int            # declared in EVERY chunk, so loss is detectable
    parts: list[dict]
    bbox: tuple[float, float, float, float]


def chunk_multipolygon(geometry: dict, budget: int = CHUNK_BUDGET) -> list[Chunk]:
    """Split on part boundaries, packing by serialised size.

    Every chunk is a valid MultiPolygon on its own: it can be validated on
    arrival, filtered by bounding box, and a missing chunk is a missing set
    of polygons rather than a corrupt shape.
    """
    geom = shape(geometry)
    parts = [mapping(p) for p in getattr(geom, "geoms", [geom])]

    groups: list[list[dict]] = []
    current: list[dict] = []
    current_size = 0

    for part in parts:
        size = len(json.dumps(part, separators=(",", ":")).encode())
        if size > budget:
            # A single coastline larger than the budget cannot be chunked on
            # part boundaries. Simplify it, tile it, or use a claim check —
            # but do not split its coordinate array.
            raise OversizedPart(f"one part is {size} bytes, budget {budget}")
        if current and current_size + size > budget:
            groups.append(current)
            current, current_size = [], 0
        current.append(part)
        current_size += size

    if current:
        groups.append(current)

    group_id = str(uuid.uuid4())
    total = len(groups)
    return [
        Chunk(group_id=group_id, index=i, total=total, parts=g,
              bbox=unary_union([shape(p) for p in g]).bounds)
        for i, g in enumerate(groups)
    ]

The reassembler is where the correctness lives. It has to be idempotent per index, tolerate arrival in any order, and give up.

python 
import asyncio
import time
from dataclasses import dataclass, field

REASSEMBLY_TIMEOUT = 120.0


class IncompleteGroup(Exception):
    def __init__(self, group_id: str, missing: set[int]) -> None:
        super().__init__(f"{group_id}: missing chunk indices {sorted(missing)}")
        self.group_id, self.missing = group_id, missing


@dataclass(slots=True)
class Partial:
    total: int
    first_seen: float
    chunks: dict[int, list[dict]] = field(default_factory=dict)


class Reassembler:
    def __init__(self, timeout: float = REASSEMBLY_TIMEOUT) -> None:
        self._timeout = timeout
        self._groups: dict[str, Partial] = {}

    def add(self, chunk: Chunk, now: float | None = None) -> dict | None:
        """Return the assembled geometry once complete, else None."""
        now = now or time.monotonic()
        partial = self._groups.setdefault(
            chunk.group_id, Partial(total=chunk.total, first_seen=now)
        )
        # Idempotent per index: a redelivered chunk overwrites itself.
        partial.chunks[chunk.index] = chunk.parts

        if len(partial.chunks) < partial.total:
            return None

        del self._groups[chunk.group_id]
        parts = [p for i in sorted(partial.chunks) for p in partial.chunks[i]]
        return {"type": "MultiPolygon",
                "coordinates": [p["coordinates"] for p in parts]}

    def expire(self, now: float | None = None) -> list[IncompleteGroup]:
        """Fail groups that have waited too long. MUST be called on a timer.

        Without this the reassembler accumulates partial groups forever, and
        because each holds real geometry the leak is measured in megabytes.
        """
        now = now or time.monotonic()
        failures = []
        for group_id, partial in list(self._groups.items()):
            if now - partial.first_seen > self._timeout:
                missing = set(range(partial.total)) - set(partial.chunks)
                del self._groups[group_id]
                failures.append(IncompleteGroup(group_id, missing))
        return failures
Three outcomes, and only one of them is a surprise Three groups are traced through the reassembler. The first arrives complete but out of order — chunk two, then zero, then one — and because chunks are stored in a dictionary keyed by index and sorted at assembly time, the order of arrival is irrelevant and the group completes correctly. The second includes a redelivery: chunk one arrives twice, and because storing by index overwrites rather than appends, the duplicate changes nothing and the assembled geometry is identical. The third loses chunk two entirely, perhaps because the producer crashed mid-publish. The group sits at two of three parts held, and without an expiry sweep it sits there permanently, holding its geometry in memory while nothing downstream ever learns the event failed. With the sweep, after the timeout it becomes an IncompleteGroup naming the group and the missing index, which is dead-lettered and can be replayed. The distinction is not between working and broken but between a failure that is visible and one that is a slow leak plus an event that silently never happened. out of order — irrelevant #2 #0 #1 stored by index, sorted at assembly → complete and correct redelivery — absorbed #0 #1 #1 #2 storing by index overwrites → identical result chunk lost — the case that needs the sweep #0 #1 #2 no sweep: held forever, in memory, and nothing learns the event failed with sweep: IncompleteGroup(group, missing={2}) → dead letter, replayable The choice is not between working and broken — it is between a visible failure and a slow leak plus an event that silently never happened.
Figure 2. Out-of-order and redelivered chunks are handled by the data structure. The lost chunk is the only case needing a decision, and the decision is to give up.

Parameter reference

Name Type Spatial constraint Default
CHUNK_BUDGET int Below the broker limit by enough for envelope and headers 716800
total int Declared in every chunk, so loss is detectable from any one —
index int Storage key — makes redelivery idempotent —
bbox tuple Per chunk, so a consumer can filter before assembling —
REASSEMBLY_TIMEOUT float Above the broker’s worst redelivery delay, below patience 120.0
expire() call Must run on a timer, not only on arrival —

Gotchas and spatial edge cases

  1. A single part larger than the budget cannot be chunked this way. One mainland coastline can exceed any reasonable message limit on its own, and OversizedPart is the honest response — the alternatives are simplifying the geometry for transport, tiling the feature, or falling back to a claim check.

  2. Validity must be checked after assembly, never per chunk. Each chunk is a valid MultiPolygon in isolation while the assembled whole can have overlapping parts, which no individual chunk can see. Run the repair from Geometry Validation Pipelines on the assembled geometry only.

  3. The reassembler is per-consumer state, so chunks must reach one consumer. With chunks spread across partitions, each consumer holds a fraction of every group and none completes. Key every chunk in a group on the group identifier, which also means the group is confined to one partition and inherits its ordering.

  4. Holes belong to their part and must not be separated from it. A polygon’s interior rings are part of its coordinate array, so splitting on part boundaries keeps them together automatically — but a packer written against a flattened ring list will happily put an interior ring in a different chunk, producing a hole that becomes a solid polygon.

  5. Expiry needs a timer, not an arrival hook. A group whose remaining chunks never arrive also never triggers add, so an expiry check that runs only on arrival never runs for exactly the groups it exists to clean up.

  6. Assembly order must come from the index, not the arrival order. Multipolygon part order is not semantically meaningful, but making it depend on network timing means the same event produces different serialisations, which breaks any content hash computed downstream — see Event Key Generation for Spatial Data.

A group split across partitions never completes anywhere A group of six chunks is published without a group-scoped partition key, so the broker distributes them across three partitions by round-robin. Three consumers each receive two chunks of the same group, and each one holds two of six — none reaches the declared total, so none assembles anything. After the reassembly timeout all three independently fail the group and dead-letter their fragments, producing three separate incomplete-group errors for one event and no assembled geometry at all. Keying every chunk in a group on the group identifier sends all six to one partition and therefore to one consumer, which assembles them normally; it also confines the group to a single partition's ordering, so the chunks arrive in a defined order even though the reassembler does not rely on it. The cost is that a very large group is processed by one consumer rather than shared, which is the correct trade because the group cannot be processed by more than one anyway. no group key — six chunks, three partitions #0 #3 → c1 #1 #4 → c2 #2 #5 → c3 each holds 2 of 6 · none reaches the declared total after the timeout: three incomplete-group errors for one event, and no assembled geometry keyed on the group id #0 #1 #2 #3 #4 #5 → one partition, one consumer assembles normally, and inherits that partition's ordering Cost: a very large group is processed by one consumer rather than shared — which is the correct trade, because the group cannot be processed by more than one anyway.
Figure 3. The reassembler is correct in every one of the three consumers. What is wrong is that the group was ever split across them.

Verification

python 
import pytest
from shapely.geometry import MultiPolygon, Polygon, mapping, shape


def islands(n: int) -> dict:
    return mapping(MultiPolygon([
        Polygon([(i, 0), (i, 0.5), (i + 0.5, 0.5), (i + 0.5, 0)]) for i in range(n)
    ]))


def test_every_chunk_is_a_valid_geometry():
    """The property that distinguishes this from coordinate-range splitting."""
    for chunk in chunk_multipolygon(islands(300), budget=2048):
        assembled = {"type": "MultiPolygon",
                     "coordinates": [p["coordinates"] for p in chunk.parts]}
        assert shape(assembled).is_valid


def test_round_trip_preserves_the_geometry():
    original = islands(300)
    chunks = chunk_multipolygon(original, budget=2048)
    r = Reassembler()
    result = None
    for chunk in reversed(chunks):                 # deliberately out of order
        result = r.add(chunk) or result
    assert shape(result).equals(shape(original))


def test_redelivered_chunk_changes_nothing():
    chunks = chunk_multipolygon(islands(60), budget=2048)
    r = Reassembler()
    for chunk in chunks[:-1]:
        r.add(chunk)
        r.add(chunk)                                # redelivery
    assert shape(r.add(chunks[-1])).is_valid


def test_lost_chunk_expires_into_a_named_failure():
    """No timeout means a leak and an event that silently never happened."""
    chunks = chunk_multipolygon(islands(60), budget=2048)
    r = Reassembler(timeout=1.0)
    for chunk in chunks[:-1]:
        r.add(chunk, now=0.0)

    failures = r.expire(now=5.0)
    assert len(failures) == 1
    assert failures[0].missing == {len(chunks) - 1}

The second test reverses the chunk order deliberately. A reassembler that appends rather than storing by index passes every other test here and produces a multipolygon whose parts are in arrival order, which is valid, wrong, and hashes differently every time.