Partitioning Kafka Topics by H3 Cell

To partition a Kafka topic by H3 cell: compute the centroid of each GeoJSON geometry in EPSG:4326 (WGS84), index it to a deliberately coarse H3 cell with h3.latlng_to_cell, and use that cell id as the aiokafka message key so Kafka’s murmur2 partitioner co-locates every event in the region on one partition and preserves per-cell ordering. The coarse cell is the routing key only — you keep a finer H3 index in the payload for analytics.

This page sits inside Broker Selection & Partitioning for Spatial Streams, part of the Queue Management, Retries & Delivery Guarantees architecture — the reference section for moving spatial events through brokers without losing order or overwhelming a single partition.

When to use this pattern

Reach for H3-keyed partitioning when:

• Your consumers depend on per-region ordering — for example a stateful geofence engine or map-tile invalidator that must apply events for one area in the sequence they occurred, but does not care about global ordering across the whole planet.
• Spatially-adjacent events should be processed by the same consumer instance to exploit warm caches, shared spatial indexes, or per-region rate limits.
• You want a deterministic, stateless routing rule — any producer can compute the same partition for the same location without coordinating.

It is not the right tool when every event must be globally ordered (use a single partition and accept the throughput ceiling), or when your workload has no locality at all and you simply want even load — in that case a random or round-robin key spreads traffic more evenly than any spatial scheme.

Why the partitioning resolution differs from the analytics resolution

Kafka guarantees ordering only within a partition, and it assigns a message to a partition by hashing the record key. So the key you choose is the unit of ordering and co-location. If you key on a fine-grained H3 cell (resolution 9, edge ~174 m), a single city block becomes its own key: a busy street splinters across dozens of partitions, adjacent events lose their shared order, and consumers can no longer keep regional state warm.

The fix is to decouple the two roles H3 plays. Use a coarse resolution for the partition key so a whole neighbourhood or metro district collapses into one cell and one partition, and keep a fine resolution in the payload for downstream aggregation and heat-mapping. H3’s hierarchy makes this cheap: a fine cell is always fully contained by exactly one coarse parent, so h3.cell_to_parent recovers the routing cell from the analytics cell without touching the original coordinates. Choosing the index family itself — H3 versus alternatives — is covered in H3 vs S2 vs Quadkey for spatial partitioning; this page assumes you have already settled on H3.

Complete runnable implementation

The producer below is self-contained. It takes a GeoJSON feature, derives a centroid in EPSG:4326 (WGS84), indexes it to a coarse H3 cell, and produces to Kafka with that cell id as the key. It uses h3 (v4 API), shapely for the centroid, and aiokafka for the async producer. The key is encoded as UTF-8 bytes so Kafka’s default murmur2 partitioner hashes it deterministically.

import asyncio
import json
from typing import Any

import h3                          # h3 v4.x
from shapely.geometry import shape  # shapely 2.x
from aiokafka import AIOKafkaProducer

# --- Configuration -----------------------------------------------------------
# Coarse resolution used ONLY for the partition key. Res 5 ~= 8.5 km edge,
# roughly a metro district: fine enough for locality, coarse enough that
# a whole neighbourhood shares one partition and stays ordered together.
PARTITION_RESOLUTION = 5
# Finer resolution carried in the payload for downstream analytics/heatmaps.
ANALYTICS_RESOLUTION = 9
TOPIC = "spatial-events"
BOOTSTRAP = "localhost:9092"


def geometry_centroid_latlng(geometry: dict) -> tuple[float, float]:
    """
    Return (lat, lng) of a GeoJSON geometry's centroid in EPSG:4326 (WGS84).

    h3.latlng_to_cell expects latitude first, longitude second, whereas
    GeoJSON coordinate arrays are [lng, lat] per RFC 7946 — so we swap.
    For a MultiPolygon, shapely's .centroid is the area-weighted centroid of
    all parts, which can fall in open water between islands; see the gotchas.
    """
    geom = shape(geometry)          # accepts any RFC 7946 geometry dict
    c = geom.centroid               # a shapely Point in the same CRS (EPSG:4326)
    return (c.y, c.x)               # (lat, lng)


def partition_key_for(geometry: dict, resolution: int = PARTITION_RESOLUTION) -> str:
    """Coarse H3 cell id (hex string) used as the Kafka partition key."""
    lat, lng = geometry_centroid_latlng(geometry)
    return h3.latlng_to_cell(lat, lng, resolution)


def analytics_cell_for(geometry: dict, resolution: int = ANALYTICS_RESOLUTION) -> str:
    """Fine H3 cell id stored in the payload — never used for partitioning."""
    lat, lng = geometry_centroid_latlng(geometry)
    return h3.latlng_to_cell(lat, lng, resolution)


async def produce_event(producer: AIOKafkaProducer, event: dict[str, Any]) -> None:
    """Serialize an event and produce it keyed by its coarse H3 cell."""
    geometry = event["geometry"]
    key = partition_key_for(geometry)                 # e.g. '85283473fffffff'

    # Enrich the payload with the fine cell so consumers get analytics
    # precision without changing the routing key.
    enriched = {**event, "h3_analytics": analytics_cell_for(geometry)}

    await producer.send_and_wait(
        TOPIC,
        key=key.encode("utf-8"),                      # murmur2 hashes these bytes
        value=json.dumps(enriched).encode("utf-8"),
    )


async def main() -> None:
    # Two distinct fine-grained events in the same coarse cell (both in
    # lower Manhattan) — they must land on the SAME partition.
    feature_a = {
        "type": "Feature",
        "properties": {"device": "sensor-a"},
        "geometry": {"type": "Point", "coordinates": [-74.0060, 40.7128]},
    }
    feature_b = {
        "type": "Feature",
        "properties": {"device": "sensor-b"},
        "geometry": {"type": "Point", "coordinates": [-74.0035, 40.7145]},
    }

    producer = AIOKafkaProducer(bootstrap_servers=BOOTSTRAP)
    await producer.start()
    try:
        await produce_event(producer, feature_a)
        await produce_event(producer, feature_b)
    finally:
        await producer.stop()


if __name__ == "__main__":
    asyncio.run(main())

Parameter reference

Gotchas and spatial edge cases

1. Too-fine a resolution causes partition skew. If the key resolution is finer than your traffic locality, most cells hold one event and co-location collapses; if it is coarse but your traffic is concentrated, a few cells carry everything. Both show up as uneven per-partition throughput — track it with Consumer Lag & Partition Skew Monitoring and tune the resolution against real traffic rather than guessing.

2. Hot partitions over dense cities. A coarse cell over Manhattan or central Tokyo can out-produce every other partition combined. Salt the key for known-hot cells only: f"{cell}:{event_id % N}" fans one cell across N partitions. You trade per-cell ordering for per-sub-key ordering, so only do this where the consumer orders by device/feature id, not by whole region.

3. Boundary-straddling geometries jump cells. A line or polygon that crosses an H3 cell edge has a centroid that can fall on either side depending on tiny coordinate changes, so two near-identical geometries route to different partitions. If stable routing matters more than geometric accuracy, snap to a canonical anchor (e.g. the first vertex) instead of the centroid, and document the choice.

4. MultiPolygon centroid can land in the wrong place. Shapely’s area-weighted centroid of a MultiPolygon (say, an archipelago) can fall in open water between the parts, indexing to a cell that contains none of the geometry. For multipart geometries, prefer the centroid of the largest part, or a representative point (geom.representative_point()), which is guaranteed to lie inside the geometry.

5. Key-to-partition hashing must be deterministic across producers. Kafka’s default partitioner uses murmur2 over the raw key bytes. If one producer sends the cell id as a UTF-8 string and another sends it as a packed 64-bit integer, identical cells hash differently and ordering breaks. Standardise the key encoding (this page uses the lowercase hex string) across every producer.

6. Changing partition count remaps keys. murmur2 is taken modulo the partition count, so growing a topic from 12 to 24 partitions moves most existing cells to new partitions and can interleave old and new orderings during the transition. Provision partition count for peak up front, or use a sticky consumer-side reshard strategy.

7. Null or empty geometry has no centroid. RFC 7946 permits "geometry": null. Guard for it before calling shape() and route null-geometry events to a dedicated fallback key so they do not crash the producer or silently collapse onto partition 0.

Verification

This snippet asserts the two core invariants: the same coarse cell always yields the same partition, and two nearby fine-grained events share it. It reimplements Kafka’s murmur2-modulo mapping so the test runs without a live broker.

import h3

PARTITION_RESOLUTION = 5


def _murmur2(data: bytes) -> int:
    """Kafka's murmur2 (matches the JVM DefaultPartitioner)."""
    length = len(data)
    seed = 0x9747B28C
    m, r = 0x5BD1E995, 24
    h = (seed ^ length) & 0xFFFFFFFF
    for i in range(0, length - length % 4, 4):
        k = (data[i] | data[i + 1] << 8 | data[i + 2] << 16 | data[i + 3] << 24)
        k = (k * m) & 0xFFFFFFFF
        k ^= (k >> r)
        k = (k * m) & 0xFFFFFFFF
        h = (h * m) & 0xFFFFFFFF
        h ^= k
    rem = length % 4
    if rem >= 3:
        h ^= data[length - rem + 2] << 16
    if rem >= 2:
        h ^= data[length - rem + 1] << 8
    if rem >= 1:
        h ^= data[length - rem]
        h = (h * m) & 0xFFFFFFFF
    h ^= (h >> 13)
    h = (h * m) & 0xFFFFFFFF
    h ^= (h >> 15)
    return h


def partition_for(cell: str, num_partitions: int) -> int:
    # Kafka masks the sign bit, then takes modulo the partition count.
    return (_murmur2(cell.encode("utf-8")) & 0x7FFFFFFF) % num_partitions


def cell_for(lng: float, lat: float) -> str:
    return h3.latlng_to_cell(lat, lng, PARTITION_RESOLUTION)


def test_same_cell_same_partition():
    # Two distinct points in lower Manhattan resolve to one coarse cell.
    cell_a = cell_for(-74.0060, 40.7128)
    cell_b = cell_for(-74.0035, 40.7145)
    assert cell_a == cell_b                       # same coarse H3 cell
    assert partition_for(cell_a, 12) == partition_for(cell_b, 12)


def test_distant_cell_may_differ():
    manhattan = cell_for(-74.0060, 40.7128)
    london = cell_for(-0.1278, 51.5074)
    assert manhattan != london                    # different cells entirely


def test_partition_is_deterministic():
    cell = cell_for(-74.0060, 40.7128)
    assert partition_for(cell, 12) == partition_for(cell, 12)

FAQ

Related

• Broker Selection & Partitioning for Spatial Streams — parent: choosing brokers and partition schemes for spatial event streams
• H3 vs S2 vs Quadkey for Spatial Partitioning — picking the spatial index family before you key on it
• Consumer Lag & Partition Skew Monitoring — detecting the hot-partition skew this pattern can introduce