Redis Streams vs Kafka for Geospatial Webhooks

Choose Redis Streams when you want a low-operations, in-memory broker for short-lived, high-rate spatial events and can shard streams yourself for locality; choose Apache Kafka when you need durable multi-hour replay of tile events, native key-based spatial partitioning, and per-region ordering at sustained high volume. Neither is universally correct — the decision turns on your replay window, geometry size, ordering needs, and how much operational surface your team can carry.

This comparison sits inside Broker Selection & Partitioning for Spatial Streams, part of the broader Queue Management, Retries & Delivery Guarantees reference for building reliable spatial webhook pipelines in Python.

When to use this pattern

• Reach for Redis Streams when your pipeline already runs Redis, your events are small (a point ping plus properties, not a multi-ring polygon), your replay window is minutes to a couple of hours, and you value a single moving part over maximum durability.
• Reach for Kafka when high-frequency sensor volume is sustained (hundreds of thousands of events per second), you must reprocess tiles from hours or days ago, and you need strict per-region ordering that survives consumer restarts and rebalances.
• Reach for either with a claim-check when geometries are heavy: keep multi-megabyte polygons in object storage and put only a reference on the stream, so payload-size limits and memory pressure stop being the deciding factor.

It is not the right tool to agonize over when your volume is a few thousand events per hour and your consumers are already idempotent — at that scale either broker is over-provisioned and the choice barely matters. Pick whichever your team operates today.

The comparison at a glance

Why the spatial workload changes the calculus

A generic “Redis vs Kafka” comparison ignores what makes geospatial traffic distinctive: payloads carry geometry, and geometry is both large and order-sensitive. A stream of vehicle GPS pings is tiny and tolerant of reordering; a stream of edited parcel polygons or re-tiled raster footprints is neither. The broker that wins depends on which of those you are moving.

The second spatial twist is locality. You usually want all events touching the same region to land on the same consumer, so per-region state (a running geofence count, a tile version) stays coherent and ordered. Kafka gives you this directly: hash the message key to a partition, key on an H3 cell, and every event for that cell is ordered on one partition. Redis Streams has no key-to-consumer affinity — a consumer group is a work-stealing pool — so you recreate locality by sharding into multiple streams. The diagram below contrasts the two models.

If per-region ordering matters, this is the crux: Kafka enforces it through the partition, whereas Redis Streams needs you to shard streams by cell to approximate it. The tradeoffs of choosing the cell system itself — resolution, cell shape, neighbour behaviour — are covered in H3 vs S2 vs Quadkey for spatial partitioning.

Complete runnable implementation

Both examples move the same event: a sensor reading tagged with an H3 cell and a small GeoJSON point (EPSG:4326 / WGS84). The Kafka version keys on the H3 cell so a region maps to a stable partition; the Redis version shows XADD plus a consumer-group read with acknowledgement.

# Apache Kafka — aiokafka producer + consumer, keyed by H3 cell.
# pip install aiokafka h3
import asyncio
import json
import h3
from aiokafka import AIOKafkaProducer, AIOKafkaConsumer

TOPIC = "spatial-events"
BOOTSTRAP = "localhost:9092"


def h3_key(lon: float, lat: float, resolution: int = 7) -> bytes:
    """Derive the partition key from the event's H3 cell (EPSG:4326 input)."""
    # h3 expects (lat, lon) order.
    return h3.latlng_to_cell(lat, lon, resolution).encode("utf-8")


async def produce() -> None:
    producer = AIOKafkaProducer(bootstrap_servers=BOOTSTRAP)
    await producer.start()
    try:
        event = {
            "sensor_id": "SN-42",
            "reading": 17.3,
            "geometry": {"type": "Point", "coordinates": [-73.965355, 40.782865]},
        }
        lon, lat = event["geometry"]["coordinates"]
        # Same key -> same partition -> per-region order preserved.
        await producer.send_and_wait(
            TOPIC,
            key=h3_key(lon, lat),
            value=json.dumps(event).encode("utf-8"),
        )
    finally:
        await producer.stop()


async def consume() -> None:
    consumer = AIOKafkaConsumer(
        TOPIC,
        bootstrap_servers=BOOTSTRAP,
        group_id="tile-workers",
        enable_auto_commit=False,        # commit only after successful processing
        auto_offset_reset="earliest",    # replay from the log start if no offset
    )
    await consumer.start()
    try:
        async for msg in consumer:
            cell = msg.key.decode()
            event = json.loads(msg.value)
            # ... idempotent processing keyed on (cell, sensor_id) ...
            print(f"partition={msg.partition} cell={cell} sensor={event['sensor_id']}")
            await consumer.commit()      # at-least-once: commit after the side effect
    finally:
        await consumer.stop()

# Redis Streams — redis.asyncio XADD producer + XREADGROUP consumer.
# pip install "redis>=5" h3
import asyncio
import json
import h3
import redis.asyncio as redis

STREAM = "spatial-events"
GROUP = "tile-workers"


async def produce(r: redis.Redis) -> None:
    event = {
        "sensor_id": "SN-42",
        "reading": 17.3,
        "geometry": {"type": "Point", "coordinates": [-73.965355, 40.782865]},
    }
    lon, lat = event["geometry"]["coordinates"]
    cell = h3.latlng_to_cell(lat, lon, 7)   # store cell so consumers can route/shard
    # MAXLEN caps memory: keep ~the last 100k entries, approximate ("~") for speed.
    await r.xadd(
        STREAM,
        {"cell": cell, "payload": json.dumps(event)},
        maxlen=100_000,
        approximate=True,
    )


async def consume(r: redis.Redis) -> None:
    # Create the group at the stream start ("0") once; ignore "already exists".
    try:
        await r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
    except redis.ResponseError as exc:
        if "BUSYGROUP" not in str(exc):
            raise
    while True:
        resp = await r.xreadgroup(
            GROUP, "consumer-1", {STREAM: ">"}, count=64, block=5000
        )
        for _stream, entries in resp or []:
            for entry_id, fields in entries:
                event = json.loads(fields["payload"])
                # ... idempotent processing ...
                print(f"id={entry_id} cell={fields['cell']} sensor={event['sensor_id']}")
                await r.xack(STREAM, GROUP, entry_id)   # at-least-once ack


async def main() -> None:
    r = redis.Redis(host="localhost", port=6379, decode_responses=True)
    await produce(r)
    await consume(r)


if __name__ == "__main__":
    asyncio.run(main())

The structural difference is visible in the API: Kafka carries a first-class key that decides the partition, while Redis Streams carries an opaque field dict and you must add the cell yourself if you later want to shard for locality.

Parameter reference

Gotchas and spatial edge cases

1. Redis Streams memory pressure with large geometries. Every entry lives in RAM until trimmed. A stream of multi-megabyte polygons at high rate can push Redis into eviction or an out-of-memory kill. Always set a MAXLEN/MINID cap sized from your real entry size, and for heavy geometries store the blob in object storage and stream only a reference (the claim-check pattern).

2. Kafka partition count versus skew. Keying by H3 cell only balances load if traffic is spread across cells. A single dense metro region can make one cell — and therefore one partition — a hotspot while others idle. Provision enough partitions for peak concurrent hot cells, and consider a higher H3 resolution or a composite key to spread a hot region. You cannot decrease partition count later without a topic migration.

3. Ordering is per-partition / per-stream, not global. Kafka guarantees order only within a partition, so two cells on different partitions can be processed out of relative order — usually fine, since regions are independent. Redis Streams keeps total order in one stream but a consumer group interleaves across consumers, so do not assume a group preserves per-region order without sharding.

4. Exactly-once is a boundary, not a switch. Kafka’s exactly-once semantics hold for Kafka-to-Kafka transactions; the moment your consumer writes to PostGIS or calls a downstream webhook, you are back to at-least-once. Redis Streams is at-least-once by design. In both, make the consumer idempotent with a deterministic key so redelivery is a no-op — the same discipline covered in Delivery Guarantees & Event Ordering.

5. Consumer crash leaves pending entries (Redis). Unacknowledged entries sit in the Pending Entries List. Without a reclaim loop using XAUTOCLAIM/XPENDING, a crashed consumer’s in-flight spatial events are never reprocessed. Kafka handles this through offset commits and rebalance, but if you commit before the side effect you can silently drop events.

6. Coordinate order mismatch in keys. h3 takes (lat, lon) while GeoJSON stores [lon, lat]. Swapping them keys events to the wrong cell, scattering a region across partitions and destroying locality. Keep a single helper (as above) so the conversion happens in exactly one place.

Verification

This test asserts the property that actually matters for spatial locality: identical H3 keys route to the same Kafka partition (deterministic ordering), while distinct cells generally spread. Run with pytest; it uses only the partitioner math, no live broker.

import h3


def default_partition(key: bytes, num_partitions: int) -> int:
    """Mirror Kafka's default murmur2-based key partitioner shape.

    The exact hash is Kafka's murmur2; here we assert the *invariant*
    that a fixed key maps to a fixed partition, which is what preserves
    per-region ordering. Swap in aiokafka's DefaultPartitioner in an
    integration test against a real cluster.
    """
    return (hash(key) & 0x7FFFFFFF) % num_partitions


def h3_key(lon: float, lat: float, resolution: int = 7) -> bytes:
    return h3.latlng_to_cell(lat, lon, resolution).encode("utf-8")


def test_same_cell_same_partition():
    """Two events in the same H3 cell must land on one partition."""
    k1 = h3_key(-73.965355, 40.782865)
    k2 = h3_key(-73.965360, 40.782860)   # metres apart, same res-7 cell
    assert k1 == k2
    assert default_partition(k1, 12) == default_partition(k2, 12)


def test_distinct_cells_are_stable():
    """A given key always maps to the same partition (deterministic routing)."""
    k = h3_key(-73.965355, 40.782865)
    assert default_partition(k, 12) == default_partition(k, 12)

FAQ

Related

• Broker Selection & Partitioning for Spatial Streams — parent guide to choosing and partitioning the broker under a spatial webhook pipeline
• Partitioning Kafka Topics by H3 Cell — the keyed-partition mechanics referenced above, in depth
• Delivery Guarantees & Event Ordering — at-least-once, ordering, and idempotent consumers across both brokers
• H3 vs S2 vs Quadkey for Spatial Partitioning — choosing the cell system you key partitions on