Event-Driven Django: Redis Streams, Kafka, and the Transactional Outbox Pattern

Synchronous calls between services are a tax: every hop adds latency and a new way to fail, and a chain of services calling services means one slow link stalls the whole request. Event-driven architecture flips the model — services publish facts ("OrderPaid") and others react on their own schedule. The hard part is not the messaging; it is never losing an event and never processing one twice. This tutorial builds event-driven Django correctly, from broker choice through the outbox pattern to idempotent consumers.

Why events instead of calls

In a request-response design, when an order is paid your checkout view calls the email service, the inventory service, the analytics service, and the loyalty service in turn. Each call couples checkout to a service that might be slow or down, the latencies add up into the user's wait, and adding a fifth reaction means editing checkout again. With events, checkout publishes a single order.paid fact and returns; the four reactions happen independently, in parallel, and a fifth consumer can subscribe later without checkout knowing it exists. You trade immediate consistency for decoupling, resilience, and the ability to evolve each side separately — usually a good trade for anything that does not need a synchronous answer.

The cost is a new class of problem. Events are asynchronous, so the caller does not learn whether a reaction succeeded; delivery is "at least once," so duplicates happen; and ordering is not free. The rest of this tutorial is about meeting those guarantees, because an event system that loses or double-applies facts is worse than the synchronous calls it replaced.

Redis Streams vs Kafka

Default to Redis Streams unless you genuinely need Kafka's durability, long retention, or replay. Redis Streams give you consumer groups, acknowledgements, and at-least-once delivery with almost no operational overhead — they are the right starting point for the vast majority of Django apps. Kafka is a serious commitment: a cluster to run, partitions to balance, consumer offsets to reason about. Adopt it when you have a real need for a durable, replayable log consumed by many independent teams — not because it appears on every "modern architecture" diagram.

The dual-write problem

The naive implementation looks obvious and is quietly broken: save the order to the database, then publish the event to the broker. Consider what happens when the process crashes between those two steps — you have committed the order but never told anyone, so the email never sends and inventory never decrements. Reverse the order and it is just as bad: you publish, consumers react, and then the database transaction rolls back, leaving everyone acting on an order that does not exist. You cannot wrap a database commit and a broker publish in a single atomic operation with a try/except, because they are two different systems with two different transaction boundaries. This is the dual-write problem, and it is the central correctness challenge of event-driven systems.

Why transaction.on_commit alone is not enough

Django developers often reach for transaction.on_commit() as the fix, and it does solve half the problem: the callback only runs after the commit succeeds, so you never publish an event for a rolled-back order. But it does nothing for the other half. The callback runs in the same process, after the commit, with nothing durable recording that it still needs to happen. If the worker is killed by a deploy, an OOM, or a lost connection to Redis in the milliseconds between commit and publish, the event is gone and nothing will ever retry it. Retries inside the callback do not help either, because the retry state lives in memory. The only way to make "this event must eventually be published" survive a crash is to persist that intention in the same commit as the business change — which is exactly what the outbox does. on_commit still has a role — registering transaction.on_commit(relay_outbox.delay) next to the outbox write is a good way to wake the relay immediately — but only as a latency optimisation, never as the source of truth.

The transactional outbox

The fix is elegant: instead of publishing to the broker directly, write the event into an outbox table in the same database transaction as your business data. One commit, fully atomic — either both the order and its event are persisted, or neither is. A separate relay process then reads unpublished rows from the outbox and ships them to the broker.

class OutboxEvent(models.Model):
    topic = models.CharField(max_length=100)
    payload = models.JSONField()
    created_at = models.DateTimeField(auto_now_add=True)
    published_at = models.DateTimeField(null=True, blank=True)

@transaction.atomic
def pay_order(order):
    order.status = "paid"
    order.save()
    OutboxEvent.objects.create(
        topic="order.paid",
        payload={"order_id": str(order.id), "amount": str(order.total)},
    )   # commits together with the order — atomic, no lost events

Because the event lives in the same database, the same transaction either commits both or rolls back both. There is no window where the order exists but the event does not. This single pattern eliminates lost events, and it is the foundation everything else builds on.

The relay process

A relay reads unpublished outbox rows and publishes them to the broker, marking each as sent. It can be a Celery beat task running every few seconds, or a small dedicated daemon for lower latency:

@shared_task
def relay_outbox():
    rows = (OutboxEvent.objects
            .filter(published_at__isnull=True)
            .order_by("created_at")[:500])
    for ev in rows:
        redis.xadd(ev.topic, {"data": json.dumps(ev.payload)})
        ev.published_at = timezone.now()
        ev.save(update_fields=["published_at"])

The relay itself must tolerate failure: if it crashes after publishing but before marking the row sent, it will republish on restart — which is fine, because consumers are idempotent (next section). That is the deliberate design choice: the relay guarantees at-least-once publication, and the consumer side absorbs the resulting duplicates. Order the relay by created_at to preserve rough ordering, and batch the reads so it scales.

Running more than one relay safely

The simple relay has a concurrency problem that appears the moment you run two copies — two Celery workers picking up overlapping beat ticks, or two daemon replicas for availability. Both read the same 500 rows and both publish them, doubling every event. Consumers would absorb the duplicates, but you waste broker capacity and your lag metrics lie. PostgreSQL's SELECT ... FOR UPDATE SKIP LOCKED solves this cleanly: each relay locks the rows it claims, and a concurrent relay skips locked rows instead of waiting on them, so the instances work through disjoint batches. Django exposes it as select_for_update(skip_locked=True):

import json

from celery import shared_task
from django.db import transaction
from django.utils import timezone


@shared_task
def relay_outbox(batch_size=200):
    with transaction.atomic():
        rows = list(
            OutboxEvent.objects
            .select_for_update(skip_locked=True)
            .filter(published_at__isnull=True)
            .order_by("created_at")[:batch_size]
        )
        if not rows:
            return 0
        now = timezone.now()
        pipe = redis_client.pipeline(transaction=False)
        for ev in rows:
            pipe.xadd(
                ev.topic,
                {"event_id": str(ev.pk), "data": json.dumps(ev.payload)},
                maxlen=1_000_000,
                approximate=True,
            )
            ev.published_at = now
        pipe.execute()                      # raises -> transaction rolls back
        OutboxEvent.objects.bulk_update(rows, ["published_at"])
    return len(rows)

Several details here are deliberate. The row locks are held for the duration of the publish, so keep batches small enough that one batch publishes in well under a second. Pipelining the XADD calls turns 200 network round trips into one. Sending the row's primary key as event_id gives consumers a stable identifier to deduplicate on — it never changes, however many times the row is republished. If the pipeline raises, the whole transaction rolls back and every row stays unpublished — some of them may already be in the stream, which is the expected at-least-once duplicate. The maxlen with approximate=True caps stream memory using Redis's cheap approximate trimming; size it so the stream comfortably holds more than your worst plausible consumer outage, because entries trimmed before a consumer group reads them are simply gone for that group.

There is one trade-off to be honest about: with several relays running, strict publication order is gone, since relay A may publish rows 1–200 while relay B publishes 201–400 slightly earlier. If per-entity ordering matters, either run a single active relay (a single-replica deployment or a lock, accepting a few seconds of failover) or make consumers tolerate out-of-order arrival, for example by carrying a per-entity version number in the payload and ignoring anything older than what has already been applied. For most systems, one relay plus a standby is plenty — a single relay publishing pipelined batches moves thousands of events per second.

Idempotent consumers

At-least-once delivery means every consumer will see duplicates — from relay restarts, broker redelivery, or consumer crashes after work but before acknowledgement. A consumer that is not idempotent will send two emails, charge twice, or decrement inventory twice. Make every handler idempotent by giving each event a stable ID and recording which IDs you have processed, so reprocessing is a harmless no-op:

@transaction.atomic
def handle(event_id, payload):
    _, created = ProcessedEvent.objects.get_or_create(event_id=event_id)
    if not created:
        return                      # already handled — skip silently
    do_the_work(payload)

The get_or_create on a unique event_id is atomic, so even two concurrent deliveries of the same event cannot both pass. Wrap the dedupe check and the work in one transaction, so if the work fails the dedupe record rolls back too and a retry can succeed. Idempotency is not optional polish — it is the other half of the correctness guarantee that the outbox starts.

A Redis Streams consumer worker

With idempotent handlers in place, the consumer loop itself is short. Run it as a Django management command under a process supervisor (systemd, a container orchestrator), one process per consumer name. The consumer group tracks which entries each consumer has received but not acknowledged — the pending entries list — so a message only leaves pending when you explicitly XACK it after the handler succeeds:

import json
import logging
import os
import socket

import redis
from django.conf import settings
from django.core.management.base import BaseCommand
from redis.exceptions import ResponseError

from myproject.events.handlers import handle as handle_event

logger = logging.getLogger(__name__)
STREAM = "order.paid"
GROUP = "email-service"


class Command(BaseCommand):
    help = "Consume order.paid events for the email service"

    def handle(self, *args, **options):
        r = redis.Redis.from_url(settings.REDIS_URL, decode_responses=True)
        consumer = f"{socket.gethostname()}-{os.getpid()}"
        try:
            r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
        except ResponseError as exc:
            if "BUSYGROUP" not in str(exc):   # group already exists: fine
                raise

        while True:
            resp = r.xreadgroup(GROUP, consumer, {STREAM: ">"}, count=50, block=5000)
            for _stream, messages in resp or []:
                for msg_id, fields in messages:
                    try:
                        handle_event(fields["event_id"], json.loads(fields["data"]))
                    except Exception:
                        logger.exception(
                            "handler failed",
                            extra={"msg_id": msg_id,
                                   "correlation_id": fields.get("correlation_id")},
                        )
                        continue              # stays pending; reclaimed later
                    r.xack(STREAM, GROUP, msg_id)

The special ID > asks for entries never delivered to any consumer in the group. A failed message is not acknowledged, so it stays in the pending list rather than disappearing; a periodic reclaim job calls XPENDING to find entries idle longer than your slowest legitimate handler run, dead-letters those whose delivery count has passed your retry limit, and hands the rest to a live consumer with XAUTOCLAIM. A consumer re-reads entries it owns by passing 0 instead of >, which is also the first thing a restarting consumer should do, to finish whatever it had in flight before it died.

Event schemas and versioning

An event is a contract between the publisher and every consumer, and consumers deploy on their own schedules, so you cannot change an event's shape freely. Treat event payloads like a public API: include a schema version, make additive changes (new optional fields) the default, and never repurpose or remove a field without a new version that old consumers can ignore. Keep payloads focused on the fact — "order 123 was paid for 49.99" — rather than dumping your entire order object, so consumers depend on as little as possible. A thin, versioned, well-named event ages gracefully; a fat one coupled to your internal model breaks consumers every time you refactor.

Ordering and consumer groups

Sometimes order matters: order.created must be processed before order.shipped. Both Redis Streams and Kafka preserve order within a single stream or partition, so the trick is to route related events to the same partition — typically by keying on the entity ID (all events for order 123 go to the same partition and stay ordered). Consumer groups let multiple workers share the load of a stream while each message goes to exactly one worker, which is how you scale a consumer horizontally without double-processing. Design your keys so that things that must be ordered share a key, and things that are independent spread across partitions for parallelism.

Handling poison messages

Some events will never process successfully — malformed payloads, references to deleted data, bugs in the handler. Without a strategy, a single poison message blocks the stream or loops forever, retrying and failing. Give consumers a bounded retry with backoff, and after N failures move the event to a dead-letter queue — a separate stream of events that need human attention — so the main flow keeps moving. Monitor the dead-letter queue; a growing one is a signal something is wrong upstream. This is the operational safety valve that keeps one bad event from stalling everything behind it.

Observability and tracing

Asynchronous systems are harder to debug because a single user action fans out across many services and processes. Propagate a correlation ID from the original request into every event payload, and log it in every consumer, so you can reconstruct the full causal chain — "this email, this inventory change, and this analytics event all came from order 123's payment." Track per-topic lag (how far behind consumers are), processing rates, and dead-letter counts. Without this, an event-driven system becomes a set of black boxes that are impossible to reason about when something goes wrong at 2am.

When not to go event-driven

Events are not free, and not every interaction should be one. If the caller needs an immediate answer — "is this coupon valid?" — a synchronous call is correct; forcing it through events adds latency and complexity for nothing. If you have a small monolith with no real decoupling problem, in-process Django signals or direct function calls are simpler and perfectly adequate. Reach for a broker, an outbox, and idempotent consumers when you have genuine asynchronous workflows, multiple independent reactions to the same fact, or services that must evolve and scale separately. Adopt the machinery for the problems it solves, not as a default architecture.

Choreography versus orchestration

There are two ways to coordinate a multi-step process across services, and the choice shapes your whole system. In choreography, each service reacts to events and emits its own, with no central controller — order-paid triggers inventory, which emits stock-reserved, which triggers shipping. It is loosely coupled and resilient, but the overall flow is implicit, spread across many services, and hard to see in one place. In orchestration, a central coordinator explicitly drives the steps, calling or signalling each service in turn. The flow is visible and easy to reason about, but the orchestrator becomes a point of coupling and a place complexity accumulates. Most systems mix both: choreography for simple fan-out reactions, orchestration for complex business processes that need a clear, auditable sequence.

Sagas: transactions across services

A single database transaction cannot span multiple services, so when a business process touches several of them you need a saga — a sequence of local transactions where each step has a compensating action that undoes it. If booking a trip reserves a flight, a hotel, and a car, and the car step fails, the saga runs compensations to cancel the hotel and the flight, returning the system to a consistent state. Sagas trade the simplicity of an all-or-nothing transaction for the reality that distributed steps cannot be atomic. They are more work to design — every step needs an undo — but they are the honest answer to consistency across services, and the outbox pattern is what makes each step's event reliable.

Why exactly-once delivery is a myth

Teams new to event-driven systems often ask for exactly-once delivery, and it is worth being clear: in a distributed system with failures, exactly-once delivery is impossible. A message can always be delivered, the acknowledgement lost, and the message redelivered. What you can achieve is exactly-once processing, by combining at-least-once delivery (the broker keeps trying) with idempotent consumers (duplicates are harmless). This is why idempotency is non-negotiable rather than a nice-to-have — it is the only thing standing between "the broker did its job and redelivered" and "the customer was charged twice." Stop chasing exactly-once delivery and build for at-least-once plus idempotency instead.

Event-driven versus event sourcing

These two terms are often confused but are different ideas. Event-driven, the subject of this tutorial, means services communicate by publishing and reacting to events while keeping their own current-state databases. Event sourcing goes further: it makes the event log itself the source of truth, storing every change as an event and deriving current state by replaying them. Event sourcing gives you a perfect audit trail and the ability to reconstruct any past state, but it is a much larger commitment that reshapes how you store and query everything. You can be fully event-driven without event sourcing, and most teams should — adopt event sourcing only when the audit and time-travel benefits clearly justify its complexity.

Testing event-driven systems

Asynchronous systems need a deliberate testing strategy because the pieces are decoupled. Test each consumer in isolation by feeding it events and asserting on the resulting state and emitted events, which is fast and where most of your coverage should live. Test idempotency explicitly by delivering the same event twice and asserting the effect happened once — this is the test that catches the most dangerous class of bug. Add a few integration tests that exercise a full flow end to end through a real broker to catch wiring mistakes. The decoupling that makes event-driven systems resilient in production also makes them straightforward to unit test, one consumer at a time.

Troubleshooting in production

Most incidents in outbox-based systems fall into a small number of shapes. Two numbers locate most of them: relay lag, the age of the oldest unpublished outbox row, and consumer lag per group, how far each group trails the stream (XINFO GROUPS reports it, alongside the pending count, on Redis 7 and later). Knowing which one moves first narrows the search quickly:

Summary

Event-driven Django comes down to two guarantees. Do not lose events: write each event into an outbox table in the same transaction as your business data, and let a relay publish it at-least-once — this closes the dual-write hole that silently corrupts naive implementations. Do not double-apply events: make every consumer idempotent, keyed on a stable event ID, so duplicates are harmless. Around that core, choose Redis Streams unless you truly need Kafka, version your event schemas like public APIs, key related events together for ordering, dead-letter the poison messages, and propagate correlation IDs so the whole asynchronous chain stays debuggable. Get the outbox and idempotency right and your services decouple cleanly without ever losing or duplicating a fact.