Event-Driven Django: Redis Streams, Kafka, and the Transactional Outbox Pattern

Synchronous calls between services are a tax: every hop adds latency and a new way to fail. Event-driven architecture flips it — services publish facts ("OrderPaid") and others react on their own time. The hard part isn't the messaging; it's not losing or double-processing events. This is how to do it correctly from Django.

Redis Streams vs Kafka

Default to Redis Streams unless you genuinely need Kafka's durability and replay. Don't run Kafka because it's fashionable — it's a serious operational commitment.

The dual-write problem

The naive approach — save to the DB, then publish an event — has a fatal race: if the process dies between the two, you've committed the order but never told anyone. Or you publish, then the DB rolls back, and consumers act on an order that doesn't exist. You cannot make two systems commit atomically with a try/except.

The transactional outbox

The fix: write the event into an outbox table in the same transaction as your business data. One commit, fully atomic. A separate relay process then reads the outbox and publishes to the broker.

class OutboxEvent(models.Model):
    topic = models.CharField(max_length=100)
    payload = models.JSONField()
    created_at = models.DateTimeField(auto_now_add=True)
    published_at = models.DateTimeField(null=True, blank=True)

@transaction.atomic
def pay_order(order):
    order.status = "paid"
    order.save()
    OutboxEvent.objects.create(
        topic="order.paid",
        payload={"order_id": str(order.id), "amount": str(order.total)},
    )   # commits together with the order — atomic

A relay (a Celery beat task or a small daemon) ships unpublished rows:

@shared_task
def relay_outbox():
    for ev in OutboxEvent.objects.filter(published_at__isnull=True)[:500]:
        redis.xadd(ev.topic, {"data": json.dumps(ev.payload)})
        ev.published_at = timezone.now()
        ev.save(update_fields=["published_at"])

Idempotent consumers

At-least-once delivery means consumers will see duplicates. Make every handler idempotent: give each event an ID and record processed IDs, so reprocessing is a no-op.

def handle(event_id, payload):
    _, created = ProcessedEvent.objects.get_or_create(event_id=event_id)
    if not created:
        return                      # already handled — skip
    do_the_work(payload)

Summary

Event-driven Django is mostly about two guarantees: don't lose events (transactional outbox — write the event in the same DB transaction as the data) and don't double-apply them (idempotent consumers keyed on an event ID). Pick Redis Streams unless Kafka's durability is a hard requirement, keep the relay simple, and your services decouple without ever losing a fact.