GDPR Engineering in Django: Retention, Anonymisation, DSAR Exports, and Audit Trails

Most privacy work in a Django project is not legal work. It is schema design, a few management commands, and the discipline to decide — once, in writing — how long each kind of row may live. Do that and the legal questions answer themselves. Skip it and every request from a user turns into an archaeology project.

Start with an inventory, not a policy

You cannot write a retention rule for data you cannot name. The first deliverable is a table of every model that holds personal data, why you hold it, and how long. Keep it next to the code so it rots less:

class Order(models.Model):
    class Privacy:
        personal_fields = ["email", "shipping_name", "shipping_address", "phone"]
        basis = "contract"            # why you may hold it
        retention_days = 2555         # 7 years — tax law wins over erasure
        anonymise_after_days = 730    # strip personal fields, keep the row

Three categories cover almost everything you have:

• Delete outright. Session logs, abandoned carts, unverified signups, raw analytics. Nothing downstream depends on them.
• Anonymise and keep. Orders, invoices, support tickets. The facts — amount, date, product — must survive for accounting and reporting; the person must not.
• Keep in full. Records a law explicitly requires you to retain. Invoices under tax law are the classic case: an erasure request does not override them, and saying so plainly is a valid answer.

That last point matters. "We removed everything except your invoices, which we must keep for seven years under tax law" is a complete and lawful response. Deleting the invoice would be the mistake.

Making the inventory enforceable

An inventory that lives in a wiki is out of date by the second sprint. Because the Privacy inner class sits on the model, you can test for it. The test below fails the build whenever someone adds a model with an obviously personal field but no declaration, which turns "remember to update the inventory" into a red CI run instead of a finding in next year's audit.

from django.apps import apps
from django.db import models

PERSONAL_HINTS = ("email", "phone", "name", "address", "ip", "birth")

# Models you do not own declare their privacy data here instead.
THIRD_PARTY_PRIVACY = {
    "auth.User": {"personal_fields": ["email", "first_name", "last_name"],
                  "basis": "contract", "retention_days": None},
}

def looks_personal(field) -> bool:
    if isinstance(field, (models.EmailField, models.GenericIPAddressField)):
        return True
    return any(hint in field.name for hint in PERSONAL_HINTS)

def test_every_personal_model_declares_privacy():
    missing = []
    for model in apps.get_models():
        if model._meta.label in THIRD_PARTY_PRIVACY or hasattr(model, "Privacy"):
            continue
        personal = [f.name for f in model._meta.concrete_fields if looks_personal(f)]
        if personal:
            missing.append(f"{model._meta.label}: {personal}")
    assert not missing, "Undeclared personal data:\n" + "\n".join(missing)

Anonymisation that does not wreck your reports

Naive anonymisation nulls the columns and destroys your analytics: revenue per country disappears with the address, repeat-customer counts collapse when every email becomes empty. Do it deliberately instead.

import hashlib
from django.conf import settings

def pseudonym(value: str) -> str:
    """Stable, non-reversible token so repeat-customer analysis survives."""
    salted = f"{settings.ANONYMISATION_SALT}{value.lower().strip()}"
    return hashlib.sha256(salted.encode()).hexdigest()[:32]


@transaction.atomic
def anonymise_order(order):
    order.customer_token   = pseudonym(order.email)   # keep the link
    order.country          = order.country            # keep the geography
    order.email            = ""
    order.shipping_name    = "[removed]"
    order.shipping_address = ""
    order.phone            = ""
    order.anonymised_at    = timezone.now()
    order.save(update_fields=[...])

Two warnings that people learn the hard way. The salt must be secret and must never be rotated casually — with the salt, an email can be checked against a token, which is exactly the linkability you were removing. And a pseudonym is still personal data under GDPR; it lowers risk, it does not take you out of scope.

Watch out for the copies. Personal data leaks into places your model layer never sees:

• historical rows from django-simple-history or your own audit tables
• Celery result backends and task arguments
• error reports (Sentry), application logs, request logs
• email queues, PDF invoices in storage, database backups
• search indexes and caches

Backups are the honest exception: you cannot rewrite them. The accepted answer is a documented backup retention window — "personal data disappears from backups within 35 days" — and a rule that a restored backup is re-anonymised before use.

Use a keyed hash, and keep the key out of the repository

The pseudonym() function above concatenates the salt and the value and hashes the result. That works, but the construction designed for exactly this job is an HMAC: a keyed hash whose security does not depend on how the key and message are glued together. It is a small change and it is what a security reviewer will expect to see:

import hashlib
import hmac

from django.conf import settings

def pseudonym(value: str) -> str:
    key = settings.ANONYMISATION_KEY.encode()
    msg = value.lower().strip().encode()
    return hmac.new(key, msg, hashlib.sha256).hexdigest()[:32]

Load ANONYMISATION_KEY from the environment or a secrets manager, never from the repository, and keep it separate from SECRET_KEY. You will rotate SECRET_KEY after an incident; rotating the pseudonymisation key splits every customer into "before" and "after" tokens, so treat any rotation as a planned data migration rather than routine hygiene.

Scrubbing the copies you do control

Of the places listed above, several are within your code's reach and should be handled in the same unit of work as the anonymisation itself:

from django.core.cache import cache
from django.db import transaction

@transaction.atomic
def anonymise_order_everywhere(order):
    anonymise_order(order)                 # live row first
    order.history.all().update(            # django-simple-history rows
        email="", shipping_name="[removed]",
        shipping_address="", phone="",
    )
    transaction.on_commit(lambda: cache.delete(f"order:{order.pk}"))
    transaction.on_commit(lambda: reindex_order.delay(order.pk))

For Celery, the rule is simpler: pass primary keys to tasks, never emails or names. Task arguments are serialised into the broker, stored in the result backend if you use one, shown in monitoring dashboards and printed in failure logs.

Retention as a scheduled job, not a promise

A retention policy that nobody runs is worse than none, because you wrote it down. Make it a command, put it in Beat, and have it report what it did.

class Command(BaseCommand):
    def add_arguments(self, parser):
        parser.add_argument("--apply", action="store_true")

    def handle(self, *args, **opts):
        now, report = timezone.now(), []

        stale = Order.objects.filter(
            created_at__lt=now - timedelta(days=730),
            anonymised_at__isnull=True,
        )
        report.append(("orders.anonymise", stale.count()))
        if opts["apply"]:
            for order in stale.iterator(chunk_size=500):
                anonymise_order(order)

        carts = Cart.objects.filter(updated_at__lt=now - timedelta(days=30),
                                    order__isnull=True)
        report.append(("carts.delete", carts.count()))
        if opts["apply"]:
            carts.delete()

        for name, count in report:
            self.stdout.write(f"{name}: {count}")
        RetentionRun.objects.create(applied=opts["apply"], report=dict(report))

Default to a dry run. The first time this executes against production data you want to read the numbers, not discover them. And log every run: "we delete after two years" is a claim; a table of runs is evidence.

Access requests: one command, not a scavenger hunt

A data subject access request has a one-month deadline. Build the export once and it takes a minute instead of a day.

def export_user_data(user) -> dict:
    return {
        "generated_at": timezone.now().isoformat(),
        "account": {
            "email": user.email,
            "joined": user.date_joined.isoformat(),
            "last_login": user.last_login and user.last_login.isoformat(),
        },
        "orders": list(
            Order.objects.filter(customer=user)
            .values("number", "created_at", "total_cents", "status")
        ),
        "support_tickets": list(
            Ticket.objects.filter(user=user).values("number", "subject", "created_at")
        ),
        "newsletter": list(
            Subscription.objects.filter(email=user.email)
            .values("confirmed_at", "unsubscribed_at", "source")
        ),
        "logins": list(
            LoginEvent.objects.filter(user=user).order_by("-at")[:100]
            .values("at", "ip_truncated", "user_agent")
        ),
    }

Three things that separate a real implementation from a demo:

• A registry, not a hand-written function. Have each app contribute its own section, so a new app cannot be forgotten by the developer who adds it.
• Other people's data stays out. A support thread may contain another customer's message; export the user's own turns, not the whole thread.
• Deliver it safely. A signed, expiring download behind a fresh login — never an email attachment, which is the one channel you do not control.

The exporter registry

Here is the registry version of the function above. Each app owns a small privacy.py module; the central code only knows how to iterate:

# privacy/registry.py
from collections.abc import Callable
from typing import Any

from django.utils import timezone

EXPORTERS: dict[str, Callable[[Any], Any]] = {}

def exporter(section: str):
    def register(fn):
        if section in EXPORTERS:
            raise ValueError(f"duplicate export section {section!r}")
        EXPORTERS[section] = fn
        return fn
    return register

def export_user_data(user) -> dict:
    data = {"generated_at": timezone.now().isoformat()}
    for section, fn in sorted(EXPORTERS.items()):
        data[section] = fn(user)
    return data


# orders/privacy.py
from privacy.registry import exporter
from .models import Order

@exporter("orders")
def export_orders(user):
    return list(Order.objects.filter(customer=user)
                .values("number", "created_at", "total_cents", "status"))


# orders/apps.py
class OrdersConfig(AppConfig):
    name = "orders"

    def ready(self):
        from . import privacy  # noqa: F401  (registers the exporter)

Pair it with the inventory test: for every app that has a model declaring Privacy with non-empty personal_fields, assert that the app registered at least one exporter. Now the developer who adds a Wishlist model with an email column gets a failing test that tells them exactly what is missing.

When serialising, use json.dumps(data, cls=DjangoJSONEncoder, indent=2) from django.core.serializers.json. values() returns datetime, Decimal and UUID objects that the standard encoder rejects, and you want to find that out in a test, not on day 29 of a request.

Erasure without breaking foreign keys

user.delete() is almost always wrong. It cascades into orders, invoices and audit rows you are required to keep, and it takes your referential integrity with it. Erase the person, keep the facts:

@transaction.atomic
def erase_user(user, *, reason):
    for order in Order.objects.filter(customer=user):
        anonymise_order(order)                     # keeps totals and dates

    Ticket.objects.filter(user=user).update(
        body="[removed at user request]", author_email="")

    Subscription.objects.filter(email=user.email).delete()

    user.email = f"deleted-{user.pk}@invalid"
    user.is_active = False
    user.set_unusable_password()
    user.first_name = user.last_name = ""
    user.save()

    ErasureLog.objects.create(user_pk=user.pk, reason=reason,
                              performed_at=timezone.now())

Note what ErasureLog does and does not contain: the fact that an erasure happened and when, never the data that was erased. That is the whole trick — you can prove you complied without keeping the thing you deleted.

The email placeholder uses @invalid, a reserved domain that can never route anywhere. Reusing the real address as a "tombstone" defeats the purpose.

Propagating erasure to processors

Your database is rarely the only place a customer lives. The email provider, the CRM, the helpdesk and the analytics tool each hold a copy, and as controller you are responsible for passing the erasure on. There is a sequencing trap: once erase_user has run, you no longer have the identifiers those systems need. Capture them first:

@transaction.atomic
def erase_user_everywhere(user, *, reason):
    external = {name: p.external_id(user) for name, p in PROCESSORS.items()}
    erase_user(user, reason=reason)
    for name, ext_id in external.items():
        if ext_id:
            transaction.on_commit(
                lambda n=name, e=ext_id: erase_in_processor.delay(user.pk, n, e)
            )


@shared_task(bind=True, max_retries=12, default_retry_delay=3600)
def erase_in_processor(self, user_pk, processor, external_id):
    try:
        PROCESSORS[processor].erase(external_id)   # must treat "not found" as success
    except ProcessorUnavailable as exc:
        raise self.retry(exc=exc)
    ProcessorErasure.objects.create(user_pk=user_pk, processor=processor,
                                    completed_at=timezone.now())

Pass the processor's own identifier — a CRM contact id, a payment-provider customer id — rather than the email, for the task-argument reasons above. Make each erase() idempotent, because retries will call it twice. And reconcile: a weekly report that lists erasures without a ProcessorErasure row for every configured processor catches the task that exhausted its retries at 3 a.m. and was never looked at again.

Consent is a record, not a boolean

A marketing_opt_in = True column cannot answer the only question that matters when challenged: when, and to what exactly, did this person agree? Store events, not state.

class ConsentEvent(models.Model):
    class Kind(models.TextChoices):
        MARKETING = "marketing"
        ANALYTICS = "analytics"

    email      = models.EmailField(db_index=True)
    kind       = models.CharField(max_length=20, choices=Kind.choices)
    granted    = models.BooleanField()
    text_hash  = models.CharField(max_length=64)   # which wording they saw
    source     = models.CharField(max_length=50)   # "checkout", "footer"
    ip_prefix  = models.CharField(max_length=20)   # truncated, not full
    created_at = models.DateTimeField(auto_now_add=True)

Current state becomes "the latest event of this kind", which is one indexed query and is defensible. text_hash is the detail people forget: you changed your consent wording last year, and you need to show which version this person actually accepted.

Consent events create a tension with erasure. You need them to prove consent, and they contain an email. The workable answer is to replace the email with its pseudonym() on erasure and keep the event. Better still, keep a suppression list of pseudonymised addresses for people who unsubscribed or asked to be forgotten: when a marketing list is re-imported next year, anything whose token is on the list is dropped.

Collect less and most of this gets easier

The cheapest privacy engineering is the field you never added.

• Truncate IP addresses at capture. 192.0.2.0 instead of 192.0.2.47 — enough for rate limiting and geography, no longer an identifier.
• Do not log request bodies on authentication, checkout or profile endpoints. Scrub by field name at the logging layer, not by hoping.
• Store what you need, not what the form could ask. Date of birth for an age check can be a boolean "over 18" computed once and thrown away.
• Set a short default retention on new tables and lengthen deliberately. The default is what you will still have in five years.

Audit trails that prove compliance without becoming a liability

The retention runs and the erasure log are two specific audit trails. The general version is an append-only record of who did what to whose data — above all, staff actions: opening a customer in the admin, exporting data, impersonating a user, changing an email address. It must itself hold as little personal data as possible, or it becomes the largest copy of personal data you have.

class AuditEvent(models.Model):
    at          = models.DateTimeField(default=timezone.now, db_index=True)
    actor_pk    = models.BigIntegerField(null=True)      # None for system jobs
    actor_kind  = models.CharField(max_length=20)        # "staff", "user", "system"
    action      = models.CharField(max_length=50)        # "customer.viewed"
    subject_ref = models.CharField(max_length=64, db_index=True)  # "user:4812"
    detail      = models.JSONField(default=dict)         # ids and counts, not values
    prev_hash   = models.CharField(max_length=64)
    hash        = models.CharField(max_length=64, unique=True)

Plain integer references instead of foreign keys are deliberate: erasing or deleting a user must not cascade into, or null out, the evidence of what was done to them, and the table can later move to a separate database without migration pain. detail records that the email changed, never the old and new values.

Append-only, enforced by the database

Overriding save() to refuse updates is not enough — QuerySet.update(), raw SQL and a curious engineer in dbshell all go around it. Enforce it in PostgreSQL with a trigger, created in a migration, that still permits deleting rows past the audit table's own retention window:

migrations.RunSQL(
    sql="""
    CREATE FUNCTION audit_append_only() RETURNS trigger AS $$
    BEGIN
        IF TG_OP = 'DELETE' AND OLD.at < now() - interval '2 years' THEN
            RETURN OLD;
        END IF;
        RAISE EXCEPTION 'audit events are append-only';
    END;
    $$ LANGUAGE plpgsql;

    CREATE TRIGGER audit_append_only
        BEFORE UPDATE OR DELETE ON privacy_auditevent
        FOR EACH ROW EXECUTE FUNCTION audit_append_only();
    """,
    reverse_sql="""
    DROP TRIGGER audit_append_only ON privacy_auditevent;
    DROP FUNCTION audit_append_only();
    """,
)

Belt and braces: have the application connect as a role that holds only INSERT and SELECT on that table, with migrations run by a separate owner role.

The audit trail you already have

Django's admin writes LogEntry rows for every add, change and delete, and each one stores object_repr — the object's __str__ at the time, which for a user model is usually the email address. After an erasure, the admin history still shows the old address. Scrub it as part of erase_user:

from django.contrib.admin.models import LogEntry
from django.contrib.contenttypes.models import ContentType

LogEntry.objects.filter(
    content_type=ContentType.objects.get_for_model(user),
    object_id=str(user.pk),
).update(object_repr=f"user {user.pk}", change_message="")

Testing privacy code like any other critical path

Privacy code fails silently, so the antidote is tests that search for data rather than assert on implementation details. The most valuable one plants a distinctive marker, erases, and then scans every declared personal field in the project for it:

import pytest
from django.apps import apps

MARKER = "erase-me-7f3a"

@pytest.mark.django_db
def test_erasure_leaves_no_marker(user_factory, order_factory, ticket_factory):
    user = user_factory(email=f"{MARKER}@example.com", first_name=MARKER)
    order_factory(customer=user, email=user.email, shipping_name=MARKER)
    ticket_factory(user=user, body=f"call me, {MARKER}")

    erase_user(user, reason="test")

    for model in apps.get_models():
        privacy = getattr(model, "Privacy", None)
        for name in getattr(privacy, "personal_fields", []):
            hits = model._default_manager.filter(**{f"{name}__icontains": MARKER})
            assert not hits.exists(), f"{model._meta.label}.{name} still holds data"

Because the loop is driven by the inventory, it grows automatically: a new model that declares its personal fields is covered by the erasure test on the day it is added, and if erase_user forgets it, the build fails. Add the admin LogEntry and history tables explicitly, since they do not carry a Privacy declaration.

Troubleshooting and failure modes

A practical checklist

• Every model with personal data has a documented basis and retention period.
• Retention runs on a schedule, defaults to dry run, and logs every execution.
• Export is one command and each app registers its own section.
• Erasure anonymises rather than cascading deletes, and writes an erasure log.
• Consent is stored as events including the wording that was shown.
• IPs are truncated at capture; sensitive fields are scrubbed from logs and error reports.
• Backup retention is documented, and restores are re-anonymised.

None of this is exotic Django. It is models, a management command and a scheduled task — the same tools you use for everything else. The difference is that you decided the rules before someone asked you to prove them.