HTTP Caching and CDNs for Django: ETags, Cache-Control, Surrogate Keys, and Purging

Application caching makes your code faster. HTTP caching makes your code unnecessary — the response is served by a CDN node near the user, and your server never learns the request happened. The mechanism is a handful of headers, and getting them slightly wrong is how you end up serving one customer's dashboard to another.

Four caches, and who obeys what

Between your view and the user there are four places a response can be stored:

• The browser cache — private to one user, obeys Cache-Control.
• The CDN / shared cache — serves many users, obeys Cache-Control plus its own rules.
• The reverse proxy (nginx, Varnish, Traefik) — same idea, closer to you.
• Django's cache framework — inside the process, unrelated to HTTP semantics.

The word that governs the dangerous ones is private versus public. private means "one user may store this"; public means "a shared cache may serve this to anyone". Mislabel a personalised page as public and a CDN will happily hand it to strangers.

Cache-Control, decoded

Cache-Control: public, max-age=300, s-maxage=3600, stale-while-revalidate=86400

• max-age=300 — browsers may reuse for 5 minutes.
• s-maxage=3600 — shared caches may reuse for an hour. Overrides max-age for the CDN only.
• stale-while-revalidate=86400 — after expiry, serve the stale copy instantly and refresh in the background. This one is the single biggest perceived-speed win available, and it is one word.

The split between max-age and s-maxage is the useful trick: short in the browser so users see changes quickly, long at the CDN so your server stays idle. A purge fixes the CDN instantly; you cannot purge a browser.

from django.views.decorators.cache import cache_control

@cache_control(public=True, max_age=300, s_maxage=3600,
               stale_while_revalidate=86400)
def tutorial_detail(request, slug):
    ...

@cache_control(private=True, no_store=True)
def account_dashboard(request):
    ...

no-store is the strong one: do not write this anywhere, ever. Use it on authenticated pages, payment flows and anything with a token in it. no-cache does not mean that — it means "store it, but revalidate before reuse", which surprises people at exactly the wrong moment.

Django's helpers and how they compose

The cache_control decorator is a thin wrapper over django.utils.cache.patch_cache_control, which merges directives into any existing header rather than replacing it. Two consequences follow. First, it knows that public and private are mutually exclusive and removes the opposite one when you set either. Second, if several layers patch the same header — a decorator, a middleware, cache_page — the directives accumulate, and when two layers set max-age the smaller value wins.

from django.utils.cache import patch_cache_control, add_never_cache_headers
from django.views.decorators.cache import never_cache


@never_cache
def checkout(request):
    # Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
    ...


def invoice_pdf(request, pk):
    response = render_invoice(request, pk)
    add_never_cache_headers(response)
    return response


def search(request):
    response = render(request, "search.html", build_context(request))
    if request.GET.get("q"):
        patch_cache_control(response, private=True, max_age=0)
    else:
        patch_cache_control(response, public=True, s_maxage=600)
    return response

Beware of cache_page in this context. It stores the rendered response in Django's cache framework and sets max-age to its timeout, so a cache_page(60 * 60) gives browsers a one-hour lifetime you may not have intended. If you want an origin-side cache with a short browser lifetime, stack an explicit cache_control decorator and check the resulting header in a test.

Make the safe policy the default

Decorators are opt-in, which means a view nobody thought about goes out with no Cache-Control at all. Browsers and some CDNs then apply heuristic freshness — often a fraction of the time since Last-Modified, or a CDN-configured default TTL. The defensive posture is to invert the default: every response is uncacheable unless a view explicitly says otherwise.

from django.utils.cache import patch_cache_control


class DefaultNoStoreMiddleware:
    """Mark any response without an explicit policy as private, no-store."""

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        if not response.has_header("Cache-Control"):
            patch_cache_control(response, private=True, no_store=True)
        return response

Place it near the top of MIDDLEWARE so it runs last on the way out, after every view and middleware has had the chance to set a policy. Static files served by nginx or WhiteNoise never pass through it, which is what you want. The cost is that forgetting a decorator now makes a page slower rather than dangerous — the right direction for a mistake to fail.

Vary: the header that decides who shares a cache entry

Vary tells shared caches which request headers change the response. Get it wrong in either direction and you have a bug.

Vary: Accept-Encoding, Accept-Language, Cookie

• Missing Vary: Cookie on a page that reads the session → the first visitor's personalised page is served to everyone. This is the classic catastrophic caching bug.
• Vary: Cookie on a public page → every distinct cookie set becomes its own cache entry, so your hit rate collapses to roughly zero. Analytics cookies alone guarantee it.

The resolution is architectural: keep public pages genuinely cookie-free. Serve anonymous visitors pages with no session cookie at all, and fetch the personalised fragment — cart count, username — with a small separate request that is never cached.

<span id="cart-count" hx-get="/cart/count/" hx-trigger="load"></span>

Now the page itself is one cache entry shared by every anonymous visitor, and the personal bit costs one tiny uncached request.

Where Vary: Cookie actually comes from

Few projects add Vary: Cookie on purpose. Django adds it for you, correctly, whenever a response might depend on the cookie — which is why it appears on pages you believed were anonymous. The common sources:

• SessionMiddleware adds Vary: Cookie whenever the session was accessed during the request, even if nothing was written. Reading request.session anywhere — a view, a context processor, a template tag — is enough.
• The auth context processor exposes user lazily. A base template that renders {% if user.is_authenticated %} on every page touches the session on every page, so every page varies on cookies.
• CsrfViewMiddleware sets the CSRF cookie and patches Vary: Cookie when a template uses {% csrf_token %}. A newsletter form in the footer is enough to make the whole site uncacheable at the edge.
• The messages framework with its default storage reads cookies and the session to find pending messages whenever a template iterates messages.
• LocaleMiddleware adds Vary: Accept-Language when the language is negotiated from headers rather than the URL. Using i18n_patterns so the language lives in the path keeps entries per language without fragmenting on every browser's header variant.

The fix in each case is the same move as the cart badge: keep the base template of public pages free of anything that reads per-user state, and move forms that need CSRF protection either onto their own uncached pages or into a fragment loaded after the page. A public base template and a separate authenticated base template is a simple, auditable way to enforce this.

Serving the fragment safely

The fragment endpoint is where personal data actually lives, so it gets the strictest policy and nothing else:

from django.shortcuts import render
from django.views.decorators.cache import never_cache
from django.views.decorators.http import require_GET


@require_GET
@never_cache
def cart_count(request):
    count = 0
    if request.user.is_authenticated:
        count = request.user.cart_items.count()
    return render(request, "partials/cart_count.html", {"count": count})

If your CDN is configured with a blanket "cache everything" rule on the whole domain, make sure it still respects origin Cache-Control for this path, or exclude /cart/ and similar prefixes by rule. Belt and braces is appropriate here: the cost of an extra origin request is trivial, the cost of caching someone's cart is not.

ETags and conditional requests

Freshness lifetimes avoid requests. ETags make the unavoidable ones cheap: the client sends the tag it holds, and you answer 304 Not Modified with an empty body.

from django.views.decorators.http import etag, last_modified

def tutorial_etag(request, slug):
    row = Tutorial.objects.filter(slug=slug).values("updated_at", "views").first()
    if not row:
        return None
    return f'"{row["updated_at"].timestamp()}"'

@etag(tutorial_etag)
def tutorial_detail(request, slug):
    ...

Two rules. The tag function must be cheaper than rendering — one indexed query on a couple of columns, not the same work as the view. And the tag must change whenever anything visible changes: template edits and CSS bumps included. Hashing your deploy SHA into it costs nothing and prevents the "the site is stale after deploy" mystery.

Django's ConditionalGetMiddleware will generate ETags from response bodies automatically. That saves bandwidth but not rendering time — the view already ran. Explicit tags skip the work; automatic tags only skip the transfer.

Folding in the release and related objects

A detail page rarely depends on one row. The tutorial page shows its category name, its author, maybe a "related tutorials" block. A robust tag combines every input that can change the output, plus the release identifier, and hashes the result so the header stays short:

import hashlib

from django.conf import settings
from django.db.models import Max
from django.views.decorators.http import condition


def tutorial_state(request, slug):
    row = (
        Tutorial.objects.filter(slug=slug, published=True)
        .values("pk", "updated_at", "category__updated_at")
        .first()
    )
    if row is None:
        return None
    related = (
        Tutorial.objects.filter(category__tutorials__slug=slug)
        .aggregate(latest=Max("updated_at"))["latest"]
    )
    raw = f'{settings.RELEASE_SHA}|{row["updated_at"]}|{row["category__updated_at"]}|{related}'
    return hashlib.sha256(raw.encode()).hexdigest()[:32]


def tutorial_last_modified(request, slug):
    return (
        Tutorial.objects.filter(slug=slug, published=True)
        .values_list("updated_at", flat=True)
        .first()
    )


@condition(etag_func=tutorial_state, last_modified_func=tutorial_last_modified)
def tutorial_detail(request, slug):
    ...

Set RELEASE_SHA from an environment variable written by your deploy pipeline (for example os.environ.get("RELEASE_SHA", "dev") in settings). The condition decorator accepts both functions; Django returns the value quoted as the ETag header and handles If-None-Match, If-Modified-Since, and — for unsafe methods — If-Match and If-Unmodified-Since, replying 412 Precondition Failed when they do not match. That last behaviour is useful beyond caching: an API client that sends If-Match with a PUT gets optimistic concurrency control for free.

Remember that the tag function runs on every request, including every cache miss and every revalidation. Two small queries against indexed columns are fine; an aggregate over a large table is not. If the dependency graph is complicated, maintain a single content_version counter per object that you bump in the same signal that triggers purges, and tag on that.

Surrogate keys: purge by meaning, not by URL

You edit one tutorial. Which URLs are now wrong? The detail page, the category listing, the tag pages, the home page, the sitemap, the feed. Enumerating them is a losing game.

Instead, tag responses as you build them and purge by tag:

class SurrogateKeyMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        request.surrogate_keys = set()
        response = self.get_response(request)
        if request.surrogate_keys and response.status_code == 200:
            response["Surrogate-Key"] = " ".join(sorted(request.surrogate_keys))
        return response


def tutorial_detail(request, slug):
    tutorial = get_object_or_404(Tutorial, slug=slug)
    request.surrogate_keys |= {
        f"tutorial-{tutorial.pk}",
        f"category-{tutorial.category_id}",
    }
    ...

Then a single signal invalidates every page that mentioned the object, wherever it appeared:

@receiver(post_save, sender=Tutorial)
def purge_tutorial(sender, instance, **kwargs):
    purge_surrogate_keys([f"tutorial-{instance.pk}",
                          f"category-{instance.category_id}"])

Not every CDN supports surrogate keys, and the header name varies. Where it is unavailable, the fallback is short s-maxage plus stale-while-revalidate — less precise, but it degrades gracefully instead of serving week-old content.

Designing a key vocabulary

Keys work best as a small, deliberate vocabulary rather than whatever each view author invents. A scheme that scales:

The collection key (list-tutorials) is what keeps headers bounded. A listing of 200 tutorials should not carry 200 individual keys — CDNs cap the header size and the number of keys per object — so listings carry the collection key and a new publication purges it. The trade-off is precision: editing one tutorial's title purges every listing, which is usually fine because listings are cheap to re-render and re-fill quickly behind stale-while-revalidate.

Implementing purge_surrogate_keys

The signal in the original example calls purge_surrogate_keys directly from post_save. In production two refinements matter: purge only after the transaction commits, otherwise the CDN refetches the page before the new data is visible and re-caches the old version; and purge asynchronously with retries, so a slow CDN API never makes the admin save hang or fail.

# content/purge.py
import logging

import httpx
from django.conf import settings

log = logging.getLogger(__name__)


def purge_surrogate_keys(keys):
    keys = sorted(set(keys))
    if not keys or not settings.CDN_PURGE_URL:
        return
    response = httpx.post(
        settings.CDN_PURGE_URL,
        headers={"Authorization": f"Bearer {settings.CDN_PURGE_TOKEN}"},
        json={"keys": keys},
        timeout=10.0,
    )
    response.raise_for_status()
    log.info("purged %d surrogate keys", len(keys))

The request shape above is a placeholder: each vendor's API differs in URL, authentication header and body format, so wrap it behind this one function and adapt the body to your provider's documentation. The signal then defers the call:

# content/signals.py
from functools import partial

from django.db import transaction
from django.db.models.signals import post_delete, post_save, pre_save
from django.dispatch import receiver

from .models import Tutorial
from .tasks import purge_keys_task


def keys_for(tutorial, old_category_id=None):
    keys = {f"tutorial-{tutorial.pk}", f"category-{tutorial.category_id}",
            "list-tutorials"}
    if old_category_id and old_category_id != tutorial.category_id:
        keys.add(f"category-{old_category_id}")
    return sorted(keys)


@receiver(pre_save, sender=Tutorial)
def remember_old_category(sender, instance, **kwargs):
    if instance.pk:
        instance._old_category_id = (
            Tutorial.objects.filter(pk=instance.pk)
            .values_list("category_id", flat=True).first()
        )


@receiver(post_save, sender=Tutorial)
@receiver(post_delete, sender=Tutorial)
def purge_tutorial(sender, instance, **kwargs):
    keys = keys_for(instance, getattr(instance, "_old_category_id", None))
    transaction.on_commit(partial(purge_keys_task.delay, keys))

Moving a tutorial between categories must purge both categories, which is why the old value is captured in pre_save. The task itself (Celery, or whichever queue you run) calls purge_surrogate_keys with retry and backoff; a purge that fails permanently should alert, because the fallback is content that stays stale until s-maxage runs out.

Paths that bypass signals

Signals are not a complete change feed. QuerySet.update(), bulk_create(), bulk_update(), raw SQL, data migrations and changes made by another service all skip post_save. Many-to-many edits fire m2m_changed, not post_save. For each of these, either purge explicitly in the code that performs them, or add a management command such as purge_all_content that purges the collection and layout keys, and run it at the end of every data migration and bulk import. Treat "purge everything" as a normal operational tool, not a panic button — with stale-while-revalidate and an origin proxy cache, a full purge is a brief load increase, not an outage.

Static assets: the easy win nobody should skip

Hashed filenames make assets immutable, and immutable assets can be cached effectively forever:

STORAGES = {
    "staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage"},
}

location /static/ {
    add_header Cache-Control "public, max-age=31536000, immutable";
}

immutable tells the browser not even to revalidate on reload. Safe only because the filename changes when the content does — which is exactly what the manifest storage guarantees. Never put immutable on an unhashed path.

Complete settings and the manifest's sharp edges

Defining STORAGES replaces Django's default dictionary entirely, so include the "default" entry as well or file uploads lose their storage backend:

STORAGES = {
    "default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
    },
}

The manifest storage has behaviours worth knowing before they bite in production. A template that references a file missing from the manifest raises ValueError at render time (controlled by the storage's manifest_strict attribute), so a typo in a {% static %} path becomes a 500 rather than a broken image — catch it by running collectstatic in CI and rendering key pages in tests with production storage. CSS url() references are rewritten to hashed names, but paths built in JavaScript at runtime are not. And the old hashed files must remain available for a while after a deploy: a browser holding yesterday's HTML (or a CDN serving it from cache) will still request yesterday's asset names, so do not delete the previous release's static files on deploy.

Caching for logged-in users

The received wisdom is "you cannot cache authenticated pages". You can cache most of them, if you separate the layers:

• Shared shell, private fragments. The product page is identical for everyone; only the cart badge and the "you own this" notice differ. Cache the page publicly, load the differences separately.
• Per-user CDN caching by key. Some CDNs can vary on a signed cookie or header, giving each user their own entry. Powerful, and easy to misconfigure — only worth it at real scale.
• Cache the expensive query, not the page. When personalisation is deep, HTTP caching is the wrong tool; use Django's cache framework on the pieces.

And whatever you choose: on login, logout and any state change, send Cache-Control: no-store and change the session cookie. The bug that leaks a logged-in page to the next visitor almost always starts with a cached redirect.

Why redirects are the weak spot

Browsers and caches treat status codes differently. A 301 or 308 is cacheable by default and browsers keep permanent redirects aggressively, sometimes indefinitely; a 302 or 303 is cacheable only if the response carries explicit freshness. So a 302 after login that inherits a site-wide public, s-maxage=3600 policy from middleware or a CDN rule is stored at the edge — along with any Set-Cookie the CDN failed to strip. Use 301 only for genuinely permanent URL moves, never for anything that depends on who is asking, and make sure authentication views (LoginView, LogoutView, password reset, OAuth callbacks) all emit no-store. Django's login() already cycles the session key, which is the "change the session cookie" half of the rule.

Testing caching behaviour

Caching bugs are invisible in development because there is no CDN in front of runserver. The headers, however, are fully testable, and a small suite catches almost every regression before it reaches the edge:

from unittest import mock

from django.test import TestCase


class CachingHeaderTests(TestCase):
    def setUp(self):
        self.tutorial = TutorialFactory(slug="some-slug", published=True)
        self.url = "/tutorials/some-slug/"

    def test_public_page_is_shareable(self):
        response = self.client.get(self.url)
        cc = response["Cache-Control"]
        self.assertIn("public", cc)
        self.assertIn("s-maxage=3600", cc)
        self.assertNotIn("Cookie", response.get("Vary", ""))
        self.assertFalse(response.cookies)

    def test_conditional_get_returns_304(self):
        first = self.client.get(self.url)
        second = self.client.get(self.url, headers={"if-none-match": first["ETag"]})
        self.assertEqual(second.status_code, 304)
        self.assertEqual(second.content, b"")

    def test_dashboard_is_never_stored(self):
        self.client.force_login(UserFactory())
        response = self.client.get("/account/")
        self.assertIn("no-store", response["Cache-Control"])
        self.assertIn("private", response["Cache-Control"])

    @mock.patch("content.signals.purge_keys_task.delay")
    def test_save_purges_after_commit(self, delay):
        with self.captureOnCommitCallbacks(execute=True):
            self.tutorial.title = "New title"
            self.tutorial.save()
        keys = delay.call_args.args[0]
        self.assertIn(f"tutorial-{self.tutorial.pk}", keys)

captureOnCommitCallbacks matters because TestCase wraps each test in a transaction that never commits, so on_commit callbacks would otherwise never run and the purge test would pass vacuously. The headers= argument to the test client is available from Django 4.2; on older code the equivalent is HTTP_IF_NONE_MATCH=....

A broader safety net is a parametrised test that walks every public URL name — or the entries in your sitemap — as an anonymous client and asserts no Set-Cookie and no Vary: Cookie. It is the single test most likely to catch the footer-form or base-template regression described earlier, and it runs in seconds.

Failure modes and incident response

When caching goes wrong in production, the symptom rarely names the cause. This table maps common reports to where to look first:

If a private page was served from a shared cache

Treat it as a data exposure, not a performance bug. First contain: deploy private, no-store on the affected route (the default-no-store middleware makes this a one-line change) and add a CDN rule bypassing cache for that path. Then purge — everything, not just the URL you know about, because query-string variants and other entry points may hold copies. Rotate what may have leaked: if the cached page contained CSRF tokens, one-time links or session identifiers, invalidate them (for sessions, clearing the session store logs everyone out, which is usually acceptable during an incident). Finally, use CDN logs to establish which cached objects were served, to whom and for how long, because that determines who must be notified. Afterwards, add the regression test that would have caught it; the header tests above are cheap to extend with one more URL.

Verifying it actually works

curl -sI https://example.com/tutorials/some-slug/ | grep -iE 'cache|vary|age|etag'

What to look for:

• Age greater than zero — the CDN is serving from cache. If it is always 0, you are not caching at all.
• An X-Cache style header showing HIT or MISS — request the same URL twice and expect the second to hit.
• Vary containing exactly what you intended and nothing more.
• A Set-Cookie on a page you believe is public — that is your hit rate leaking away, and most CDNs refuse to cache such a response at all.

Measure hit rate per route rather than overall. A 90% average can hide the one expensive page that never caches — and that page is your load.

Checklist

• Authenticated and payment pages: private, no-store.
• Public pages: public, short max-age, long s-maxage, plus stale-while-revalidate.
• No session cookie on anonymous public pages; personalisation loaded separately.
• Vary set deliberately — never accidentally including Cookie.
• Static files hashed and served immutable.
• Surrogate keys on content pages, purged from model signals.
• Deploy SHA folded into ETags so a release invalidates cleanly.
• Hit rate monitored per route, not as one number.

Done properly, most of your traffic stops arriving. The server you were about to upgrade turns out to be idle — which is the cheapest performance work available, and it is mostly headers.