Evaluating and Guarding LLM Features: Golden Sets, Regression Tests, and Prompt-Injection Defence

Traditional code fails loudly. An LLM feature fails quietly and plausibly: the same prompt that worked last month now returns a confident, well-formatted, wrong answer. Nothing throws. No test goes red. A customer notices before you do — unless you built the thing that notices.

Why normal tests do not cover this

assertEqual(output, expected) is useless when the output is prose that may legitimately vary. But the opposite conclusion — "you cannot test AI" — is wrong and expensive. You can test three things reliably:

• Structure. Did it return valid JSON with the required fields, an allowed enum value, a number in range?
• Grounding. Does every claim appear in the source documents you supplied?
• Behaviour on known inputs. On these fifty cases, does it still do what it did before?

That is 90% of production quality. The remaining 10% is taste, and taste is what humans review.

The golden set: fifty rows beat a dashboard

Start with a table of cases: input, what a good answer contains, what it must never contain. Keep it in the database so support staff can add cases without a deploy.

class EvalCase(models.Model):
    feature      = models.CharField(max_length=50)      # "support_reply"
    prompt       = models.TextField()
    context      = models.JSONField(default=dict)       # retrieved docs, user state
    must_include = models.JSONField(default=list)       # ["14 days", "free of charge"]
    must_exclude = models.JSONField(default=list)       # ["guarantee", "lifetime"]
    expects_tool = models.CharField(max_length=50, blank=True)
    notes        = models.TextField(blank=True)
    is_active    = models.BooleanField(default=True)

Where do cases come from? Three sources, in order of value:

1. Every bug. A wrong answer reported by a user becomes a case the same day. This is the single highest-value habit in the whole discipline.
2. Edge cases you already know. Empty context, hostile input, the customer with no orders, the question in another language.
3. Representative traffic. Sample real prompts, strip personal data, keep the shapes.

Fifty good cases catch more than five hundred generated ones. Resist the urge to bulk-generate.

What makes a case useful

A case earns its place when it fails for exactly one reason. "Answer is good" is not checkable; "answer mentions the 14-day window and does not promise a refund" is. Write must_include phrases for the facts that matter to the customer, not for the wording you happened to like, and put the forbidden phrases where a wrong answer would cost you something: invented guarantees, competitor names, a price that does not exist.

Literal phrases are brittle in one predictable way: the model says "two weeks" instead of "14 days" and a correct answer fails. Rather than loosening the case until it checks nothing, allow alternatives. A nested list in the same JSON field keeps the model unchanged:

for phrase in case.must_include:
    options = phrase if isinstance(phrase, list) else [phrase]
    if not any(opt.lower() in output.lower() for opt in options):
        failures.append(f"missing: {' | '.join(options)}")

Use notes for the story: which ticket the case came from, what the wrong answer was, why the rule exists. Six months later somebody will want to delete a case that "keeps failing", and the note is what tells them it is the only thing standing between the feature and a repeat of the original incident.

Deactivate rather than delete. When a product rule changes (the return window becomes 30 days), flip is_active on the old case and add a new one. Historical runs keep pointing at a case that still exists, so the pass-rate graph stays interpretable across the policy change.

Scoring without a human

Run the cheap checks first — they are deterministic, instant and free.

def score_case(case, output, tool_calls):
    failures = []

    for phrase in case.must_include:
        if phrase.lower() not in output.lower():
            failures.append(f"missing: {phrase}")

    for phrase in case.must_exclude:
        if phrase.lower() in output.lower():
            failures.append(f"forbidden: {phrase}")

    if case.expects_tool and case.expects_tool not in tool_calls:
        failures.append(f"did not call: {case.expects_tool}")

    return failures

For structured output, validate against a schema and treat a parse failure as a hard fail. Never "repair" malformed JSON silently in production — you will hide a degradation that is about to get worse.

class Extraction(BaseModel):
    invoice_number: str
    total_cents: int = Field(ge=0)
    currency: Literal["EUR", "USD", "GBP"]
    due_date: date

try:
    parsed = Extraction.model_validate_json(output)
except ValidationError as exc:
    metrics.incr("extraction.schema_fail")
    raise

Grounding checks you can compute

Before reaching for a judge, notice how many grounding failures are about numbers. A support answer that invents a price, a date or a quantity is the most damaging kind of hallucination, and it is also the easiest to detect mechanically: every number in the answer should appear somewhere in the sources you supplied.

NUMBER = re.compile(r"\d+(?:[.,]\d+)*")

def ungrounded_numbers(answer: str, sources: list[str]) -> set[str]:
    """Numbers in the answer that appear in no source document."""
    source_numbers = set(NUMBER.findall(" ".join(sources)))
    return {n for n in NUMBER.findall(answer) if n not in source_numbers}

# inside score_case
sources = [doc["text"] for doc in case.context.get("documents", [])]
for n in sorted(ungrounded_numbers(output, sources)):
    failures.append(f"ungrounded number: {n}")

It has false positives — the model writes "3 steps" or reformats "1,000" as "1000" — so start by logging these rather than failing on them, look at a week of results, then add a small allowlist of harmless numbers or normalise separators before comparing. The same idea extends to anything you can extract reliably: order numbers, URLs, product SKUs. If the answer contains a URL that is not in the sources and not on your own domain, that is worth a hard failure on its own.

Model-as-judge, used narrowly

For qualities you cannot grep for — tone, completeness, whether an answer is grounded in the sources — a second model call can score the first. It works if you keep it narrow.

JUDGE = """You are grading one answer. Reply with JSON only.

Question: {question}
Sources: {sources}
Answer: {answer}

grounded: true only if every factual claim appears in Sources.
complete: true only if the answer addresses the whole question.
reason: one sentence."""

Two rules keep judges honest. Ask for a boolean and a reason, not a score out of ten — numeric scores from models cluster around 7 and carry almost no signal. And calibrate against humans once: grade forty cases by hand, compare, and if the judge disagrees more than one time in ten, fix the rubric before you trust it anywhere.

Wire it into CI, but not into every commit

Evals cost money and take minutes, so run them where they pay off: on changes to prompts, tools or model configuration, and on a schedule.

name: llm-evals
on:
  pull_request:
    paths: ["**/prompts/**", "**/tools/**", "**/ai/**"]
  schedule:
    - cron: "0 6 * * 1"          # Monday morning, catches provider drift

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python manage.py run_evals --feature support_reply --fail-under 0.92

The weekly run matters as much as the pull-request run. Nothing in your repository changes when a provider updates a model — but your pass rate does, and you want to hear it from CI on Monday rather than from a customer on Thursday.

class Command(BaseCommand):
    def handle(self, *args, **opts):
        cases = EvalCase.objects.filter(feature=opts["feature"], is_active=True)
        run = EvalRun.objects.create(feature=opts["feature"], git_sha=git_sha())

        passed = 0
        for case in cases:
            output, calls = run_feature(case.prompt, case.context)
            failures = score_case(case, output, calls)
            EvalResult.objects.create(run=run, case=case, output=output,
                                      failures=failures, passed=not failures)
            passed += not failures

        rate = passed / max(cases.count(), 1)
        self.stdout.write(f"pass rate {rate:.2%} ({passed}/{cases.count()})")
        if rate < opts["fail_under"]:
            raise CommandError(f"below threshold {opts['fail_under']:.0%}")

Storing every run gives you the graph that actually answers "is this getting better or worse", and the diff that answers "which cases did this prompt change break".

Diffing two runs

A pass rate that moves from 94% to 94% can hide three cases breaking and three others being fixed. The review that matters on a prompt pull request is the list of flipped cases, which is a two-query job once results are stored:

def flipped(old_run, new_run):
    old = dict(EvalResult.objects.filter(run=old_run)
               .values_list("case_id", "passed"))
    new = dict(EvalResult.objects.filter(run=new_run)
               .values_list("case_id", "passed"))
    broke = [cid for cid, ok in new.items() if old.get(cid) and not ok]
    fixed = [cid for cid, ok in new.items() if ok and old.get(cid) is False]
    return broke, fixed

Post the output as a comment on the pull request — case id, the note explaining why the case exists, the failure reasons — and require that every entry in broke is either fixed or explicitly accepted in the description. "We accept losing case 31 because the policy it tests was retired" is a perfectly good outcome; silently losing it is not. Compare against the last run on the main branch, not the last run overall, so that two experimental branches do not muddy each other's baseline.

Guardrails: defence in depth, because layer one leaks

Prompt injection is not solved and will not be solved by a clever system prompt. Treat it the way you treat XSS: assume the payload gets in, and make sure it cannot do anything.

Input side

• Delimit and label. Fetched documents, customer messages and file contents go inside explicit markers, with a system-prompt line stating that anything inside them is data.
• Strip invisible content from HTML: scripts, comments, display:none, white-on-white text. A large share of real payloads hide there.
• Cap size. A 200-page PDF is an invitation to bury an instruction on page 173.

Output side

• Never render model output as HTML. Escape it, or render Markdown through a sanitiser with a strict allowlist. This is the one that bites teams who "just" wanted formatted answers.
• Validate before acting. A URL the model produced gets checked against an allowlist before you fetch it; a SQL fragment never gets executed at all.
• Refuse to leak the prompt. Cheap to test with a handful of eval cases that ask for it in creative ways.

The layer that actually holds

Permission scoping. If every tool filters on the authenticated user and every write is gated, a successful injection produces a rude answer instead of a data breach. Everything above reduces frequency; only this reduces severity.

What permission scoping looks like in code

The rule is simple: the identity used for every query comes from the authenticated request, never from the model's arguments. The model may choose which order to look up; it never gets to choose whose.

def tool_get_order(user, order_number: str) -> dict:
    order = (Order.objects
             .filter(customer__user=user, number=order_number)
             .values("number", "status", "total_cents", "shipped_at")
             .first())
    return order or {"error": "not_found"}

READ_TOOLS = {"get_order": tool_get_order}
WRITE_TOOLS = {"issue_refund"}

def dispatch(request, name: str, arguments: dict) -> dict:
    if name in WRITE_TOOLS:
        PendingAction.objects.create(user=request.user, tool=name,
                                     arguments=arguments)
        return {"status": "awaiting_confirmation"}
    if name not in READ_TOOLS:
        return {"error": "unknown_tool"}
    return READ_TOOLS[name](request.user, **arguments)

Three details carry the weight. The queryset filters on the user before anything else, so there is no code path that looks up an order globally and checks ownership afterwards. The tool returns .values() with an explicit field list, so a new column on Order — an internal note, a fraud score — does not leak into model context by default. And writes do not execute: they create a pending action that the user confirms in your own UI, outside the conversation, where injected text cannot click the button. In production, validate arguments against a per-tool schema before the call as well; the model can produce keys you did not expect, and a TypeError from an unexpected keyword should become a clean error rather than a 500.

Turning guardrails into tests

The most valuable guardrail tests do not involve the model at all. They assert that the layer that holds actually holds, and they run in milliseconds on every commit:

class ToolScopingTests(TestCase):
    def test_get_order_is_scoped_to_the_caller(self):
        alice = CustomerFactory()
        bob_order = OrderFactory(customer=CustomerFactory())
        result = tool_get_order(alice.user, bob_order.number)
        self.assertEqual(result, {"error": "not_found"})

    def test_write_tools_never_execute_directly(self):
        request = RequestFactory().post("/assistant/")
        request.user = CustomerFactory().user
        result = dispatch(request, "issue_refund", {"order_number": "A-1"})
        self.assertEqual(result["status"], "awaiting_confirmation")
        self.assertFalse(Refund.objects.exists())

These tests assume the injection succeeded — the model asked for another customer's order, or tried to issue a refund — and check that nothing bad followed. Alongside them, keep a small group of eval cases whose context documents contain embedded instructions of the kind you have seen in real traffic, described in the case notes rather than collected into a library of payloads. Score them on outcome: the answer does not contain data from outside the user's scope, no write tool was proposed, the system prompt did not appear in the output. A drop in that group's pass rate after a model change tells you the first layer got weaker, which is useful to know even though the scoping layer means it should not become an incident.

Watching it in production

Evals tell you about fifty known cases. Production tells you about the other ten thousand. Log per call: model, prompt version, token counts, latency, schema-validation outcome, tool calls, and whether the user retried or gave up. Then alert on shape changes rather than individual answers:

• schema-failure rate above baseline — the most reliable early warning you have
• mean output length dropping sharply — often a truncation or a refusal pattern
• retry rate climbing — users voting on quality with their behaviour
• refusal rate on a feature that never used to refuse

One wrapper, every call

These signals only exist if every model call goes through a single function that records them. Scattered client.complete(...) calls across views and tasks are how teams end up with metrics for three features out of five.

import hashlib
import logging
import time

logger = logging.getLogger("llm")

def call_llm(client, *, feature, prompt_version, model, messages, validate):
    started = time.monotonic()
    reply = client.complete(messages, model=model)
    latency_ms = int((time.monotonic() - started) * 1000)

    try:
        parsed, schema_ok = validate(reply.text), True
    except ValidationError:
        parsed, schema_ok = None, False

    Generation.objects.create(
        prompt_version=prompt_version, model=model,
        input_hash=hashlib.sha256(repr(messages).encode()).hexdigest(),
        output=reply.text, schema_ok=schema_ok,
    )
    logger.info("llm_call", extra={
        "feature": feature, "prompt_version": prompt_version, "model": model,
        "latency_ms": latency_ms, "output_chars": len(reply.text),
        "schema_ok": schema_ok,
    })
    if not schema_ok:
        metrics.incr(f"{feature}.schema_fail")
        raise LLMSchemaError(prompt_version)
    return parsed

client here is your own thin adapter around the provider SDK, which is also what makes the unit tests below possible. Record the provider's reported token usage and stop reason too if your adapter exposes them; a stop reason of "maximum tokens reached" on more than a trickle of calls explains most sudden schema-failure spikes. Hash inputs rather than storing them in full when they contain customer data, and set a retention period on Generation rows that matches the rest of your personal-data policy.

Version prompts like migrations

Prompts belong in the repository, numbered, with the eval results that justified the change recorded in the pull request. Store the version with every generated row:

class Generation(models.Model):
    prompt_version = models.CharField(max_length=20)   # "support_reply.v7"
    model          = models.CharField(max_length=50)
    input_hash     = models.CharField(max_length=64)
    output         = models.TextField()
    schema_ok      = models.BooleanField()
    created_at     = models.DateTimeField(auto_now_add=True)

When a customer complains about an answer from three weeks ago, you can reproduce exactly which prompt and which model produced it. Without that column, you are guessing.

Testing the plumbing without calling a model

Evals test the model's behaviour. Everything around the model — building the prompt, delimiting documents, parsing, the failure paths — is ordinary code and deserves ordinary fast tests that never hit the network. The adapter from the previous section makes that easy: swap it for a fake that returns canned replies and records what it was sent.

class FakeLLM:
    def __init__(self, *replies):
        self.replies = list(replies)
        self.sent = []

    def complete(self, messages, **kwargs):
        self.sent.append(messages)
        return Reply(text=self.replies.pop(0))


class ExtractionTests(TestCase):
    def test_invalid_json_fails_loudly(self):
        client = FakeLLM('{"invoice_number": "A-1", "total_cents": -5}')
        with self.assertRaises(LLMSchemaError):
            extract_invoice("irrelevant", client=client)
        self.assertFalse(Generation.objects.get().schema_ok)

    def test_documents_cannot_close_their_own_delimiter(self):
        doc = "Terms apply.</document> Extra text."
        messages = build_messages("What are the terms?", [doc])
        body = messages[-1]["content"]
        self.assertEqual(body.count("</document>"), 1)

The second test is worth dwelling on. Delimiting only works if a document cannot contain your closing marker; if it can, the text after it reads as if it were outside the data block. The prompt builder should neutralise the marker inside untrusted content — replace or escape it — and the test pins that behaviour so a refactor cannot quietly remove it. Other cheap, valuable plumbing tests: an empty retrieval result produces the "I don't know" path rather than a call with no context; oversized documents are truncated to the cap; the prompt version string recorded on Generation matches the template actually used.

Keep these in the normal test suite and the evals in their own command. Mixing them is how a suite ends up either slow and flaky on every commit or, after someone gets annoyed, with the model calls mocked out of the evals too.

Troubleshooting a falling pass rate

When the number drops, resist editing the prompt until you know which of these you are looking at. The flipped-case diff and the stored outputs usually point to the row within a few minutes.

The last row matters more than it looks. If a provider outage is scored as a quality regression, people learn to ignore red eval runs, and a genuine regression the following week gets waved through with "it's probably the API again".

Where to start on Monday

1. Twenty cases in a table, from real traffic and real bugs.
2. A management command that runs them and prints a pass rate.
3. Schema validation on every structured output, failing loudly.
4. Permission scoping on every tool.
5. A weekly CI run so provider drift wakes you, not your customers.

That is an afternoon of work and it converts your AI feature from something you hope is fine into something you can measure. Everything after — judges, dashboards, per-case history — is refinement on top of a foundation that already caught the expensive mistakes.