Evaluating and Guarding LLM Features: Golden Sets, Regression Tests, and Prompt-Injection Defence

Traditional code fails loudly. An LLM feature fails quietly and plausibly: the same prompt that worked last month now returns a confident, well-formatted, wrong answer. Nothing throws. No test goes red. A customer notices before you do — unless you built the thing that notices.

Why normal tests do not cover this

assertEqual(output, expected) is useless when the output is prose that may legitimately vary. But the opposite conclusion — "you cannot test AI" — is wrong and expensive. You can test three things reliably:

• Structure. Did it return valid JSON with the required fields, an allowed enum value, a number in range?
• Grounding. Does every claim appear in the source documents you supplied?
• Behaviour on known inputs. On these fifty cases, does it still do what it did before?

That is 90% of production quality. The remaining 10% is taste, and taste is what humans review.

The golden set: fifty rows beat a dashboard

Start with a table of cases: input, what a good answer contains, what it must never contain. Keep it in the database so support staff can add cases without a deploy.

class EvalCase(models.Model):
    feature      = models.CharField(max_length=50)      # "support_reply"
    prompt       = models.TextField()
    context      = models.JSONField(default=dict)       # retrieved docs, user state
    must_include = models.JSONField(default=list)       # ["14 days", "free of charge"]
    must_exclude = models.JSONField(default=list)       # ["guarantee", "lifetime"]
    expects_tool = models.CharField(max_length=50, blank=True)
    notes        = models.TextField(blank=True)
    is_active    = models.BooleanField(default=True)

Where do cases come from? Three sources, in order of value:

1. Every bug. A wrong answer reported by a user becomes a case the same day. This is the single highest-value habit in the whole discipline.
2. Edge cases you already know. Empty context, hostile input, the customer with no orders, the question in another language.
3. Representative traffic. Sample real prompts, strip personal data, keep the shapes.

Fifty good cases catch more than five hundred generated ones. Resist the urge to bulk-generate.

Scoring without a human

Run the cheap checks first — they are deterministic, instant and free.

def score_case(case, output, tool_calls):
    failures = []

    for phrase in case.must_include:
        if phrase.lower() not in output.lower():
            failures.append(f"missing: {phrase}")

    for phrase in case.must_exclude:
        if phrase.lower() in output.lower():
            failures.append(f"forbidden: {phrase}")

    if case.expects_tool and case.expects_tool not in tool_calls:
        failures.append(f"did not call: {case.expects_tool}")

    return failures

For structured output, validate against a schema and treat a parse failure as a hard fail. Never "repair" malformed JSON silently in production — you will hide a degradation that is about to get worse.

class Extraction(BaseModel):
    invoice_number: str
    total_cents: int = Field(ge=0)
    currency: Literal["EUR", "USD", "GBP"]
    due_date: date

try:
    parsed = Extraction.model_validate_json(output)
except ValidationError as exc:
    metrics.incr("extraction.schema_fail")
    raise

Model-as-judge, used narrowly

For qualities you cannot grep for — tone, completeness, whether an answer is grounded in the sources — a second model call can score the first. It works if you keep it narrow.

JUDGE = """You are grading one answer. Reply with JSON only.

Question: {question}
Sources: {sources}
Answer: {answer}

grounded: true only if every factual claim appears in Sources.
complete: true only if the answer addresses the whole question.
reason: one sentence."""

Two rules keep judges honest. Ask for a boolean and a reason, not a score out of ten — numeric scores from models cluster around 7 and carry almost no signal. And calibrate against humans once: grade forty cases by hand, compare, and if the judge disagrees more than one time in ten, fix the rubric before you trust it anywhere.

Wire it into CI, but not into every commit

Evals cost money and take minutes, so run them where they pay off: on changes to prompts, tools or model configuration, and on a schedule.

name: llm-evals
on:
  pull_request:
    paths: ["**/prompts/**", "**/tools/**", "**/ai/**"]
  schedule:
    - cron: "0 6 * * 1"          # Monday morning, catches provider drift

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python manage.py run_evals --feature support_reply --fail-under 0.92

The weekly run matters as much as the pull-request run. Nothing in your repository changes when a provider updates a model — but your pass rate does, and you want to hear it from CI on Monday rather than from a customer on Thursday.

class Command(BaseCommand):
    def handle(self, *args, **opts):
        cases = EvalCase.objects.filter(feature=opts["feature"], is_active=True)
        run = EvalRun.objects.create(feature=opts["feature"], git_sha=git_sha())

        passed = 0
        for case in cases:
            output, calls = run_feature(case.prompt, case.context)
            failures = score_case(case, output, calls)
            EvalResult.objects.create(run=run, case=case, output=output,
                                      failures=failures, passed=not failures)
            passed += not failures

        rate = passed / cases.count()
        self.stdout.write(f"pass rate {rate:.2%} ({passed}/{cases.count()})")
        if rate < opts["fail_under"]:
            raise CommandError(f"below threshold {opts['fail_under']:.0%}")

Storing every run gives you the graph that actually answers "is this getting better or worse", and the diff that answers "which cases did this prompt change break".

Guardrails: defence in depth, because layer one leaks

Prompt injection is not solved and will not be solved by a clever system prompt. Treat it the way you treat XSS: assume the payload gets in, and make sure it cannot do anything.

Input side

• Delimit and label. Fetched documents, customer messages and file contents go inside explicit markers, with a system-prompt line stating that anything inside them is data.
• Strip invisible content from HTML: scripts, comments, display:none, white-on-white text. A large share of real payloads hide there.
• Cap size. A 200-page PDF is an invitation to bury an instruction on page 173.

Output side

• Never render model output as HTML. Escape it, or render Markdown through a sanitiser with a strict allowlist. This is the one that bites teams who "just" wanted formatted answers.
• Validate before acting. A URL the model produced gets checked against an allowlist before you fetch it; a SQL fragment never gets executed at all.
• Refuse to leak the prompt. Cheap to test with a handful of eval cases that ask for it in creative ways.

The layer that actually holds

Permission scoping. If every tool filters on the authenticated user and every write is gated, a successful injection produces a rude answer instead of a data breach. Everything above reduces frequency; only this reduces severity.

Watching it in production

Evals tell you about fifty known cases. Production tells you about the other ten thousand. Log per call: model, prompt version, token counts, latency, schema-validation outcome, tool calls, and whether the user retried or gave up. Then alert on shape changes rather than individual answers:

• schema-failure rate above baseline — the most reliable early warning you have
• mean output length dropping sharply — often a truncation or a refusal pattern
• retry rate climbing — users voting on quality with their behaviour
• refusal rate on a feature that never used to refuse

Version prompts like migrations

Prompts belong in the repository, numbered, with the eval results that justified the change recorded in the pull request. Store the version with every generated row:

class Generation(models.Model):
    prompt_version = models.CharField(max_length=20)   # "support_reply.v7"
    model          = models.CharField(max_length=50)
    input_hash     = models.CharField(max_length=64)
    output         = models.TextField()
    schema_ok      = models.BooleanField()
    created_at     = models.DateTimeField(auto_now_add=True)

When a customer complains about an answer from three weeks ago, you can reproduce exactly which prompt and which model produced it. Without that column, you are guessing.

Where to start on Monday

1. Twenty cases in a table, from real traffic and real bugs.
2. A management command that runs them and prints a pass rate.
3. Schema validation on every structured output, failing loudly.
4. Permission scoping on every tool.
5. A weekly CI run so provider drift wakes you, not your customers.

That is an afternoon of work and it converts your AI feature from something you hope is fine into something you can measure. Everything after — judges, dashboards, per-case history — is refinement on top of a foundation that already caught the expensive mistakes.