How to Build a Test Suite for LLM-Powered Features

Traditional software testing assumes that the same input should produce the same output. LLM-powered features break that assumption. Two valid answers can use different wording, order, or levels of detail. An answer can also sound excellent while being factually wrong.

That does not make AI features untestable. It means teams must stop treating exact text as the contract. The contract is usually a combination of facts, constraints, actions, and prohibited behavior.

A useful LLM test suite evaluates those dimensions separately. It combines deterministic assertions, model-based scoring, human review, and repeated sampling. The goal is not to prove that a model is universally correct. It is to decide whether a specific feature is reliable enough for its intended use.

Start with the feature contract

Do not begin by collecting hundreds of prompts. First define what the feature is supposed to do.

Consider an assistant that drafts replies to customer support tickets. Its contract might require it to:

• identify the customer’s actual request;
• use only approved policy information;
• ask for missing details instead of inventing them;
• avoid exposing internal notes or personal data;
• produce a concise draft in the company’s tone;
• escalate requests involving refunds above a set amount.

These requirements are more testable than “give a helpful response.” Each can become an assertion or rubric criterion.

Separate hard constraints from preferences. A fabricated refund policy is a failure. A response that is slightly too formal may be acceptable. If both appear as a single quality score, serious defects can hide behind good style.

For every criterion, define the cost of failure. This determines whether the release gate should require perfect compliance, a high pass rate, or merely a monitored baseline.

Build cases from risks, not averages

A random sample of ordinary user requests will overestimate quality. Most production inputs are easy. Incidents emerge from ambiguity, missing context, conflicting instructions, malformed data, and rare business rules.

Construct the test set around risk categories:

• common, well-formed requests;
• ambiguous requests requiring clarification;
• incomplete or contradictory context;
• boundary values and uncommon policy cases;
• prompt injection and instruction conflicts;
• unsupported languages, formats, or domains;
• sensitive data and authorization boundaries;
• tool failures, timeouts, and stale results;
• long conversations that approach context limits.

Production examples are valuable, but redact them and preserve their structural difficulty. Synthetic cases can expand coverage, especially for rare failures, but engineers should review them. Model-generated tests often inherit the same assumptions as the system under test.

Each case should include the input, relevant context, expected properties, forbidden outcomes, severity, and a short rationale. Where a single reference answer is misleading, store several acceptable examples or an evaluation rubric instead.

Keep separate development and holdout sets. If engineers repeatedly tune prompts against every case, the suite becomes training data and stops measuring generalization.

Use assertions before judges

The cheapest and clearest checks should run first. Many LLM failures are ordinary software failures wrapped in natural language.

Use deterministic assertions for properties such as:

• valid JSON and schema compliance;
• required fields and allowed enum values;
• citation or identifier formats;
• maximum length;
• banned phrases or sensitive-data patterns;
• correct tool name and argument types;
• expected escalation flags;
• arithmetic recomputed by trusted code.

For a support assistant, the test harness can verify that a high-value refund triggers escalation without asking another model for an opinion. For structured extraction, compare normalized field values rather than prose.

Avoid brittle exact-string assertions unless wording itself is contractual. Normalization, set comparison, semantic fields, and domain validators usually produce more useful tests.

Deterministic checks provide actionable failures. “Missing required account ID” is easier to fix than “judge score fell from 0.86 to 0.81.”

Judge semantics with explicit rubrics

Some properties cannot be reduced to code. Helpfulness, completeness, groundedness, and appropriate tone require semantic evaluation. Human review remains the reference, but it is too slow for every build. LLM judges can provide scalable signals if their task is tightly defined.

Ask judges narrow questions. Instead of “rate this answer,” score individual dimensions with concrete anchors. For example:

• Groundedness: Are all factual claims supported by the supplied policy excerpt?
• Completeness: Does the response address every explicit customer request?
• Actionability: Does it state the next step and required information?
• Tone: Is it professional without making commitments the company cannot honor?

Use pass/fail judgments for release-critical criteria. If a scale is necessary, define what each point means and avoid averaging unrelated dimensions.

Validate judges against a human-labeled set before trusting them. Measure agreement by criterion and inspect disagreements. A judge may be reliable for tone but weak at domain-specific factuality.

Do not let the evaluated output influence the judge through hidden instructions. Delimit inputs, tell the judge to treat candidate text as data, and require structured results with a brief justification. Periodically change or audit the judge model; its behavior can shift after provider updates.

Test workflows, not just final answers

An AI feature that calls tools or modifies systems has two outputs: what it says and what it does. Final-answer evaluation alone can miss an incorrect search, an unauthorized action, or a tool call made with fabricated parameters.

Test the execution trace at stable boundaries. Verify that the system:

• selects an allowed tool for the user’s intent;
• supplies arguments supported by the conversation;
• respects authorization and confirmation rules;
• handles tool errors without pretending success;
• stops after the task is complete;
• does not execute instructions found in untrusted tool output.

Use mocked tools for most CI tests. They make failure modes reproducible and prevent tests from creating real tickets, payments, or database changes. Add a smaller number of sandbox integration tests to catch schema drift and provider-specific behavior.

Do not assert every internal reasoning step. That couples tests to an implementation detail and can penalize harmless changes. Assert observable decisions, calls, state transitions, and outcomes.

Measure variance and regressions

A single successful run is weak evidence. Sampling, routing, and provider infrastructure can make behavior vary even at low temperature.

Run critical cases multiple times. Track pass-at-one, pass rate across repeated samples, and worst-case failures. Five runs that pass four times are not equivalent to a deterministic pass, especially for actions involving money, permissions, or regulated data.

When comparing a new prompt or model, run both versions on the same cases and settings. Report regressions and improvements by severity and category rather than relying on one aggregate score.

A practical release report might include:

• zero new critical safety or authorization failures;
• no more than a defined regression budget on high-severity cases;
• schema validity above the required threshold;
• tool completion rate by workflow;
• latency and cost at representative context sizes;
• human review of all changed failures and a sample of passes.

Store the model identifier, prompt version, tool schemas, retrieval fixtures, generation settings, and evaluator version with every result. Without that metadata, a failed test is difficult to reproduce.

Put the suite in the delivery pipeline

Use a small, fast suite on every pull request. Reserve repeated sampling, adversarial cases, live integrations, and broad model comparisons for scheduled or pre-release runs.

Require a reason when updating expected behavior. Snapshot replacement should never be an automatic response to failures. A changed output may reveal a legitimate product decision, or it may normalize a regression.

Ownership also matters. QA engineers can design coverage and harnesses, but product and domain experts must define acceptable behavior. Security should own abuse and data-boundary criteria. Engineering should own tool correctness and reproducibility. No single team can infer the complete contract alone.

Takeaway

LLM QA works when teams test contracts instead of sentences. Encode hard rules as deterministic assertions, score semantic properties with validated rubrics, test tool effects separately, and measure repeated behavior. The result is not certainty; it is a defensible release decision based on known risks.