Regression Testing

Definition

Regression testing in AI systems is the practice of running a defined test suite after every change to verify that previously working behaviors still work correctly. Unlike feature testing (does the new thing work?), regression testing asks "did we break anything that was already working?" In AI agent systems, this is particularly challenging because changes can have unexpected side effects: a prompt change that improves performance on one task may degrade another, or a model upgrade may shift behavior in subtle ways that aren't immediately obvious.

Engineering Context

AI regression testing is harder than traditional software testing because outputs aren't bit-identical. Use semantic similarity thresholds or LLM-as-judge scoring rather than exact matching. Maintain a regression test suite of 50+ cases covering known edge cases and past failures. Integrate into CI/CD: any prompt or model change triggers the regression suite, and merges are blocked if key metrics degrade beyond defined thresholds. Track regression test results over time to identify trends—slow degradation is harder to catch than sudden failures. When regressions are found in production, add the failing case to the regression suite before fixing it, to prevent recurrence.

Related Terms