Testing AI Features: What Unit Tests Can and Can't Catch
Our standard unit test suite works fine for the deterministic parts of an AI feature — the prompt template renders correctly, the API client retries on a 500, the output parser handles malformed JSON without crashing. Those tests caught real bugs early and we kept all of them. What they never caught was the feature actually being wrong: a summarization endpoint that ran clean through every unit test while quietly dropping a critical caveat from the source text.
We added a second layer: a fixed evaluation set of about 150 real (anonymized) inputs with human-reviewed expected properties, not exact-match expected outputs. For the summarizer, that meant checking whether specific required facts appeared in the output, not whether the output matched a golden string word for word, since the model's phrasing legitimately varies run to run even at low temperature.
We run that eval set on every prompt or model change and track pass rate over time, treating a drop as a regression the same way we'd treat a failing test — except the threshold is a percentage, not a boolean. We set the bar at 95% fact-coverage pass rate before a change ships, and log which specific cases failed so we're not just staring at an aggregate number.
The layer unit tests and eval sets both miss is drift in production traffic that doesn't resemble the eval set. A support-ticket classifier that performed well on our test set started failing on a new product line's tickets three months after launch, because the phrasing was different enough that it fell outside what we'd evaluated. We now sample live production outputs weekly for human spot review specifically looking for input patterns our eval set doesn't cover.
The honest summary: unit tests verify the code around the model. Eval sets verify the model's behavior against known cases. Production sampling catches the cases neither of the first two anticipated. Skipping any one of the three leaves a real gap, and we learned that the expensive way — by shipping features that passed every test we had and still misled users.
