Blog · · 6 min read
Shipping AI features that survive a model update
Structured outputs, evaluation harnesses, and why prompt tuning is not a substitute for tests.
An AI feature that works in a demo and one that survives a model update are different pieces of software. The difference is almost entirely test infrastructure.
Validate at the boundary
Free text coming out of a model and into your database is a bug waiting for a slow afternoon. Every extraction feature I ship defines a schema first, validates against it, and treats a validation failure as a retry with a narrower prompt, then a human queue. No partial writes.
Keep a golden set
Thirty real examples with known-correct outputs will tell you more about a prompt change than any amount of eyeballing. Providers update models underneath you. A golden set turns that from an outage into a diff.
Decide what happens when it is wrong
Clients ask about accuracy. The better question is what the system does with the cases it gets wrong: save them silently, flag them, or ask a person. That answer shapes the product more than the model choice does.