Prototyping AI Features Before the Model Behavior Is Finalized

On an AI-assisted document review tool we built for a legal services client, the underlying model was still being tuned when the interface design work needed to start. Waiting for final model behavior would have pushed the timeline out by months, so we built the entire first design pass against hand-written, deliberately messy fake outputs, including the kind of overconfident, slightly wrong answers a real model produces, not just its best-case responses.

That decision changed what we designed. If we'd only mocked up the model's best outputs, we'd have shipped an interface with no room for the model being wrong, which is not how these systems actually behave. Designing against realistic failure modes from day one meant the uncertainty-indicator UI, the 'flag this for human review' action, and the inline citation pattern were all built into the core flow rather than bolted on after the model's real limitations became apparent post-launch.

We also prototype with a Wizard of Oz approach for genuinely novel interactions, where a human on our team manually generates the 'model' responses in a live session with real users, before any model integration exists at all. This is slower per session but far cheaper than building against an API that will change twice before launch, and it tells you quickly whether the interaction concept works at all, independent of model quality.

One thing we got wrong initially: we designed a single, generic loading state for 'the AI is thinking,' assuming response time would be roughly constant. Real model latency varied by an order of magnitude depending on document length, so a static spinner either felt broken on long documents or wastefully elaborate on short ones. We replaced it with a state that reflects actual processing stages, chunking the document, retrieving relevant sections, drafting a response, which also happened to make the tool feel more trustworthy, since users could see roughly what was happening.

The core lesson: don't wait for the model to stabilize to start designing the interface around it, but do make sure your prototypes include the model's realistic failure modes, not just its highlight reel. The interface work that matters most in AI products is almost always the work that accounts for the system being wrong sometimes.