Multimodal AI: Combining Vision and Language Models in Production
We built a product inspection tool for a manufacturing client: photos from the line get analyzed for defects, and the results get written up in plain-language reports for floor supervisors. The obvious architecture was a vision model to detect and localize defects, then a language model to turn the structured output into a readable summary. The seam between the two is where almost every bug lived.
The first problem was information loss at the handoff. Early on, we passed only the vision model's top label and confidence score to the language model, which meant the write-up couldn't reference where on the part the defect was or how it compared to the last ten inspections. We changed the interface to pass bounding boxes, defect class, confidence, and a short structured history, and the reports immediately got more useful and more trustworthy to the supervisors reading them.
The second problem was cost and latency stacking. Running a vision model and then a separate LLM call sequentially added 1.8 seconds per image on average, which was fine for batch processing but too slow for the live-inspection mode the client also wanted. We split the pipeline: batch runs get the full two-stage treatment, live mode gets a lighter vision-only pass with template-based text generation and no LLM call at all.
We also had to handle disagreement between modalities directly instead of hiding it. When the vision model flagged a defect but the language model, given the same image caption, described the part as fine, that mismatch was a real signal — usually a borderline confidence case. We now surface those as 'needs review' rather than letting the language model silently smooth over uncertainty the vision model was trying to express.
The lesson that generalized beyond this project: multimodal systems fail less often inside either model and more often in the translation layer between them. Treat that interface as a first-class piece of the architecture — schema it, test it, and don't let it collapse rich structured output into a single string just because that's the easiest thing to prompt with.
