Structured Output From LLMs: Function Calling and JSON Mode in Practice
Before native JSON mode and function calling matured, we were parsing free-text LLM output with regular expressions, and it broke constantly — a model would add a friendly preamble, wrap the JSON in markdown fences inconsistently, or rename a field halfway through a long response. Moving to structured output APIs eliminated an entire class of parsing failures that used to account for a meaningful share of our production incidents.
Schema design matters more than people expect. We learned to keep required fields to a minimum and make everything else optional with sensible defaults, because a model under pressure to fill a required field it doesn't have good information for will often hallucinate a plausible-looking value rather than admit uncertainty. Adding an explicit "confidence" or "source" field per claim gave us a lever to catch that.
Function calling changed how we build agents that take real actions — booking, updating records, sending notifications — because the model's job narrows to selecting the right function and populating typed arguments, which it's noticeably more reliable at than free-form generation. On one internal tools project, switching from prompted JSON to native function calling cut malformed-argument errors from around 8% of calls to under 1%.
We still validate everything server-side regardless of how structured the output claims to be. Nested objects, enums, and array lengths are exactly the places where models drift under long contexts or unusual inputs, and a validation layer that rejects malformed output and retries with the error message fed back to the model catches the vast majority of these before they reach a database or downstream system.
The pattern that's worked best across projects: strict schema, generous retries with error feedback, and a hard fallback to a human-reviewed queue after two failed attempts rather than looping indefinitely or accepting a best-effort malformed response. It's a small amount of extra engineering that turns "usually works" into something we're comfortable putting in front of paying customers.
