Editorial illustration for Understanding JSON Mode, Function Calling, and Structured Output in LLMs
Understanding JSON Mode, Function Calling, and...
Understanding JSON Mode, Function Calling, and Structured Output in LLMs
Most tutorials I’ve written end with a free‑text answer: you ask, the model writes a paragraph, you display it. That works for chat‑style apps, but it falls short when the next step needs a precise data structure. Imagine a workflow that must pull a name, date, and amount from a paragraph, feed those values into a database, or fire off an API call. A plain string isn’t useful there.
Enter two distinct routes for getting machine‑readable results from large language models: JSON Mode and Function Calling, sometimes called tool use. Both aim to replace “a wall of text” with something a program can parse, yet they solve different problems. OpenAI has layered a stricter variant on top of Function Calling—Structured Outputs—that enforces a schema more rigorously.
Understanding how each mechanism operates, what constraints they impose, and when one is preferable to the other is essential for anyone building production‑grade AI services. Let’s unpack the three approaches, see them in action, and map out the scenarios where each shines.
But let's get back to machine-readable outputs now, and we'll talk more about agentic AI workflows and Function Calling in some other post. A stricter variation of Function Calling is Structured Outputs. Even if Function Calling guides the model to provide an output following a defined schema, this is not really hard-constrained.
In practice, this means that some deviations from this defined schema may still occur. Such deviations may be: - A field marked as required that is, in fact, omitted if the model struggles to figure out its value - Extra fields not defined in our schema are added - A field defined as integer comes back as a string"32" instead of32 …and so on. This happens because, in Function Calling, the model is trying to follow the schema, but this is still a best-effort generation.
Like any LLM output, the output here is still fundamentally tokens being predicted one by one, with the schema being just a strong hint. There's still a good chance for that token-by-token generation to be derailed somewhere along the route and produce outputs that deviate from the defined schema. Structured Outputs, on the other hand, takes Function Calling one step further by guaranteeing that every field in the defined schema will always appear in the output exactly as defined, with no surprises, no missing or extra fields.
The key differentiator is that OpenAI uses constrained decoding behind the scenes. This means that at each token step, the model is only allowed to generate tokens that keep the output valid according to the schema.
Why this matters
We’ve seen LLMs answer in prose for ages, but the shift toward JSON mode and function calling forces us to think about downstream pipelines. When a model returns a structured object, developers can feed it directly into databases, dashboards, or other services without fragile parsing. Yet the article reminds us that function calling already nudges the model toward a schema, and Structured Outputs tighten that guidance even further.
Does the extra rigidity justify the loss of flexibility? For founders building products that rely on reliable data extraction, the trade‑off may be worth exploring, but the piece stops short of proving a clear performance edge. Researchers can measure consistency across prompts, but the summary leaves open which approach scales best under real‑world noise.
In practice, we must experiment with both free‑text and machine‑readable paths, weighing implementation cost against the certainty of downstream consumption. Until broader benchmarks emerge, the promise of cleaner pipelines remains tentative, and our teams should proceed with measured optimism.
Further Reading
- Introducing function calling and structured output for open-source LLMs - Baseten
- The guide to structured outputs and function calling with LLMs - Agenta
- Structured model outputs | OpenAI API - OpenAI Developer Documentation
- When should I use function calling, structured outputs or JSON mode? - Vellum
- Prompting vs JSON Mode vs Function Calling vs Constrained Decoding - BoundaryML
Related Reading
LLMs & Generative AI LWiAI Podcast #228: OpenAI unveils GPT-5.2, Runway rolls out first world model
LLMs & Generative AI OpenAI's Codex powers Lovable AI, letting millions create apps from text
LLMs & Generative AI Google releases FunctionGemma, a tiny model for natural-language mobile control
From Archives Audit AI tools: inventory, VPN/zero-trust, continuous fingerprinting
From Archives