Editorial illustration for Understanding JSON Mode, Function Calling, and Structured Output in LLMs
Understanding JSON Mode, Function Calling, and...
Ask a large language model for a JSON object and you'll often get a broken one. It might look right, but the data types will be wrong or key fields will just vanish. This is the core problem of building anything real with LLMs.
Developers initially hoped Function Calling would solve it. The idea was simple: give the model a schema, a blueprint for the data you want. The model would then try its best to fill in the blanks.
That "try its best" part is the flaw. A schema in Function Calling is a strong suggestion, not a rule. The model, generating one token at a time, can still wander off script.
It might skip a required field it finds confusing. It might add an extra one. It will absolutely return numbers as strings inside quotes.
Structured Outputs exists to end the guessing. It’s Function Calling with handcuffs. Where one politely requests, the other mandates.
The technical magic is constrained decoding. At every single step of generation, the model is only permitted to choose the next token if it keeps the entire output valid against your schema. No deviations.
No surprises.
Obtaining machine-readable outputs from LLMs and choosing the appropriate approach for doing so can make a huge difference in the reliability and maintainability of any AI application. Freetext responses are great for conversational interfaces, but the moment our LLM is a component in a larger system (like feeding data downstream, triggering actions, populating databases, etc.), structured responses are essential.
The choice isn't about which tool is better. It's about what breaks your system. If a malformed integer crashes your database import, you need Structured Outputs.
That guarantee is the product. If you're building a chatbot that can gracefully re-ask a question when a field is missing, Function Calling offers plenty of power with less overhead. One is for prototypes and polite conversations.
The other is for production, where software must work exactly as specified, every single time.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Audit AI tools: inventory, VPN/zero-trust, continuous fingerprinting
From Archives Adobe adds AI Assistant to Photoshop, Premiere, Illustrator, InDesign, Frame.io
Common Questions Answered
Why does asking a large language model for a JSON object often result in broken output?
LLMs frequently produce JSON with incorrect data types or missing key fields, even when the structure appears correct at first glance. This fundamental problem occurs because models generate text token-by-token without strict validation, making it unreliable for production systems that depend on precise data formatting.
What is the main limitation of Function Calling when used with LLMs?
Function Calling relies on a schema as a blueprint for desired data, but the model only tries its best to follow it without guarantees. The 'try its best' approach means the schema is not enforced, leaving room for malformed outputs that can break downstream systems.
How do Structured Outputs differ from Function Calling in terms of reliability?
Structured Outputs provide a guarantee that the model will return data exactly as specified in the schema, making them suitable for production environments where failures cannot be tolerated. Function Calling, by contrast, offers less overhead but lacks this strict enforcement, making it better suited for prototypes or applications that can gracefully handle missing fields.
When should you choose Structured Outputs over Function Calling?
You should use Structured Outputs when a malformed response would crash your system, such as during database imports where incorrect data types cause failures. Structured Outputs are the right choice for production systems where software must work exactly as specified every single time without exceptions.
What use cases are appropriate for Function Calling despite its lack of strict guarantees?
Function Calling works well for chatbots and conversational applications that can gracefully re-ask questions when fields are missing or incomplete. It also suits prototype development where less overhead is preferred and the application can handle imperfect responses without critical failures.
Further Reading
- Structured model outputs | OpenAI API — OpenAI Developer Documentation
- Introducing function calling and structured output for open-source LLMs — Baseten
- The guide to structured outputs and function calling with LLMs — Agenta
- When should I use function calling, structured outputs or JSON mode? — Vellum
- Prompting vs JSON Mode vs Function Calling vs Constrained Decoding — BoundaryML
