Structured Outputs

Definition: Structured outputs are LLM responses constrained to match a predefined schema, typically JSON Schema, a Pydantic model, a Zod or TypeScript type, or a formal grammar. Instead of trusting the model to produce valid JSON by instruction alone, the inference engine compiles the schema into a grammar and masks every token that breaks it. The result is output your code can parse on the first try. The guarantee covers the shape of the answer, not whether the values are true.

TL;DR: Structured outputs turn "please reply in JSON" into a runtime guarantee. OpenAI launched them in its API on August 6, 2024, and Anthropic followed with a public beta on November 14, 2025 that reached general availability on February 4, 2026. Format is guaranteed, correctness is not, so validate values in code. In Taskade, the Ask Agent with Structured Output action turns agent replies into typed automation variables.

For production systems that turn LLM output into code, database rows, API calls, or UI components, structured outputs are the difference between a demo that works sometimes and a product that ships. They are the cousin of function calling: same constrained-decoding machinery, different framing. Function calling says "pick a tool and give its arguments." Structured outputs say "give me your answer in this shape."

Why Unconstrained JSON Was a Problem

Before structured outputs, developers asked the model nicely: Return a JSON object with fields "title" (string) and "rating" (integer 1-5). The model complied most of the time. The rest of the time you got markdown fences around the JSON, extra prose ("Here is the JSON:"), missing required fields, numbers as strings, trailing commas, broken escapes, or a different schema once the prompt got complex. At scale, even a small failure rate means a steady stream of parse errors, retries, and on-call pages. Retries with the error message sometimes work, but they burn tokens and add latency.

JSON mode was the first fix, and it only solved half the problem. OpenAI's own documentation still warns that JSON mode "will not guarantee the output matches any specific schema, only that it is valid and parses without errors" (OpenAI Structured Outputs guide). The same guide notes that if you forget to tell the model to produce JSON, it "may generate an unending stream of whitespace" until it hits the token limit, which is why the API throws an error when the string "JSON" does not appear in the context.

Structured Outputs Timeline (2023-2026)

How Constrained Decoding Works

Constrained decoding rests on one insight: an LLM generates one token at a time by sampling from a probability distribution. If you mask that distribution at each step so that only schema-valid continuations remain, the model cannot produce a malformed answer. Anthropic describes its version this way: "Structured outputs work by compiling your JSON schemas into a grammar that constrains Claude's output." The first request with a new schema pays a compile cost, and compiled grammars are cached for 24 hours from last use (Anthropic docs). OpenAI documents the same first-request latency for a new schema.

At each decoding step, the runtime consults the grammar and an incremental parser. Only tokens that can legally continue the partial output stay in play, and the sampler picks from that masked distribution. The sketch below shows one step. The candidate tokens are illustrative, not measured.

Schema:          { "rating": integer, enum [1, 2, 3, 4, 5] }
Output so far:   {"rating":

Candidate next token     Legal under the grammar?    Result
-----------------------  --------------------------  ----------------
 4                       yes                         kept
 "four"                  no  (string, not integer)   masked out
 4.5                     no  (not in the enum)       masked out
 null                    no  (field is not nullable) masked out
 Here is                 no  (not JSON)              masked out

The sampler then picks only from the kept tokens.

The engineering challenge is speed. A vocabulary holds tens of thousands of tokens, and checking each one against a grammar on every step is expensive. Outlines treats generation as moves through a finite-state machine and precomputes an index over the vocabulary, which guides output with regular expressions and context-free grammars (Willard and Louf, 2023). Nested JSON needs a context-free grammar, where the parser must track a stack of open objects and arrays. XGrammar handles that case by prechecking the "context-independent" tokens ahead of time and interpreting only the "context-dependent" ones at runtime, and it overlaps grammar work with GPU execution (Dong et al., 2024).

Three Levels of Structure

JSON mode (first shipped by OpenAI in late 2023) only guarantees valid JSON. Schema-guided mode (OpenAI Structured Outputs in 2024, Anthropic structured outputs in 2025, and the structured output mode in the Gemini API) guarantees valid JSON and schema conformance. Strict typed mode wraps schema-guided mode in SDK helpers that hand your code a typed object.

Prompt-Only JSON vs JSON Mode vs Constrained Decoding vs Strict Tool Calls

Anthropic frames the split cleanly: JSON outputs "control Claude's response format (what Claude says)," while strict tool use "validates tool parameters (how Claude calls your functions)." You can use both in one request, which is the common setup for agentic workflows that need reliable tool calls and a structured final answer.

OpenAI vs Anthropic: Schema Rules Side by Side

Both major hosted APIs support a subset of JSON Schema, and the subsets differ. A schema that compiles on one provider can fail with a 400 error on the other, so check the rules before you port an agent between them.

Sources: OpenAI supported schemas and Anthropic JSON Schema limitations.

One interaction matters in practice. At Anthropic, a nullable field written as "type": ["string", "null"] counts as a union-type parameter, so a schema full of "nullable" escape hatches can hit the cap of 16. The docs also note that "each optional parameter roughly doubles a portion of the grammar's state space." A good portable pattern is a few explicit status or error enums instead of many nullable fields.

Self-Hosted Structured Generation

If you run open-weight models yourself, the same guarantees come from open-source engines. JSONSchemaBench tested several of them against 10,000 real-world schemas and the official JSON Schema Test Suite, on efficiency, coverage of constraint types, and output quality (Geng et al., 2025).

vLLM exposes five constraint types through its OpenAI-compatible server: a fixed choice, a regex, a JSON schema, a context-free grammar, and a "structural tag" that applies a JSON schema only inside marked sections of the text (vLLM docs). Other libraries in the same family include LMQL and JSONFormer.

Format Is Guaranteed, Correctness Is Not

A schema guarantees that a price field holds a number. It does not guarantee the number is right. OpenAI's guide says it plainly: "Structured Outputs can still contain mistakes," and "the model will always try to adhere to the provided schema, which can result in hallucinations if the input is completely unrelated to the schema" (OpenAI). A forced field is an invitation to fill it, which is why structured outputs can make hallucinations look more trustworthy, not less.

Even the format guarantee has documented edges. Anthropic lists three cases where output may not match the schema: a safety refusal (stop_reason: "refusal"), a response cut off at max_tokens, and string enum values that differ only in capitalization (Anthropic docs). OpenAI returns a separate refusal field and flags cut-off responses the same way. Treat a schema as a type checker, not a fact checker. Validate business rules in code, check the stop reason, and give the model a legal way to say "unknown." Content injected into the input can also fill a well-typed field with a hostile value, so pair schemas with prompt injection defenses and guardrails, and measure field accuracy with evals.

Structured Outputs vs Function Calling

What Structured Outputs Unlock

Data extraction. Pull typed fields from resumes, contracts, invoices, and support tickets with no regex and no parse failures. UI state generation. Generate form values, chart configs, and Kanban card states. The UI renders the schema and the model fills it. Validation loops. Downstream code checks semantic rules (prices above zero, dates in the future) and retries with the error as context. Agent memory serialization. Persist agent state as typed objects, which makes agent memory auditable and versionable. Composability. Typed outputs feed directly into the typed inputs of the next tool, which is what makes prompt chaining dependable.

Failure Modes

Over-constraining. If the schema does not permit "I don't know" or "error," the model will invent a value that fits. Always include an unknown enum value, a nullable path, or an error field. Enum drift is the slow version of the same problem: if a schema changes the allowed values of an enum, old records may no longer validate, so version schemas and migrate explicitly.

Schema complexity limits. Grammar size grows fast. Anthropic caps a request at 20 strict tools, 24 optional parameters, and 16 union-type parameters across all strict schemas, and it rejects unsupported features such as minimum, maxLength, and recursive schemas with a 400 error. A schema that passes every listed limit can still fail with "Schema is too complex for compilation" (Anthropic docs). OpenAI rejects allOf, not, and if/then/else, and requires the root to be an object rather than an anyOf (OpenAI). Keep schemas flat and mark fields required where you can.

Loss of reasoning. A schema that accepts only the final answer gives the model no room to think, and research found reasoning drops under strict format rules (Tam et al., 2024). For hard tasks, add a reasoning string field before the answer so the schema wraps chain-of-thought. OpenAI emits keys in schema order, so list that field first. Anthropic emits required properties first, so make that field required. With Anthropic extended thinking, the grammar applies only to the final output, not to the thinking block, so the model can think freely before it answers in the schema. See reasoning models for the wider trade-off.

Best Practices for Multi-Step Agent Chains

In a chain, one step's output is the next step's input, so a loose shape compounds downstream. Structure is the contract between steps.

1. One schema per step, small and flat. Split big extractions into several steps rather than one deep schema.
2. Pass typed fields, not transcripts. Hand the next step the fields it needs. This also slows context rot in long chains.
3. Use enums for anything you branch on. Compare them case-insensitively, per Anthropic's casing caveat.
4. Add confidence, unknown, or error fields so a step can decline instead of guessing.
5. Check the stop reason and validate values in code before any step with side effects runs.
6. Use strict tool calls for actions and schema outputs for answers. Keep the two roles separate.
7. Version schemas and log every step's typed output so a bad value can be traced to the step that produced it.

The sequence below shows where each check sits in one step of a chain.

Structured Outputs in Taskade

Taskade puts structured outputs into automations as a no-code step. The Ask Agent with Structured Output action sends input to one of your agents and asks for typed fields instead of prose. Click Add Output and pick a type: Text, Number, Boolean, Date, Time, Date and Time, URL, Email, Single Select, JSON Object, or JSON Array. Each field becomes a variable that later steps can read (how it works).

Those variables drive the rest of the flow. The table maps each output type to the step it usually feeds.

The same Add Output option works on Run Agent Command. Multi-agent work gets typed output too. The Ask Agent Team action sends a prompt to one of your AI Teams in Auto, Everyone, or Orchestrate mode, and its Processing Type setting turns the team's conversation into fields. Pick a preset such as Summary or Lead Qualification, or pick Custom and list the fields you want extracted. Text, Email, and URL fields can also flow out to any of the 100+ bidirectional integrations.

This is how structure connects the three pillars of Workspace DNA: Memory + Intelligence + Execution. An agent (Intelligence) returns typed fields, an automation (Execution) writes them back into projects (Memory), and Taskade Genesis apps that read those projects show the new values with no parsing layer. Agents can use built-in tools and any frontier model from top AI labs, and several agents can share one job in multi-agent teams. Build an automation or create an app to try it.

Related Concepts

Function Calling (the cousin pattern) · Tool Calling (typed arguments for tools) · Tool Use · Prompt Chaining (where typed handoffs matter most) · Hallucinations (why a valid schema can hold a wrong value) · Guardrails · Tokenizer · Temperature · Prompt Engineering · Large Language Models · AI Agents

Frequently Asked Questions About Structured Outputs

What are structured outputs in LLMs?

Structured outputs make an LLM emit a response that matches a predefined schema, such as JSON Schema, a Pydantic model, a TypeScript type, or a formal grammar. The inference engine compiles the schema into a grammar and masks invalid tokens at every decoding step, so the output parses. Documented exceptions are refusals and responses cut off at the token limit.

How is this different from asking for JSON in the prompt?

Asking for JSON trusts the model to comply. Structured outputs enforce compliance through constrained decoding: the model cannot pick a token that breaks the schema. That is a runtime guarantee, not a prompt suggestion.

What is the difference between JSON mode and structured outputs?

JSON mode guarantees valid JSON but not your keys or types. Structured outputs guarantee both valid JSON and a match to your schema. OpenAI's guide says: "We recommend always using Structured Outputs instead of JSON mode when possible."

Do structured outputs replace function calling?

No, they complement it. Function calling picks a tool and provides arguments. Structured outputs shape the model's own response. Modern APIs support both, often in the same request, and they share the same constrained-decoding machinery.

Are structured outputs always correct?

No. Format is guaranteed, correctness is not. OpenAI states that structured outputs "can still contain mistakes," and a forced field can pull a model into hallucinating a value. Validate values in code and give every schema a way to say "unknown."

Why does my schema work on OpenAI but fail on Anthropic?

The two APIs support different subsets of JSON Schema. OpenAI requires every field in required and supports minimum and pattern on base models. Anthropic allows optional fields but caps them at 24, caps union-type parameters at 16, and rejects minimum and maxLength. Check both rule sets before you port a schema.

What are best practices for structured outputs in multi-step agent chains?

Give each step one small, flat schema. Pass typed fields rather than full transcripts. Use enums for anything you branch on, and add unknown or error fields. Validate values and check the stop reason before side effects, use strict tool calls for actions, and version your schemas. See prompt chaining for the wider pattern.

Do Taskade agents use structured outputs?

Yes, in automations. The Ask Agent with Structured Output action and Run Agent Command both let you add typed output fields (Text, Number, Boolean, Date, Single Select, JSON Object, JSON Array, and more). The Ask Agent Team action turns a team conversation into fields through its Processing Type setting. Each field becomes a variable that Branch, Loop, and write-back steps can use without parsing prose.

What are the downsides of structured outputs?

The main risk is over-constraining: if the schema has no room for "I don't know," the model will invent a value that fits. Other costs are compile latency on a new schema, provider limits on schema complexity, and weaker reasoning when the schema leaves no space to think.

Further Reading

• Ask Agent with Structured Output (Learn Taskade)
• Ask Agent Team (Learn Taskade)
• What Are AI Agents?
• How Many Tools Should an AI Agent Have?