What Structured Outputs Actually Guarantee — And Three Ways You Get Nothing

Structured outputs are usually introduced as a guarantee: the model will return valid JSON matching the schema you supplied. That is accurate and it is narrower than it sounds. The guarantee covers shape, it covers a restricted dialect of JSON Schema, and there are two documented paths where it does not apply at all.

None of this is hidden — it is stated plainly in the provider documentation, and has been since the feature was introduced. It is just that most integrations are written against the happy path, and the two other paths are not errors. They are ordinary responses with a different shape.

It guarantees shape, never correctness

This is the misunderstanding with the largest blast radius. A schema constrains the structure of the output: which keys exist, what types their values take, which strings are drawn from an enum. It says nothing about whether the values are right.

A model asked to extract an invoice total can return a perfectly conforming object with the wrong number in it. Validation passes. Types check. The pipeline is green and the figure is fabricated. Teams routinely treat schema conformance as a quality gate, and it is a serialisation guarantee wearing the costume of one.

The practical consequence is that structured outputs remove one class of bug — malformed JSON, parse failures, hand-written extraction regexes — and remove none of the evaluation work. If anything they make evaluation more necessary, because the output now looks trustworthy.

The schema language is a subset, and the gaps have shape

The structured outputs guide is explicit that only a subset of JSON Schema is supported. Two of the omissions change how you model data rather than merely restricting what you can express.

Optionality becomes nullability

This is the constraint people meet first and misdiagnose most often. You cannot mark a field optional and let the model omit it. Every field is required, so the way to express "this may not be present" is to allow null explicitly.

// Wrong: the field is simply absent from `required`, which the API rejects.
const bad = {
  type: "object",
  properties: {
    invoiceNumber: { type: "string" },
    dueDate: { type: "string" },      // "optional"
  },
  required: ["invoiceNumber"],
  additionalProperties: false,
} as const;

// Right: every field required, absence expressed as an explicit null.
const good = {
  type: "object",
  properties: {
    invoiceNumber: { type: "string" },
    dueDate: { type: ["string", "null"] },
  },
  required: ["invoiceNumber", "dueDate"],
  additionalProperties: false,
} as const;

// The consequence is in the application type, not only the schema:
type Invoice = {
  invoiceNumber: string;
  dueDate: string | null;   // not `dueDate?: string`
};

The distinction matters downstream. An absent key and a null value are different things in most type systems and in most databases, and code written against an optional property will not compile against a nullable one. Decide early which your domain actually means, because retrofitting it across a codebase is tedious.

The two paths where the guarantee does not apply

Both are documented, both are ordinary, and both arrive as successful HTTP responses. Neither raises an exception in any SDK, which is why they tend to be discovered in production rather than in tests.

Refusal

If the model declines a request on safety grounds, the response carries a refusal field rather than content matching your schema. The documentation frames this as making refusals programmatically detectable, which is exactly right — it is a feature, and it means the check is a branch rather than a parse failure.

Retrying a refusal with the same input is the reflex and it is the wrong move. The model refused deterministically enough that a retry mostly reproduces it, and each attempt costs a full request. This is the specific pattern that turns a handled edge case into a runaway bill.

Truncation

If generation hits the output token limit, the response status becomes incomplete with incomplete_details.reason set to max_output_tokens, and what you have is a JSON fragment. It will not parse, and no amount of repair logic recovers the data that was never generated.

Unlike refusal, this one is worth retrying — but only with a change. Raising the limit, or reducing what you asked for, addresses the cause. Retrying identically produces an identical truncation at identical cost.

Handling all three explicitly

The shape that works is a discriminated union returned by the call site, so that callers are forced by the type system to consider each path. Collapsing this into an exception loses the distinction precisely where the recovery strategies differ.

type Extraction<T> =
  | { status: "ok"; data: T }
  | { status: "refused"; reason: string }
  | { status: "truncated"; partial: string };

async function extract<T>(
  request: ExtractRequest,
  validate: (value: unknown) => T,
): Promise<Extraction<T>> {
  const response = await client.responses.create(request);

  // Refusal: a normal response, not an error. Do not retry unchanged.
  const refusal = response.output?.[0]?.content?.[0]?.refusal;
  if (refusal) {
    return { status: "refused", reason: refusal };
  }

  // Truncation: the JSON is a fragment. Retry only with a larger budget
  // or a smaller ask — an identical retry truncates identically.
  if (response.status === "incomplete") {
    if (response.incomplete_details?.reason === "max_output_tokens") {
      return { status: "truncated", partial: response.output_text ?? "" };
    }
  }

  // Conforming: the shape is guaranteed, the values are not.
  // `validate` is where domain rules the schema cannot express are applied.
  return { status: "ok", data: validate(JSON.parse(response.output_text)) };
}

The `validate` callback is doing real work rather than restating the schema. It is where constraints the schema language cannot carry are enforced — that a date falls in a plausible range, that a total equals the sum of its line items, that an identifier matches a pattern. Those checks catch the conforming-but-wrong case that the guarantee explicitly does not cover.

Common mistakes

1. Treating schema conformance as a correctness check. It constrains structure only; a validating response can contain entirely fabricated values.
2. Retrying refusals unchanged. Refusal is largely deterministic for a given input, so the retry reproduces it and bills you again for the privilege.
3. Retrying truncation unchanged. Same output budget, same truncation point. Raise the limit or reduce the request.
4. Modelling optional fields by leaving them out of required. The API rejects it. Express absence as an explicit null and propagate that through your types.
5. Assuming a JSON Schema generated by a library will be accepted. Most emit allOf or conditional constructs that are unsupported, and generators targeting large models can exceed the property and character limits.
6. Wrapping the whole call in one try/catch. Refusal and truncation arrive as successful responses; a catch block never sees them.

Security and performance considerations

A closed schema is a genuine security control and an underused one. Because additionalProperties is false and the field set is fixed, a model cannot introduce unexpected keys into a structure your application will consume — which removes an injection path when the output feeds something that iterates over keys, builds a query, or populates a template.

That protection stops at the values. A string field will faithfully contain whatever the model produced, including content drawn from untrusted input the model was asked to summarise. Structured output is not sanitisation, and a conforming string is still attacker-influenced text if any part of the prompt was.

On cost, the all-fields-required rule means the model emits every field on every call, including the nulls. Wide schemas therefore cost output tokens proportional to their width rather than to the information actually present, and output tokens are the expensive half on every major provider. Combined with unchanged retries against refusals, this is exactly the compounding described in why agent bills explode — individually reasonable calls, an unreasonable total.

Troubleshooting

1. 200 OK with nothing parsed — check that strict mode is actually enabled. Without it the request succeeds and the conformance guarantee simply does not apply.
2. API rejects the schema — look for allOf, if/then/else, or an optional field missing from required. Library-generated schemas produce all three routinely.
3. Intermittent parse failures under load — almost always truncation. Check status and incomplete_details.reason before blaming the model.
4. Everything null — the model is conforming while finding nothing. That points at the prompt or the input, not the schema.
5. Works on short documents, fails on long ones — output budget rather than input. The extracted structure grew past max output tokens.
6. Schema accepted in testing, rejected after adding categories — the 1,000 enum value or 120,000 character budget, which is shared across the whole schema rather than per field.

The useful mental model is that structured outputs move a whole class of problem from parsing to semantics. You stop writing defensive extraction code and start needing an answer to a harder question: is this well-formed object actually right? That question does not have a library, which is why the evaluation work matters more once the parsing problem disappears.

If you are putting model output into a system where a wrong value has consequences — billing, records, downstream automation — that boundary is worth designing rather than discovering. Our AI engineering work covers it, and the integration side, where model output meets an existing system, sits with our API engineering practice.