Getting Reliable JSON from LLMs: Structured Outputs

A demo that returns JSON is easy. A production system that returns valid JSON on every request, including the ugly ones, is not.

This guide covers the three approaches, the schema rules that prevent most failures, and the validation loop that catches the rest.

For the agent layer, see best AI model for agents. For the build side, see custom software development.

Three approaches

Approach	What it guarantees	Best for	Main failure
JSON mode	Valid JSON syntax	Simple extraction	Wrong shape or missing fields
Structured outputs	JSON matching your schema	APIs, pipelines, forms	Schema too strict or ambiguous
Function calling	A named tool with arguments	Actions and multi-step tools	Wrong tool or bad arguments

JSON mode is the weakest guarantee. If the output must match a contract, use structured outputs or function calling.

Schema design rules

1. Make required fields explicit. Do not rely on the model to infer what is optional.
2. Use enums for categories. Free text becomes inconsistent within a week.
3. Allow null where the data can be missing. Force the model to say “unknown” instead of inventing.
4. Keep nesting shallow. Deep schemas increase errors and make debugging harder.
5. Use short, clear field names. invoice_total, not totalAmountIncludingTaxesAndFees.
6. Put descriptions on ambiguous fields. The model reads them.
7. Avoid free-text fields for values you will process. Use dates, numbers and enums.

A schema that works

{
  "type": "object",
  "required": ["vendor", "invoice_number", "total", "currency", "line_items"],
  "properties": {
    "vendor": { "type": "string" },
    "invoice_number": { "type": ["string", "null"] },
    "total": { "type": "number" },
    "currency": { "type": "string", "enum": ["INR", "USD", "EUR"] },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["description", "amount"],
        "properties": {
          "description": { "type": "string" },
          "amount": { "type": "number" }
        }
      }
    }
  }
}

Note the nullable invoice number. Invoices in the real world are missing fields, and the schema should let the model say so.

The validation loop

Never trust the first response. Run three checks:

1. Parse. Is it valid JSON?
2. Validate. Does it match the schema?
3. Check business rules. Do the totals add up? Is the date in range? Does the currency match the vendor?

If any check fails, retry once with the validation error included in the prompt. Cap retries at two, then fail visibly.

result = model(prompt, schema)
for attempt in 1..2:
    errors = validate(result)
    if not errors:
        return result
    result = model(prompt + format_errors(errors), schema)
raise ValidationError(errors)

Silent retries are worse than a visible failure. Log the attempt, the error and the model.

Handling hallucinations

Structured output does not stop invented values. It only guarantees shape.

• Cross-check totals against line items.
• Verify identifiers such as GSTIN or invoice numbers against a known list.
• Require evidence for extracted fields, such as a page or bounding box.
• Set a confidence or “not found” option for every optional field.
• Sample and review a fixed number of outputs every week.

Temperature and determinism

Set temperature to 0 for extraction. It reduces variation but does not guarantee identical output. If you need idempotency, hash the input and cache the result.

Common failures

Symptom	Likely cause	Fix
Extra prose around JSON	JSON mode not enabled	Use structured outputs
Missing field	Field not marked required	Make it required
Wrong enum value	Enum not in schema	Add the value or use a string
Truncated output	Token limit too low	Raise max tokens
Invented value	Field optional but expected	Add null and a “not found” rule
Inconsistent dates	Free-text date field	Use ISO format with a pattern

When to use function calling instead

Use function calling when the model needs to do something, not just describe it: create an order, send an email, query a database.

The rules are similar: define a narrow tool, validate arguments, require approval for irreversible actions, and log every call.

A production checklist

• Schema versioned in code.
• Validation on every response.
• Two retries maximum.
• Business-rule checks separate from schema checks.
• Structured logs with model, prompt version and outcome.
• A sample review process.
• A fallback path when validation fails.

FAQ

What is the difference between JSON mode and structured outputs?

JSON mode guarantees valid JSON but not a specific shape. Structured outputs constrain the response to your schema.

Can structured outputs still hallucinate?

Yes. They guarantee the shape, not the truth. Validate values against your own data.

Which model is best for structured extraction?

Any current model with schema-constrained output works. Test on your documents, because accuracy varies by domain.

Should I retry on validation failure?

Once or twice, with the error included. Then fail visibly and log it. Unlimited retries hide a broken prompt.

Next step

If you are building an extraction or agent pipeline and want the schema and validation reviewed, contact 24Bit System. Our custom software development team builds these systems, and AI model routing covers the cost side.