How to Get Reliable JSON From an LLM With Structured Outputs

Free-text parsing is where LLM features quietly break in production. You ship a prompt that says "reply with JSON," it works in testing, then a customer input nudges the model into a markdown code fence or a trailing apology and your json.loads throws. The fix is constrained decoding: hand the model a JSON Schema and force every generated token to fit it, so the response parses every time. Every major provider now ships this, and turning it on takes one parameter.

What You Need Before You Start

• Python 3.9 or later.
• Either an API key for a hosted provider (OpenAI or Anthropic) or a running local server (ollama serve, or vllm serve with a model loaded).
• The matching SDK: pip install openai, pip install anthropic, or pip install ollama.
• Pydantic (pip install pydantic) to define your schema once and generate the JSON Schema from it.

How to Force a Schema-Valid JSON Response in Four Steps

This is the whole procedure. The example targets OpenAI, but the shape is identical everywhere: define a schema, attach it, parse, and guard the edges. The next section shows the exact swap for Claude, Ollama, and vLLM.

1. Define the output shape as a Pydantic model. Writing it once gives you both the schema and a typed object to validate against.

   from pydantic import BaseModel
   
   class Contact(BaseModel):
       name: str
       email: str
       plan_interest: str
       demo_requested: bool

2. Send the request with the schema attached and strict mode on. The SDK's parse helper accepts the Pydantic class directly and sets strict: true under the hood.

   from openai import OpenAI
   
   client = OpenAI()
   
   completion = client.chat.completions.parse(
       model="gpt-4o-2024-08-06",
       messages=[
           {"role": "system", "content": "Extract contact details from the email."},
           {"role": "user", "content": "John Smith ([email protected]) wants the Enterprise plan and a demo Tuesday at 2pm."},
       ],
       response_format=Contact,
   )

3. Read the parsed object. No json.loads, no regex, no code-fence stripping. You get a typed instance back.

   contact = completion.choices[0].message.parsed
   print(contact.email)           # [email protected]
   print(contact.demo_requested)  # True

4. Guard the two edges constrained decoding does not cover. A safety refusal returns no parsed object, and hitting the token ceiling truncates mid-object. Check both before you trust the result.

   choice = completion.choices[0]
   
   if choice.finish_reason == "length":
       raise RuntimeError("Response truncated, raise max_tokens")
   
   msg = choice.message
   if msg.refusal:
       print("Model refused:", msg.refusal)
   else:
       contact = msg.parsed

If you prefer the raw API over the parse helper, pass the schema explicitly. The rules that make strict mode work live here: set strict: true, mark additionalProperties: false, and list every property in required.

response_format={
    "type": "json_schema",
    "json_schema": {
        "name": "contact",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "email": {"type": "string"},
                "plan_interest": {"type": "string"},
                "demo_requested": {"type": "boolean"}
            },
            "required": ["name", "email", "plan_interest", "demo_requested"],
            "additionalProperties": False
        }
    }
}

How to Do the Same With Claude, Ollama, or vLLM

The concept ports cleanly. Only the parameter name and the client change, and in every case Contact.model_json_schema() gives you the schema from the same Pydantic class.

Claude

Anthropic exposes structured outputs through output_config.format. The response text comes back as valid JSON matching your schema. Mind the field-name detail: it is additionalProperties, not additional_properties, because JSON Schema keys stay camelCase even in Python.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Extract the contact details: John Smith ([email protected]) wants the Enterprise plan and a demo Tuesday."}],
    output_config={
        "format": {
            "type": "json_schema",
            "schema": Contact.model_json_schema(),
        }
    },
)
print(response.content[0].text)

When you want the structure enforced on a tool call rather than the final message, add strict: true to the tool definition instead. That path is covered in Anthropic's strict tool use guide, and it pairs well with the patterns in agentic orchestration with Anthropic APIs.

Ollama

For local models, Ollama takes the schema in a format field on the chat call. Set temperature to 0 for repeatable extraction, since local models drift more than hosted ones.

from ollama import chat

response = chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Extract the contact details from: John Smith, [email protected], Enterprise plan, demo Tuesday."}],
    format=Contact.model_json_schema(),
    options={"temperature": 0},
)
contact = Contact.model_validate_json(response.message.content)

If you are new to running models on your own hardware, start with running large language models locally with Ollama and come back to this step.

vLLM

vLLM speaks the OpenAI wire format, so you point the OpenAI client at your server and pass the same response_format object. Under the hood vLLM compiles the schema into a grammar with its structured-outputs backend (xgrammar by default).

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
model = client.models.list().data[0].id

completion = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Extract the contact details."}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "contact", "schema": Contact.model_json_schema()},
    },
)
print(completion.choices[0].message.content)

Serving details, including how to pick and tune that backend, live in serving an open LLM with vLLM.

Which Approach Fits Your Stack?

All four constrain the output to your schema. They differ in where they run, how the schema is passed, and how strict the enforcement is out of the box.

Common Errors and How to Fix Them

The API Rejects My Schema

Strict mode is unforgiving. If you hit an error about an unsupported schema, check three things: every object carries additionalProperties: false, every property appears in that object's required array, and you have not used a keyword the provider does not support (some minLength or pattern constraints are ignored or rejected). When a field is genuinely optional, model it as a union with null and keep it in required rather than dropping it from the list.

I Get Valid JSON but the Values Are Wrong

This is the trap teams fall into. Constrained decoding guarantees the shape, never the content. The model can still return an empty string for a name it could not find, or a plausible but invented email. Add semantic checks after parsing, and describe each field precisely in the schema. Pydantic field descriptions carry through to the JSON Schema and steer the model.

Output Is Truncated Mid-Object

A response cut off by the token limit is invalid JSON no matter how good the grammar. Watch for finish_reason == "length" (or the provider's equivalent) and raise max_tokens. Deeply nested arrays are the usual culprit, so cap list sizes in the schema when you can.

Local Models Ignore the Schema

Older or heavily quantized models sometimes fight the grammar and stall or repeat. Set temperature to 0, restate the instruction in the prompt ("return the answer as JSON"), and if it persists, move to a newer instruct-tuned model. Heavy quantization tends to make this worse, so try a less aggressive quant before you give up on the schema.

What to Do Next

• Replace your prompt-and-hope JSON calls with schema-enforced calls in one endpoint, then delete the retry-and-repair code it makes obsolete.
• Move shared schemas into a single Pydantic module so hosted and local backends stay in sync from one source of truth.
• Read the primary docs for the backend you ship on: OpenAI's Structured Outputs guide, Anthropic's structured outputs page, and vLLM's structured outputs reference.
• Once extraction is reliable, feed those typed objects into a larger flow. The reasoning in context engineering for long-horizon agents covers how structured state keeps multi-step runs on track.

Sources

• OpenAI: Structured Outputs
• Anthropic: Structured Outputs
• Anthropic: Strict Tool Use
• Ollama: Structured Outputs
• vLLM: Structured Outputs
• JSON Schema: Object Reference