How to Optimize Prompts Automatically With DSPy

Hand-tuning a prompt is guessing with extra steps. You reword the instruction, rerun a handful of examples, and hope the change holds on inputs you never saw. DSPy takes that loop away: you declare the task, define a metric, and a compiler searches the space of instructions and few-shot examples for you. The one-line version of how: write a signature, wrap it in a module, write a metric, and call an optimizer's compile() on a small training set.

Key Takeaways

The short answer: install DSPy, declare a signature and metric, then run MIPROv2(...).compile(program, trainset=...) to get an optimized program you can save and reload.
DSPy optimizes two things a human usually tunes by hand: the instruction text and the few-shot demonstrations attached to each predictor.
Pick the optimizer by data size. Under roughly 50 labeled examples, bootstrap few-shot demos; at a few hundred, reach for instruction search with MIPROv2.
Your metric is the objective the compiler maximizes, so a sloppy metric produces a sloppy program. Treat it as the most important line of code you write.
Always measure a baseline with dspy.Evaluate before and after compiling, or you cannot tell whether optimization actually helped.

What You Need Before You Start

The setup is light. You need Python 3.10 or newer, an API key for at least one model (or a local endpoint), and a labeled dataset, even a tiny one.

An installed DSPy: pip install -U dspy
A model DSPy can call, configured through dspy.LM. Any OpenAI-compatible or local endpoint works, including a local model served with Ollama.
Between 20 and a few hundred labeled examples. More helps the instruction optimizers, but you can start small.
A metric function that scores a prediction against the truth. If you plan to grade open-ended output, read how to keep an LLM judge from drifting first.

How Do You Compile a DSPy Program?

Follow these steps in order. By the end you will have an optimized program saved to disk, with a measured before-and-after score. Each block runs as written once your API key is set.

Configure a language model. This one call sets the default model for every predictor in your program.
```
import dspy

lm = dspy.LM("openai/gpt-4o-mini", api_key="YOUR_API_KEY")
dspy.configure(lm=lm)
```
Declare the task as a signature, not a prompt string. A signature names the inputs and outputs and describes the job in its docstring. DSPy writes the actual prompt.
```
class Classify(dspy.Signature):
    """Classify a support ticket by urgency."""
    ticket: str = dspy.InputField()
    urgency: str = dspy.OutputField(desc="one of: low, medium, high")
```
For quick experiments an inline signature works too: dspy.Predict("ticket -> urgency").
Wrap the signature in a module. ChainOfThought adds a reasoning step before the answer; Predict asks for the answer directly.
```
program = dspy.ChainOfThought(Classify)
```
Write a metric. It receives the gold example and the prediction and returns a score. Return an integer for exact match, or a float between 0 and 1 for partial credit.
```
def urgency_match(example, pred, trace=None):
    return int(example.urgency == pred.urgency)
```

Build a training set of dspy.Example objects. Call .with_inputs() so DSPy knows which fields are inputs versus labels.

trainset = [
    dspy.Example(ticket="Server is down for every user", urgency="high").with_inputs("ticket"),
    dspy.Example(ticket="Typo on the pricing page", urgency="low").with_inputs("ticket"),
    # ... add more labeled examples
]

Pick an optimizer and compile. Start with auto="light" to keep the call count and cost down while you confirm the pipeline works.

from dspy.teleprompt import MIPROv2

tp = MIPROv2(metric=urgency_match, auto="light")
optimized = tp.compile(program, trainset=trainset)

Save the compiled program and reload it later into the same module structure.

optimized.save("classifier.json")

loaded = dspy.ChainOfThought(Classify)
loaded.load("classifier.json")

Measure the gain. Run the same evaluator on the original and optimized programs against a held-out set.

from dspy.evaluate import Evaluate

evaluator = Evaluate(devset=devset, metric=urgency_match,
                     num_threads=8, display_progress=True)

evaluator(program)     # baseline
evaluator(optimized)   # after compile

Which Optimizer Should You Choose?

DSPy calls its optimizers teleprompters, and they differ in what they tune and how much data they want. The right choice tracks your training-set size more than anything else.

Optimizer	What It Tunes	Trainset Size	Use When
`LabeledFewShot`	Few-shot demos from your labels	Any	You want a fast baseline before real optimization.
`BootstrapFewShot`	Self-generated few-shot demos	~50 and up	The task is learnable from examples and you want cheap gains.
`BootstrapFewShotWithRandomSearch`	Demos across many candidate programs	~50-300	You want to explore several demo sets and keep the best.
`MIPROv2`	Instructions and demos jointly	200+	Wording matters and you have enough data to search instructions.
`BootstrapFinetune`	Model weights plus prompting	300+	You want to distill a compiled program into a smaller model.

A practical path: run BootstrapFewShot first because it is fast and cheap, then graduate to MIPROv2 once you have a metric you trust and a few hundred labeled rows. MIPROv2 proposes candidate instructions and uses Bayesian search to combine them with demo sets, which is where it earns its extra cost.

How the Metric Shapes the Result

The optimizer only ever sees the number your metric returns, so it will happily exploit a weak one. If exact string match is too brittle, score a normalized form or partial overlap. For subjective output, an LLM judge can be the metric itself.

class Judge(dspy.Signature):
    """Rate whether the answer is faithful to the context."""
    context: str = dspy.InputField()
    answer: str = dspy.InputField()
    faithful: bool = dspy.OutputField()

judge = dspy.ChainOfThought(Judge)

def faithfulness(example, pred, trace=None):
    return int(judge(context=example.context, answer=pred.answer).faithful)

When the output is a structured object rather than a label, pair DSPy with typed fields and read how to get reliable JSON from an LLM so your metric parses cleanly.

Troubleshooting Common Failures

Every bootstrapped example fails the metric. The optimizer cannot build demos from traces that score zero. Confirm your signature field names match what the metric reads, and loosen an over-strict exact match.
Rate limits or errors during compile. Optimization fans out many calls. Lower num_threads, and raise the optimizer's max_errors so a few transient failures do not abort the run.
The loaded program ignores its instructions. You loaded a saved file into a different module than you saved. Recreate the same signature and module type before calling .load().
Cost climbs faster than accuracy. Stay on auto="light" until the pipeline is proven, and use a cheap model as the task LM while reserving a stronger model only for instruction proposal.
Scores swing between runs. A small dev set makes evaluation noisy. Grow the held-out set before trusting a two-point difference.

What to Do Next

Once a single module compiles cleanly, extend the same pattern outward.

Compose several predictors into one dspy.Module and compile the whole pipeline at once. The official RAG tutorial walks through a multi-step program end to end.
Read the optimizer overview and the MIPROv2 API reference for every compile argument, including max_bootstrapped_demos and max_labeled_demos.
Wire your DSPy metric into a standing test suite so regressions surface early. If you already run evals with Promptfoo, keep the same held-out set across both.
Compare DSPy's search against reflective approaches such as GEPA to understand when each style of prompt evolution wins.