Learn · Developer

Prompting for Programs: When the Reader Is Code

A human forgives a rambling answer. Your JSON parser will not.

Prompting in a chat window is a conversation; prompting through an API is a specification. The difference is the reader. A human skims past a model's throat-clearing and forgives a wandering answer. Your code, waiting to parse a response at three in the morning, forgives nothing. This guide covers what changes when the consumer of AI output is a program.

The system prompt is your contract

API calls separate the system prompt — standing instructions about role, rules, and format — from the user content being processed. Treat the system prompt the way you treat any interface contract:

State the job, the format, and the refusal path. What the model does, exactly what shape it answers in, and what it should output when the input is unusable — because 'garbage in' will happen, and a model with no instructed failure path improvises one.
Show, don't only tell. One or two worked examples of input and desired output outperform paragraphs of description. Examples are the closest thing prompting has to a type system.
Put rules before data. Keep instructions in the system prompt and the material being processed in the user message. Mixing them invites the model to treat instructions inside the data as commands — which is also the root of [prompt injection](/build/building-agents/), covered properly in the agents guide.
Version it like code. The system prompt determines behavior as much as any function does. It belongs in the repository, in review, and in [your test suite](/build/evals-and-testing/) — not pasted into a dashboard and forgotten.

Structured output: stop parsing prose

The single biggest reliability upgrade in API work: ask the platform, not the prose, for structure. Every major provider now supports some form of structured output or schema enforcement, where you define the fields you need and the platform constrains the model's response to match. Use it whenever output feeds code. The old folk techniques — 'respond ONLY with valid JSON', regexes fishing brackets out of markdown — were always a negotiation, and the schema features ended the negotiation.

Two design notes that pay off immediately. Give every extraction field an escape hatch — an explicit null or 'not found' option — because a model forced to fill a field it has no answer for will invent one; the schema should make honesty expressible. And keep one call to one job: a prompt asked to extract, evaluate, and summarize simultaneously does all three worse. Chains of small, testable calls beat one heroic prompt, the same way small functions beat a thousand-line one.

Creativity you did not order

Model output has a randomness dial — usually named temperature — and turning it to zero makes responses more repeatable. It does not make them deterministic, and this is the mental adjustment programming against a model requires: you are calling a function that is subtly different every time. The engineering response is the same as for any unreliable dependency, and it is liberating once accepted:

Validate at the boundary. Schema enforcement handles shape; your code still checks sense — dates that parse, categories from the allowed list, numbers in plausible ranges.
Design the retry. A failed validation can simply re-call the model, often with the error message included. One retry converts a surprising share of failures; more than two usually means the prompt, not the dice, is the problem.
Log the pairs. Keep inputs and outputs from production. That log is where you discover what your prompt actually does at scale — and it becomes the raw material for [evals](/build/evals-and-testing/), the subject two guides ahead.

The craft transfers, the standard rises

Everything from [prompting that works](/guides/prompting-that-works/) — context, examples, iteration — applies verbatim here. What changes is the bar: a chat prompt that works four times out of five is a good prompt; a production prompt that works four times out of five is an incident generator. The rest of this track is essentially about closing that gap: grounding the model in [your own data](/build/building-with-rag/), giving it [tools to act with](/build/building-agents/), and [proving it behaves](/build/evals-and-testing/) before your users do the proving for you.

Common questions

What is a system prompt?

Standing instructions sent separately from user content on every API call, defining the model's role, rules, output format, and failure behavior. In an application it functions as a behavioral contract, and it deserves the treatment code gets: version control, review, and tests.

How do I get reliable JSON from a language model?

Use the provider's structured-output or schema feature, which constrains the response to fields you define, rather than instructing the model to 'respond only with JSON' and parsing prose. Add escape-hatch values like null for missing information so the model is never forced to invent, and validate the result in code regardless.

Does setting temperature to zero make LLM output deterministic?

It makes output substantially more repeatable, not deterministic — responses can still vary between identical calls. Production designs treat the model like any unreliable dependency: validate outputs at the boundary, retry once on failure with the error included, and log input-output pairs to see real behavior.

Why do multi-task prompts perform worse?

A single call asked to extract, judge, and summarize at once tends to do each job worse and is far harder to test and debug. Chains of small calls with one job each are more reliable, independently testable, and cheaper to fix when one link misbehaves — the same reasoning that favors small functions.