Prompts in Applications: the API

From a chat window to an API request

When you type into an AI assistant, the app quietly does a lot of work: it adds its own system prompt, keeps your earlier messages, picks settings and shows the answer. When you build AI into your own software, your code does that work. It sends an HTTP request to the provider's API and receives JSON back. The basics — the request shape, the API key, the SDK and the difference between a chat app and the API — are on Working with LLMs; this page goes into what changes for the prompt itself.

Three things are different from chatting:

• The prompt is split into fields. Standing instructions go in a system or developer field, the conversation in a list of messages, tools in their own list. Where a piece of text goes changes how much weight the model gives it.
• Nothing is remembered. The API is stateless (see stateless models): every request must contain everything the model needs, including the earlier turns of the conversation.
• The same prompt runs thousands of times. A wording that works once in a chat must now work for every ticket, every customer and every odd input. That is why application prompts need a test set (see Iterating and Testing Prompts) and code that checks every answer.

Messages and roles

Every chat-style API takes the conversation as a list of messages, each with a role:

• system or developer: the application's standing instructions — the system prompt. OpenAI calls this the developer role and describes it as instructions from the application developer that are "prioritized ahead of user messages" (see Roles and System Prompts).
• user: what the person typed, plus anything your code adds on their behalf, such as a ticket text or retrieved documents.
• assistant: the model's earlier answers. You send them back so the model sees the conversation so far.

Roles set priority, not security. A higher-priority instruction usually wins over a lower one, but a model can still be talked out of it, so never rely on the system prompt to keep a secret or enforce a rule (see Prompt Security and Responsible Use).

Three API styles you will meet

As of October 2026, these are the request styles you are most likely to see in code and documentation:

• The Messages API is Anthropic's API for Claude models (POST /v1/messages). The system prompt is a top-level system field, the conversation goes in messages with user and assistant turns, and max_tokens is required.
• The Responses API is OpenAI's current API, recommended for all new projects. Standing instructions go in a top-level instructions field (or a developer message), the conversation in input, and the SDK offers the generated text as output_text. OpenAI's older Assistants API was shut down on 26 August 2026.
• The Chat Completions API is OpenAI's earlier API, in which the whole conversation, including the system or developer message, is one list of messages. OpenAI still supports it, and many other providers and local model servers copy its request shape, so a lot of example code you find uses it.

Google's Gemini API and the cloud platforms (Microsoft Foundry, Amazon Bedrock, Google Cloud) have their own request formats; the ideas on this page apply to them all, but check field names in the current documentation, as they change.

The example builds the same conversation in two of these shapes, without sending anything. The system prompt, the history and the new question are kept apart in the code and only combined at the last moment, which makes it easy to change one part, test it, and switch provider.

import json

# The parts of one prompt, kept separate in the application
SYSTEM = ("You are a service desk assistant at Kestrel Energy. "
          "Answer in at most 60 words.")
HISTORY = [
    ("user", "The customer portal shows error 503 when I log in."),
    ("assistant", "Is it every page, or only the login page?"),
]
QUESTION = "Only the login page. Other pages load fine."
MODEL = "MODEL_ID_FROM_CONFIG"  # filled from configuration, never hard-coded


def messages_request(system, history, question, max_tokens):
    """Request body in the shape of Anthropic's Messages API."""
    messages = [{"role": role, "content": text} for role, text in history]
    messages.append({"role": "user", "content": question})
    return {"model": MODEL, "max_tokens": max_tokens,
            "system": system, "messages": messages}


def responses_request(system, history, question, max_tokens):
    """Request body in the shape of OpenAI's Responses API."""
    items = [{"role": role, "content": text} for role, text in history]
    items.append({"role": "user", "content": question})
    return {"model": MODEL, "max_output_tokens": max_tokens,
            "instructions": system, "input": items}


for name, build in [("Messages API", messages_request),
                    ("Responses API", responses_request)]:
    body = build(SYSTEM, HISTORY, QUESTION, 2000)
    print(f"--- {name}: top-level fields ---")
    print(", ".join(body))
    turns = body.get("messages") or body.get("input")
    print(f"turns sent: {len(turns)}, last role: {turns[-1]['role']}")

print()
print(json.dumps(responses_request(SYSTEM, HISTORY, QUESTION, 2000),
                 indent=2))

--- Messages API: top-level fields ---
model, max_tokens, system, messages
turns sent: 3, last role: user
--- Responses API: top-level fields ---
model, max_output_tokens, instructions, input
turns sent: 3, last role: user

{
  "model": "MODEL_ID_FROM_CONFIG",
  "max_output_tokens": 2000,
  "instructions": "You are a service desk assistant at Kestrel Energy. Answer in at most 60 words.",
  "input": [
    {
      "role": "user",
      "content": "The customer portal shows error 503 when I log in."
    },
    {
      "role": "assistant",
      "content": "Is it every page, or only the login page?"
    },
    {
      "role": "user",
      "content": "Only the login page. Other pages load fine."
    }
  ]
}

Two things to notice. All three turns are sent, because the model has no memory of the first two. And the last message is always from the user: that is the turn the model answers.

Sending it is one call in the provider's SDK. The Anthropic version is on Working with LLMs; this is the Responses API with OpenAI's Python SDK, as documented in October 2026 (not run here: it needs a key and the network):

import os

from openai import OpenAI  # the vendor's SDK: pip install openai

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])  # key from the environment

response = client.responses.create(
    model=os.environ["MODEL_ID"],  # a pinned model id from your configuration
    instructions="You are a service desk assistant at Kestrel Energy. "
    "Answer in at most 60 words.",
    input=[
        {"role": "user", "content": "The customer portal shows error 503 when I log in."},
        {"role": "assistant", "content": "Is it every page, or only the login page?"},
        {"role": "user", "content": "Only the login page. Other pages load fine."},
    ],
    max_output_tokens=2000,
)

if response.status == "incomplete":
    print("Stopped early:", response.incomplete_details.reason)
else:
    print(response.output_text)

Parameters that shape the answer

Maximum output tokens

Maximum output tokens is the request setting that caps how many tokens the model may generate for one response; when the cap is reached, generation simply stops, mid-sentence if need be. It is called max_tokens in the Messages API and max_output_tokens in the Responses API (as of October 2026). It is a cost and safety limit, not a length instruction: the model does not plan its answer to fit, so ask for the length you want in the prompt ("at most 60 words") and set the limit comfortably above it.

On models that reason before answering, the thinking tokens count toward the same limit and are billed as output. Anthropic describes max_tokens as a hard limit on thinking plus response text; OpenAI warns that you can pay for input and reasoning tokens without receiving any visible answer when max_output_tokens is too low. A limit that worked for an older, non-thinking model can therefore produce cut-off or empty answers on a newer one.

Temperature and other sampling settings

Temperature and the related top_p setting control how much randomness goes into choosing each next token (see How a Model Reads Your Prompt). Older tutorials tell you to set temperature to 0 for repeatable answers. That advice has aged badly:

The practical lesson: control the answer with the prompt, a schema and checks in code, not with sampling settings, and never build a process on the assumption that the same request returns the same text.

Reasoning effort and model choice

Newer models offer a setting for how much they think before answering instead: output_config.effort in the Messages API, reasoning.effort in the Responses API, and a thinking level in the Gemini API, each with values from low to high and beyond (as of October 2026). Lower effort is faster and cheaper for simple classification; higher effort helps on hard problems. The details and the vendor differences are on Reasoning: Step by Step and Thinking Models.

The model field matters as much as any setting. Read the model id from configuration, pin it to a specific version, and re-run your tests before you switch, because a new model can follow the same prompt differently (see Prompt Templates and Reuse).

Reading the response

A response is more than text. In the Messages API it contains a list of content blocks — on current models usually a thinking block first, then text, and possibly a tool request — plus token usage and the stop reason. Code that prints "the first block" breaks as soon as a thinking block comes first, so always select blocks by their type.

Stop reasons

A stop reason is the field in an API response that tells you why the model stopped generating: because it finished, hit the token limit, wants to call a tool, or declined to answer. As of October 2026 the Messages API documents these values for stop_reason:

The Responses API reports the same situations differently: a response that ran out of tokens has status "incomplete" with incomplete_details.reason set to "max_output_tokens". Either way, a refusal or a cut-off answer comes back as an ordinary HTTP 200, not as an error, so code that only checks the HTTP status will happily pass half an answer on to a customer. The example handles three recorded responses:

import json

# Three recorded responses in the shape of the Messages API (shortened).
RECORDED = [
    """{"stop_reason": "end_turn",
        "content": [{"type": "thinking", "thinking": "..."},
                    {"type": "text", "text": "Clear the browser cache and retry."}],
        "usage": {"input_tokens": 412, "output_tokens": 236}}""",
    """{"stop_reason": "max_tokens",
        "content": [{"type": "thinking", "thinking": "..."},
                    {"type": "text", "text": "The 503 on the login page suggests"}],
        "usage": {"input_tokens": 412, "output_tokens": 300}}""",
    """{"stop_reason": "refusal",
        "content": [],
        "usage": {"input_tokens": 398, "output_tokens": 4}}""",
]


def handle(raw):
    response = json.loads(raw)
    reason = response["stop_reason"]
    text = "".join(block["text"] for block in response["content"]
                   if block["type"] == "text")
    usage = response["usage"]
    tokens = f"{usage['input_tokens']:>4} in /{usage['output_tokens']:>4} out"
    if reason == "end_turn":
        return f"OK       {tokens}  {text!r}"
    if reason == "max_tokens":
        return f"CUT OFF  {tokens}  partial text kept for the log only"
    if reason == "refusal":
        return f"REFUSED  {tokens}  show a fallback message, route to a person"
    return f"CHECK    {tokens}  unexpected stop reason {reason!r}"


for raw in RECORDED:
    print(handle(raw))

OK        412 in / 236 out  'Clear the browser cache and retry.'
CUT OFF   412 in / 300 out  partial text kept for the log only
REFUSED   398 in /   4 out  show a fallback message, route to a person

The second response looks like the start of a useful answer, which is exactly the danger. The last line of handle matters too: providers add new stop reasons over time, and an unknown one should be noticed, not silently treated as success. The usage numbers are worth logging for every call; they are what you pay for.

Tools in a request

Tool calling (also called function calling) lets the model ask your code to run a function, such as looking up a service status. You send tool definitions with the request; each has a name, a description and a JSON Schema for its parameters. In the Messages API (as of October 2026):

{
  "name": "get_service_status",
  "description": "Returns the current monitoring status of one Kestrel Energy service: operational, degraded or down. Use it when a user reports that a service is slow or unavailable, before suggesting any fix. It only reads status; it cannot restart anything. Service names are lowercase with hyphens, such as portal-login or meter-upload.",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "service": {
        "type": "string",
        "enum": ["portal-login", "portal-web", "meter-upload", "vpn"]
      }
    },
    "required": ["service"],
    "additionalProperties": false
  }
}

The Responses API uses the same ideas with slightly different field names: "type": "function", and parameters instead of input_schema.

Tool descriptions are prompts

The model decides which tool to use, and with which arguments, only from the names, descriptions and schemas you send. Anthropic calls detailed descriptions "by far the most important factor in tool performance" and suggests at least three to four sentences per tool: what it does, when to use it (and when not), what each parameter means, and its limits. OpenAI's function calling guide offers a similar test: could a new intern use the function correctly from the description alone? Both recommend:

• Enums for fixed choices, as in the example, so the model cannot invent a service name.
• Strict mode ("strict": true), so the arguments are guaranteed to match the schema (see Structured Output).
• Few, clearly separated tools. OpenAI suggests staying under about 20 functions as a soft limit; two tools that overlap confuse the model as much as they would confuse a person.

Tool choice

Tool choice is the request setting that tells the model whether it may, must or must not call a tool. The default, auto, lets the model decide; none forbids tools. Forcing a call — any tool, or one specific tool — is possible in some APIs (OpenAI's required or a named function), but as of October 2026 Anthropic's newest models (Claude Opus 5.5, Sonnet 5.5 and Fable 5.1) reject the forcing options with an error. Anthropic's advice there is to use auto with strict tool use, or structured outputs when you simply want data in a fixed shape. OpenAI also offers parallel_tool_calls: false when you want at most one tool call per turn.

Sending a tool result back

When the stop reason is tool_use, the model is waiting. Your code checks the request against an allow-list, runs the function, and sends a new request containing the full history: the assistant turn with the tool request, unchanged, followed by a user turn with the result, linked by the request's id.

import json

# Your own function, exposed to the model as a tool
STATUS = {"portal-login": "degraded", "portal-web": "operational"}


def get_service_status(service):
    return {"service": service, "status": STATUS.get(service, "unknown")}


TOOLS = {"get_service_status": get_service_status}  # the allow-list

# The conversation so far, and a recorded assistant turn that asks for a tool
messages = [{"role": "user", "content": "Is the portal login down right now?"}]
recorded_turn = {
    "role": "assistant",
    "content": [
        {"type": "text", "text": "Let me check the monitoring status."},
        {"type": "tool_use", "id": "toolu_01", "name": "get_service_status",
         "input": {"service": "portal-login"}},
    ],
}
messages.append(recorded_turn)  # the assistant turn goes back unchanged

results = []
for block in recorded_turn["content"]:
    if block["type"] != "tool_use":
        continue
    if block["name"] in TOOLS:
        output = TOOLS[block["name"]](**block["input"])
    else:
        output = {"error": f"unknown tool {block['name']}"}
    results.append({"type": "tool_result", "tool_use_id": block["id"],
                    "content": json.dumps(output)})

messages.append({"role": "user", "content": results})

for message in messages:
    content = message["content"]
    kinds = content if isinstance(content, str) else \
        ", ".join(part["type"] for part in content)
    print(f"{message['role']:<9} {kinds}")
print()
print(messages[-1]["content"][0]["content"])

user      Is the portal login down right now?
assistant text, tool_use
user      tool_result

{"service": "portal-login", "status": "degraded"}

The tool result becomes part of the prompt for the next turn. Keep it short and relevant — a status, not a 500-line monitoring dump — and remember that text coming back from a tool is data from outside, which can contain instructions an attacker planted (see indirect prompt injection). Running the loop until the model gives a final answer is the agent loop.

Structured outputs

When software reads the answer, ask for JSON and let the API enforce the shape. How to design the schema and check the result is on Structured Output; in the request it looks like this (as of October 2026):

• Messages API: an output_config field with a format of type json_schema and your schema.
• Responses API: a text field with a format of type json_schema, "strict": true and your schema. The older JSON mode guarantees valid JSON but not your schema.

"output_config": {
  "format": {
    "type": "json_schema",
    "schema": {
      "type": "object",
      "properties": {
        "category": {"type": "string", "enum": ["Network", "Account", "Portal", "Billing"]},
        "summary": {"type": "string"}
      },
      "required": ["category", "summary"],
      "additionalProperties": false
    }
  }
}

Enforced structure still needs checking in code. A refusal or a max_tokens stop can break the shape, and Anthropic notes that it does not guarantee the capitalization of enum values. A schema guarantees the shape of the answer, never that the content is true.

Prompt caching and cost

Application prompts are mostly the same from call to call: the same system prompt, the same tool definitions, the same examples, with only the ticket text changing. Prompt caching is a provider feature that stores the processed form of a prompt's unchanged beginning (the prefix) for a short time, so that later requests starting with exactly the same text are cheaper and faster. Only an identical prefix counts: one changed character early in the prompt and everything after it has to be processed again.

That turns into a simple rule for building prompts: stable parts first, variable parts last. OpenAI's prompt engineering guide says exactly that. Anthropic's API builds the prefix in the order tools, then system prompt, then messages, and lets you mark where the cacheable part ends with a cache_control field; by default an entry lives five minutes (as of October 2026). The example measures how much two prompts for different tickets share at the start:

import os

INSTRUCTIONS = "Classify the ticket as Network, Account, Portal or Billing. " * 20
EXAMPLES = "Example: 'VPN drops every hour' -> Network\n" * 30
TICKETS = ["INC-40311: field engineer locked out after password change",
           "INC-40312: portal shows error 503 on the login page"]


def stable_first(ticket):
    return INSTRUCTIONS + EXAMPLES + "Ticket: " + ticket


def ticket_first(ticket):
    return "Ticket: " + ticket + "\n" + INSTRUCTIONS + EXAMPLES


for build in (stable_first, ticket_first):
    first, second = (build(ticket) for ticket in TICKETS)
    shared = len(os.path.commonprefix([first, second]))
    print(f"{build.__name__:<13} prompt {len(first)} chars, "
          f"identical start {shared} chars ({shared / len(first):.0%})")

stable_first  prompt 2556 chars, identical start 2506 chars (98%)
ticket_first  prompt 2557 chars, identical start 16 chars (1%)

The repeated strings stand in for a long system prompt and a set of examples. Real caching works on tokens and has a minimum length, so this is an illustration of the principle, not of a provider's exact rules. With the ticket first, the two prompts share only Ticket: INC-4031, and nothing can be reused. For work that does not need an answer right away, such as classifying a night's tickets, batch processing is another way to save: Anthropic's batch API, for example, costs half the normal price (as of October 2026). How tokens turn into money is on Working with LLMs.

What does this do?

{"model": "example-model", "max_tokens": 300, "system": "You are a service desk assistant at Kestrel Energy. Answer in at most 60 words.", "messages": [{"role": "user", "content": "Portal login shows error 503."}, {"role": "assistant", "content": "Only on the login page?"}, {"role": "user", "content": "Yes, other pages work."}]}

Below is the JSON body of a request to a chat-style language model API (a fictional example). For each top-level field, say in one sentence what it does. Then answer: (1) Which text is the system prompt? (2) Which message will the model answer? (3) Does max_tokens tell the model how long its answer should be, or does it only cut the output off? (4) Why are the first two messages sent again? Answer in a numbered list. If you are not sure how a specific provider treats a field, say so instead of guessing.

{"model": "example-model", "max_tokens": 300, "system": "You are a service desk assistant at Kestrel Energy. Answer in at most 60 words.", "messages": [{"role": "user", "content": "Portal login shows error 503."}, {"role": "assistant", "content": "Only on the login page?"}, {"role": "user", "content": "Yes, other pages work."}]}

Before an AI call goes to production

• Keys come from an environment variable or a secrets manager, one key per application, with a spending limit.
• The model id is pinned and read from configuration; a model change is tested like a code change.
• Prompts live in version control, separate from the code that fills them (see Prompt Templates and Reuse), with a test set that runs on every change.
• Every response is checked: stop reason first, then the shape (schema), then the content where possible.
• Failures are expected. Set timeouts, retry rate-limit (HTTP 429) and server errors with increasing waits (see Web and API security), and have a fallback that does not need the model.
• Output is untrusted input. Encode it before showing it in a web page, never pass it straight to a shell or a database, and let a person approve actions that are hard to undo (see defenses in layers).
• Logging respects privacy. Log prompts, responses and token usage for debugging and cost, but treat those logs as containing whatever personal data went into the prompt.