Working with LLMs

What happens when you send a prompt

A large language model (LLM) is a function from text to text. The text you send is split into tokens, word pieces of roughly three quarters of an English word on average (the ratio depends on the model's tokenizer; Dutch and most other languages need more tokens per word). The model then predicts one next token at a time, appends it, and predicts again, until it produces a stop token or reaches the maximum length you allowed. Everything you receive is generated this way, including code, JSON and "I don't know".

Two consequences follow directly. First, input and output are measured, limited and billed in tokens. Second, the model has no access to anything that is not in the request: not your ticketing system, not yesterday's conversation, not today's date, unless the application puts it in the text (or gives the model a tool to look it up, see AI Agents and Tool Use).

Stateless: the history is sent again

The model itself is stateless: each request stands alone. A chat application creates the illusion of memory by keeping the conversation and sending all of it again with every new question. The third turn of a conversation is really one request like this (the format is the common "list of messages" shape most chat APIs use):

{
  "system": "You are a service desk assistant at Kestrel Energy. Answer briefly.",
  "messages": [
    { "role": "user", "content": "My VPN gives error 809." },
    { "role": "assistant", "content": "Error 809 usually means the VPN traffic is blocked. Are you on a hotel or guest network?" },
    { "role": "user", "content": "Yes, hotel Wi-Fi. What now?" }
  ]
}

This explains several everyday observations. Long chats get slower and more expensive, because every turn resends everything before it. Very long chats eventually hit the model's context window, and the application has to drop or summarize old turns, which is when the model "forgets" what you said at the start. And starting a fresh chat is a real reset: nothing carries over unless the application adds a memory feature that stores notes and inserts them into later prompts.

Prompts and prompt engineering

A prompt is the input text you give a language model: the instructions, the context and the question or data it should work on. Because the model continues from what it is given, the prompt is your main way of steering it. Prompt engineering is the practice of designing and testing prompts so that a model reliably produces the output you need. The word "engineering" is a bit grand: most of it is the same clear briefing you would give a new colleague, followed by testing on real cases.

The levers that matter most:

• Be specific about the task. "Summarize" leaves length, audience and focus to chance; "Summarize in three bullets for the on-call engineer" does not.
• Give context. The model does not know your company, your systems or your customers. Paste the relevant facts, or retrieve them automatically (see Embeddings and Retrieval).
• Give a role. "You are a service desk analyst at an energy company" sets vocabulary, tone and what counts as a sensible answer.
• Show examples of good input and output (see few-shot prompting).
• Specify the format: length, structure, language, JSON fields.
• State the constraints: what it must not do, and what to do when it does not know ("If the ticket does not say which laptop, ask for the asset tag; do not guess").

Before and after: a service desk example

A service desk wants a draft reply to a customer ticket. The same ticket, with a weak and a better prompt:

The "before" prompt often produces a plausible but generic reply, sometimes with invented steps. The "after" prompt narrows what the model can reasonably write, and gives it the facts so it does not have to fill gaps with guesses (the cause of most hallucinations in this kind of task). Prompt engineering is empirical: keep a small set of real test tickets, run every prompt change against all of them, and compare. A change that fixes one ticket can break another.

System prompts and user prompts

Most chat APIs separate two kinds of input. A system prompt holds instructions from the application builder that apply to the whole conversation: the role, the rules, the format, background facts. The user prompt (the user messages) holds what the person types in each turn. In the JSON above, the "system" field is the system prompt and the "user" messages are user prompts.

The split is useful because the builder writes the system prompt once and users never see or edit it, and models are trained to give it more weight than user messages. It is not a security boundary, though. A user can type "ignore your instructions" and sometimes succeed, and text in documents the model reads can contain hidden instructions too. That is prompt injection, covered in AI Security. Two practical rules follow: never put secrets (passwords, API keys, internal URLs you would not publish) in a system prompt, and never rely on the system prompt alone to stop a model from doing something harmful; enforce that in code and permissions.

Few-shot prompting

Asking a model to do a task with only instructions is called zero-shot. Few-shot prompting adds a handful of worked examples of input and desired output to the prompt, so the model can copy the pattern. The term comes from machine-learning research; the 2020 GPT-3 paper by Brown et al., "Language Models are Few-Shot Learners", showed that large models pick up a task from a few examples in the prompt without any retraining.

Classify each ticket as network, account or hardware.

Ticket: Outlook keeps asking for my password since the reset.
Category: account

Ticket: The docking station does not charge my laptop any more.
Category: hardware

Ticket: Teams calls drop every few minutes on the office Wi-Fi.
Category: network

Ticket: VPN error 809 from the hotel.
Category:

Examples communicate things that are hard to describe: the exact label names, the tone of a reply, where the line between two categories lies. Choose them carefully. The model imitates them closely, including their mistakes and their quirks; if all examples are short, the answers will be short. Cover the tricky cases, vary the examples, and keep them realistic. Examples also cost tokens on every call, as the cost calculation below shows. Nothing is learned permanently: the model's weights do not change, and the next request without the examples gets zero-shot behavior again. Changing the weights is fine-tuning, a different and much heavier tool.

Temperature

At each step the model computes a score for every token in its vocabulary. A function called softmax turns those scores into probabilities that add up to 100%, and the application then picks (samples) the next token according to those probabilities. Temperature is a setting that divides the scores before softmax: below 1 it sharpens the distribution so the most likely token wins more often, above 1 it flattens it so less likely tokens get a real chance.

The example below takes four candidate next words after "The VPN connection keeps ..." with made-up scores, and shows the probabilities at three temperatures. Real models do exactly this, but over a vocabulary of tens of thousands of tokens and on a GPU.

import math

# Scores (logits) a model might give the candidate next words after
# "The VPN connection keeps ..."
logits = {"dropping": 3.2, "failing": 2.9, "timing out": 2.1, "working": 0.4}


def softmax(scores, temperature):
    scaled = {word: s / temperature for word, s in scores.items()}
    top = max(scaled.values())  # subtract the maximum to avoid overflow
    exps = {word: math.exp(v - top) for word, v in scaled.items()}
    total = sum(exps.values())
    return {word: e / total for word, e in exps.items()}


for t in (0.2, 1.0, 1.5):
    print(f"temperature {t}")
    for word, p in softmax(logits, t).items():
        print(f"  {word:<11} {p:6.1%}")

temperature 0.2
  dropping     81.5%
  failing      18.2%
  timing out    0.3%
  working       0.0%
temperature 1.0
  dropping     46.8%
  failing      34.7%
  timing out   15.6%
  working       2.8%
temperature 1.5
  dropping     40.8%
  failing      33.4%
  timing out   19.6%
  working       6.3%

At 0.2 the top word wins four times out of five and "working" is practically impossible; at 1.5 every word is in play. Because a long answer is hundreds of such choices in a row, a low temperature gives answers that are similar from run to run, and a high one gives answers that wander.

• Low (0 to about 0.3): classification, extraction, code, anything where you want the same answer for the same input.
• Medium (about 0.7 to 1): general chat and writing; many providers use a value around 1 as the default.
• High (above 1) (not every provider allows values above 1): brainstorming, alternative phrasings; quality drops quickly if you go too far.

Tokens and cost

API providers charge per token, usually quoted per million tokens, with separate prices for input and output tokens. Output tokens typically cost several times more than input tokens, because generating them one by one is the expensive part. Reasoning models add hidden "thinking" tokens that are billed as output. Prices differ widely between models and fall over time, so always check the provider's current price list.

The example estimates the cost of letting a model classify 1,000 tickets, once with instructions only and once with five worked examples added. The prices are made up for illustration; only the arithmetic matters.

# ILLUSTRATIVE prices, made up for this example. Real prices differ per
# model and provider and change often: check the provider's price list.
PRICE_IN = 2.00    # USD per 1 million input tokens
PRICE_OUT = 10.00  # USD per 1 million output tokens


def estimate_tokens(text):
    """Rule of thumb for English text: about 4 characters per token."""
    return round(len(text) / 4)


ticket = ("Since this morning I cannot connect to the VPN from home. "
          "I get error 809 after entering my password. Laptop KE-LT-0412.")
ticket_tokens = estimate_tokens(ticket)
answer_tokens = 120   # a short JSON answer
tickets = 1_000

# The same instructions, without and with five worked examples (few-shot).
setups = {"instructions only": 300, "plus 5 examples": 300 + 5 * 150}

print(f"Tokens in one ticket: about {ticket_tokens}")
for name, system_tokens in setups.items():
    tokens_in = tickets * (system_tokens + ticket_tokens)
    tokens_out = tickets * answer_tokens
    cost = tokens_in / 1e6 * PRICE_IN + tokens_out / 1e6 * PRICE_OUT
    print(f"{name:<18} in {tokens_in:>9,}  out {tokens_out:>7,}  "
          f"cost ${cost:.2f}")

Tokens in one ticket: about 30
instructions only  in   330,000  out 120,000  cost $1.86
plus 5 examples    in 1,080,000  out 120,000  cost $3.36

Three lessons hide in these numbers. The ticket itself is the smallest part: the instructions and examples are sent with every ticket and make up most of the input. Adding examples more than tripled the input tokens, which is worth it only if it measurably improves the answers. And the 120 output tokens per ticket cost more than all the input in the first setup, so asking for a short answer is the cheapest optimization there is. The four-characters rule is only an estimate; providers offer exact token counting, and their responses report the tokens used. Many providers also discount repeated identical prompt prefixes (prompt caching) and offer cheaper batch processing for work that does not need an immediate answer. For budgeting and cost alerts in general, see Cloud Costs and FinOps.

Structured output

When a program, not a person, reads the answer, free text is a problem: "The category is probably network" is hard to parse reliably. Structured output means making the model answer in a fixed machine-readable format, usually JSON that follows a schema you supply. Many APIs can enforce this: you pass a JSON Schema, and the provider constrains generation so the answer always parses and has the required fields. Without that feature you ask for JSON in the prompt and hope.

A schema for the ticket classifier:

{
  "type": "object",
  "properties": {
    "category": { "type": "string", "enum": ["network", "account", "hardware"] },
    "urgency": { "type": "integer", "minimum": 1, "maximum": 4 },
    "summary": { "type": "string", "maxLength": 80 }
  },
  "required": ["category", "urgency", "summary"],
  "additionalProperties": false
}

And an answer that conforms to it:

{
  "category": "network",
  "urgency": 2,
  "summary": "VPN error 809 on hotel Wi-Fi; network blocks VPN traffic"
}

A schema guarantees the shape, not the content: "network" is a valid category even when the ticket is about a printer. And when the format is only requested, not enforced, models sometimes wrap the JSON in chatty text or invent a category. So the code that receives the answer still validates it, exactly like any other untrusted input:

import json

ALLOWED = {"network", "account", "hardware"}

# Four answers as a model might return them (text, not yet data).
replies = [
    '{"category": "network", "urgency": 2, "summary": "VPN error 809"}',
    '{"category": "printer", "urgency": 3, "summary": "Paper jam"}',
    'Sure! Here is the JSON: {"category": "account"}',
    '["network", 2]',
]


def check(reply):
    try:
        data = json.loads(reply)
    except json.JSONDecodeError:
        return "rejected: not valid JSON"
    if not isinstance(data, dict):
        return "rejected: not a JSON object"
    if data.get("category") not in ALLOWED:
        return f"rejected: unknown category {data.get('category')!r}"
    if data.get("urgency") not in (1, 2, 3, 4):
        return "rejected: urgency must be 1-4"
    return f"accepted: {data['category']}, urgency {data['urgency']}"


for reply in replies:
    print(check(reply))

accepted: network, urgency 2
rejected: unknown category 'printer'
rejected: not valid JSON
rejected: not a JSON object

This check is deliberately small. A real validator also checks the type of every field, ideally with a JSON Schema library instead of hand-written tests. Note that in Python True counts as an int and equals 1, so both this check and a plain isinstance(value, int) test would let "urgency": true through.

A rejected answer can be retried, sent to a person, or put in a default queue; what it must not do is flow unchecked into a ticketing system, a database query or a shell command. That last risk is "improper output handling" in the AI Security page.

Calling a model through an API

Behind every chat app is an HTTP API (application programming interface): your program sends a JSON request with the model name, the system prompt, the messages and settings such as the maximum output length, and receives JSON back with the generated text and the token usage. The shape is similar across providers, and it is the same REST style you may know from network automation. As an example, this is a request to Anthropic's Messages API as documented in October 2026 (other providers differ in URL, header names and field names, not in the idea):

POST https://api.anthropic.com/v1/messages
x-api-key: (read from the ANTHROPIC_API_KEY environment variable)
anthropic-version: 2023-06-01
content-type: application/json

{
  "model": "(a current model id from the provider's documentation)",
  "max_tokens": 2000,
  "system": "You are a service desk analyst at Kestrel Energy. Answer in at most 80 words.",
  "messages": [
    { "role": "user", "content": "VPN error 809 from the hotel. What can I do?" }
  ]
}

max_tokens is a hard limit on the whole output. On models that reason before they answer, the thinking counts toward it too, so a limit that is too low can return an answer that is cut off, or no text at all.

In practice you use the provider's SDK (software development kit), a library that builds this request for you. The same call with Anthropic's Python SDK, again as an example from October 2026:

import os

import anthropic  # the vendor's SDK, installed with: pip install anthropic

# The key comes from an environment variable, never from the source code.
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

response = client.messages.create(
    model=os.environ["MODEL_ID"],  # look up current model ids in the docs
    max_tokens=2000,
    system="You are a service desk analyst at Kestrel Energy. "
    "Answer in at most 80 words.",
    messages=[
        {"role": "user", "content": "VPN error 809 from the hotel. What can I do?"},
    ],
)

for block in response.content:
    if block.type == "text":
        print(block.text)
print("tokens in/out:", response.usage.input_tokens, response.usage.output_tokens)

Treat the API key like a password: keep it in an environment variable or a secrets manager, out of source code and out of Git, give each application its own key, and set a spending limit in the provider's console. The cloud platforms offer the same models through their own services with their own authentication, such as Microsoft Foundry, Amazon Bedrock and Google's Gemini Enterprise Agent Platform (names as of 2026); see AI Infrastructure and Cloud Services.

Chat UIs versus APIs

A chat app and the API can use the same model but behave differently. The chat app adds its own long system prompt, memory features, file handling, web search and safety layers, and chooses settings for you. The API gives you the bare model: you write the system prompt, choose the settings, and build history, retrieval and tools yourself. Use the chat app for your own one-off work; use the API when a process must run the same way a thousand times, be tested, logged and costed. Terms differ as well: for business accounts and APIs, providers usually state that your data is not used for training, while consumer plans may differ. Check your organization's agreement before pasting anything sensitive.

Common mistakes

Smaller mistakes that cost time and money:

• Testing a prompt on one example. It worked once; that says little. Keep a set of real cases and rerun them after every change.
• Asking the model about itself. "Which version are you?" or "Did you use the knowledge article?" gets a plausible answer, not a measured one. Check the API response and your logs instead.
• Letting histories grow without limit. Long conversations cost more per turn and degrade quality. Start a new chat for a new task.
• Hard-coding a model name and forgetting it. Providers retire models on a schedule. Keep the model id in configuration and track deprecation notices.