How a Model Reads Your Prompt

What the model does with your text

The mechanics of a language model are explained in Transformers and Large Language Models. For writing prompts, one fact matters most. Next-token prediction is what a language model does at every step: given all the tokens so far, it computes a probability for every token in its vocabulary as the next one, one token is chosen, it is appended, and the process repeats. Your prompt is only the first part of that sequence; the answer is the rest.

Three practical consequences follow:

• Your prompt sets the scene. The model continues the text it is given, so the style of the prompt rubs off on the answer. A prompt full of bullet points and Markdown headings tends to get bullets and headings back; a prompt in flowing prose tends to get prose.
• The answer builds on itself. Each token is chosen to fit the tokens already written, including the model's own earlier words. A model that starts with "Yes," will go on to justify yes. If you want the evidence weighed before a verdict, ask for the evidence first and the verdict last. Reasoning models do part of this on their own, by thinking before they answer (see Reasoning: Step by Step and Thinking Models).
• Nothing is looked up. Unless the application gives the model a search tool or pastes documents into the prompt, everything it writes comes from patterns learned in training plus the text in front of it. A version number, ticket status or policy detail that is not in the prompt is predicted, not retrieved.

Tokens: how your text is measured

Models read text as tokens, pieces of words with a number each. Limits and prices are counted in tokens, so you need an idea of how many tokens a text will take. A token estimate is an approximate token count calculated from the number of characters or words in a text with a rule of thumb, used for planning; the real count comes only from the model's own tokenizer or the provider's token-counting tool.

The rules of thumb differ per provider (as of October 2026):

So the same text costs a different number of tokens on different models. Dutch and most other languages need more tokens per word than English, and log lines, IP addresses, hexadecimal IDs and code are expensive because they contain few common words. The example compares the two common rules on three short texts.

# A ROUGH token estimate, not a tokenizer. Real counts come from the
# provider's token counter and differ from model to model.
texts = {
    "English": "The customer portal was unavailable from 08:10 to 09:05. "
               "Meter readings submitted during that time were not saved.",
    "Dutch": "Het klantportaal was niet beschikbaar van 08:10 tot 09:05. "
             "Meterstanden die in die tijd zijn doorgegeven, zijn niet opgeslagen.",
    "Log line": "2026-10-03T02:14:07Z fw-edge-02 ikev2: SA 0x7f3a9c21 deleted, "
                "peer 203.0.113.10 DPD timeout",
}


def by_characters(text):
    return round(len(text) / 4)              # about 4 characters per token


def by_words(text):
    return round(len(text.split()) / 0.75)   # about 3/4 word per token


print(f"{'text':<9}{'chars':>6}{'words':>6}{'by chars':>10}{'by words':>10}")
for name, text in texts.items():
    print(f"{name:<9}{len(text):>6}{len(text.split()):>6}"
          f"{by_characters(text):>10}{by_words(text):>10}")

text      chars words  by chars  by words
English     114    18        28        24
Dutch       127    19        32        25
Log line     91    10        23        13

For ordinary English the two rules roughly agree. For the log line they disagree by almost a factor of two, because ten "words" hide timestamps, an address and a hexadecimal ID. On current Claude models the by-words estimate undercounts; there a word is closer to 1.8 tokens. Use estimates like these to decide whether something will fit, with a wide margin, and the provider's counter (or the token usage reported with each response) when the number matters. What tokens cost is covered in Tokens and cost.

Choosing the next token: sampling

The model's output at each step is a list of probabilities, not a single word. Token sampling is the step that picks the next token from those probabilities. Always taking the most likely token is called greedy decoding; most applications instead draw a token at random according to its probability, so a token with 46% probability is chosen about 46 times in a hundred. Sampling keeps text varied and natural, and it is the main reason the same prompt gives different answers. Temperature reshapes the probabilities before the draw: lower values favor the likeliest tokens, higher values give the others more chance.

Top-p sampling

A second common setting cuts off the unlikely tail. Top-p sampling (also called nucleus sampling) draws the next token only from the smallest group of most likely tokens whose probabilities add up to at least p, after rescaling their probabilities to a total of 100%; everything outside that group cannot be chosen. A related setting, top-k, keeps a fixed number of the most likely tokens instead.

# Made-up probabilities (in percent) for the next word after
# "The VPN tunnel to the Zwolle office ..."
probs = {"dropped": 46, "failed": 22, "flapped": 12, "recovered": 9,
         "is": 6, "rebooted": 3, "sang": 2}


def nucleus(probs, top_p):
    """The smallest set of most likely words that together reach top_p."""
    kept, total = [], 0
    for word, p in sorted(probs.items(), key=lambda kv: (-kv[1], kv[0])):
        kept.append(word)
        total += p
        if total >= top_p:
            break
    return kept, total


for top_p in (50, 80, 95, 100):
    kept, total = nucleus(probs, top_p)
    print(f"top_p {top_p / 100:.2f}: {len(kept)} candidates, {total}% -> {', '.join(kept)}")

top_p 0.50: 2 candidates, 68% -> dropped, failed
top_p 0.80: 3 candidates, 80% -> dropped, failed, flapped
top_p 0.95: 5 candidates, 95% -> dropped, failed, flapped, recovered, is
top_p 1.00: 7 candidates, 100% -> dropped, failed, flapped, recovered, is, rebooted, sang

With top-p 0.95 the absurd "sang" can never be picked; with 1.00 it keeps its 2% chance. Over an answer of hundreds of tokens, those small chances add up, which is why sampling settings change the character of long answers more than of short ones.

Sampling settings today

Older guides tell you to set temperature 0 for consistent answers. Current guidance (as of October 2026) has moved away from tuning these settings, especially for reasoning models:

In most chat assistants you cannot change these settings at all. So in practice you control variation through the prompt: the more precisely a prompt pins down the answer, the less room sampling has to wander.

Suggest a name for Kestrel Energy's new internal status page for outages. Reply with the name only.

Kestrel Energy needs a name for its new internal status page for outages. The staff use it during incidents, so the name must say what the page is for. Choose one of these three names and reply with the name only: Kestrel Status, Grid Pulse, Outage Board.

The context window is a budget

The context window is the maximum number of tokens a model can take into account at once. As of October 2026 the current models of the large providers accept roughly 200,000 to about 1 million tokens: for example 1 million for Claude Opus 5.5 and Sonnet 5.5, 200,000 for Claude Haiku 4.5, about 1.05 million for the GPT-6 family and 1,048,576 input tokens for Gemini 3.8 Flash. That sounds like room for anything, but your pasted text is not the only thing in it:

• the assistant's own system prompt, which chat apps add without showing it;
• the earlier messages of the conversation, resent every turn;
• your instructions and your pasted material;
• the model's hidden reasoning: OpenAI's documentation says reasoning tokens "still occupy space in the model's context window and are billed as output tokens", and Google bills thinking tokens as output too;
• and the answer itself.

# How much of a context window is left for your material?
# All numbers are illustrative; real sizes differ per model and product.
CONTEXT_WINDOW = 200_000          # tokens the model can take in at once
TOKENS_PER_LOG_LINE = 30          # estimate for one typical syslog line

already_used = {
    "the assistant's own system prompt": 3_000,   # added by the chat app
    "earlier messages in this chat": 6_500,
    "your instructions and question": 400,
    "room for thinking": 16_000,                  # reasoning models
    "room for the answer": 4_000,
}

for part, tokens in already_used.items():
    print(f"{part:<36}{tokens:>8,}")
left = CONTEXT_WINDOW - sum(already_used.values())
print(f"{'left for pasted material':<36}{left:>8,}")
print(f"about {left // TOKENS_PER_LOG_LINE:,} log lines would fit")

the assistant's own system prompt      3,000
earlier messages in this chat          6,500
your instructions and question           400
room for thinking                     16,000
room for the answer                    4,000
left for pasted material             170,100
about 5,670 log lines would fit

Thousands of log lines would fit. That does not mean you should paste them, for two reasons: what you paste leaves your hands (see What never to paste), and more text makes the model worse at using it.

Context rot

Context rot is the gradual decline in how well a model finds and uses information as the amount of text in its context grows. Anthropic's engineering team describes it with the image of a limited "attention budget": in a transformer every token relates to every other token, which makes "n² pairwise relationships for n tokens", and each extra token draws on that budget. A fact that is easy to find in 50 lines can be overlooked in 5,000. The practical rule: give the model the minimum material that answers the question, such as the lines around an error instead of the whole log. How to select and mark reference material is on Giving Context and Reference Material.

Position matters

Lost in the middle

Lost in the middle is the tendency of language models to use information at the start or end of a long input more reliably than information in the middle. The name comes from a 2023 study by Liu and colleagues, which found that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts". Those measurements were made on 2023 models; newer models are generally reported to do better, but the providers still give placement advice.

Microsoft's guide adds a related effect, recency: information at the end of the prompt "might have more significant influence over the output than information at the beginning", which is why it suggests repeating key instructions at the end. The vendor advice (as of October 2026):

• Anthropic: put long documents at the top and the question at the end; in its tests, queries at the end improved response quality "by up to 30 percent", especially with complex inputs of several documents.
• Google: supply all the context first and put the specific instructions or question at the very end.
• Microsoft: consider repeating the instructions at the end. Its guide also recommends stating the task at the start, while noting that in its testing ChatGPT and GPT-4 gave the same response whether or not that was done.

The common ground: long material first, the question last, and key instructions short enough to repeat. The same layout, before and after:

What's wrong here? Also keep it short.
[300 lines of firewall log]
By the way, the problem started around 02:14.

Below is the syslog of fw-edge-02 from 02:00 to 02:30.
<log>
[the 40 lines around 02:14]
</log>
The VPN tunnel to 203.0.113.10 dropped at 02:14. Which three lines in the log best explain why? Quote each line exactly, then give one sentence of explanation per line.

The second prompt states the time and the question after the material, trims the log to the relevant window, and asks for exact quotes, which you can search for in the log to check that they exist. Placement effects depend on the model and the task, so the only reliable way to know is to test. The example builds three variants of one prompt that differ only in where the key fact sits; running each several times and counting correct answers is a small experiment of the kind described in Iterating and Testing Prompts.

# Build three versions of one prompt that differ ONLY in where the key
# fact sits. Run each version several times and count the correct answers.
key_fact = "02:10 CHG-2207 applied: VLAN 30 moved to port Gi0/3 on sw-core-01"
noise = [f"02:{minute:02d} sw-core-01 routine health check OK" for minute in range(40)]
question = "Which change most likely explains the outage that started at 02:14?"


def build(position):
    lines = list(noise)
    index = {"start": 0, "middle": len(lines) // 2, "end": len(lines)}[position]
    lines.insert(index, key_fact)
    return "\n".join(lines) + "\n\n" + question


for position in ("start", "middle", "end"):
    prompt_lines = build(position).splitlines()
    print(f"{position:<7} key fact on line {prompt_lines.index(key_fact) + 1:>2} "
          f"of {len(prompt_lines)}, question on line {len(prompt_lines)}")

start   key fact on line  1 of 43, question on line 43
middle  key fact on line 21 of 43, question on line 43
end     key fact on line 41 of 43, question on line 43

With 43 lines, a current model is very likely to find the fact in every position; the effect shows up with long inputs. The point of the construction is the method: change one thing, keep everything else equal, and measure.

In a chat, everything earlier is part of the prompt

A chat feels like a conversation, but the model itself remembers nothing between requests. The application sends the whole conversation again with every new message (see Stateless: the history is sent again). Your latest message is therefore only the end of a long prompt, and everything before it still steers the answer: an early misunderstanding, a pasted document you no longer need, a format you asked for three questions ago.

That explains a familiar pattern: after a long session the assistant seems to drift, repeat old mistakes or ignore a correction. The history is working against you. Start a new chat for a new task, and when a chat has gone wrong, start over with a better first prompt instead of adding corrections. Some assistants also have memory features that store notes about you and insert them into later chats; those notes are part of the prompt too.

What this means for your prompts

• Put in what the model needs. It cannot look up what you leave out; it will predict something plausible instead.
• Ask for things in the order you want them thought through. Evidence before verdict, facts before advice.
• Long material first, question last, and the key instruction short and clear enough to repeat.
• Send less, not more. Trim logs, documents and histories to what the task needs.
• Expect variation. Judge a prompt on several runs, and narrow the answer with the prompt rather than hoping for settings you cannot change.
• Count tokens properly when it matters. Rules of thumb are for rough planning only.