Transformers and Large Language Models

What a large language model is

A large language model (LLM) is a neural network with billions of parameters, trained on very large amounts of text to predict the next token (a word or a piece of a word); it generates text by repeating that prediction. Chat assistants, coding assistants and the "summarize this ticket" button in a service desk tool are all applications built on such a model; they are the most visible form of generative AI.

The pipeline, from the inside out: text is cut into tokens; each token becomes a list of numbers (an embedding); a stack of transformer layers mixes information between the tokens through attention; the last layer produces a probability for every token in the vocabulary being the next one. One token is chosen, appended, and the whole process runs again. The rest of this page takes those steps one at a time. How to use these models well — prompts, temperature, structured output, APIs — is on Working with LLMs.

Tokens and tokenizers

A neural network cannot read letters; it needs numbers from a fixed list. A token is the unit of text a language model reads and writes: a whole word, part of a word, a punctuation mark or a single character, each with a number in the model's vocabulary. A tokenizer is the component that splits text into those tokens (and turns tokens back into text), using a vocabulary fixed before training.

Why pieces of words, and not whole words? A vocabulary of every word in every language — plus every product name, typo and error code — would be endless. A vocabulary of single characters would make texts very long. Subword pieces are the compromise: common words get one token, rare words are built from several pieces. The toy tokenizer below has 15 pieces and splits each word by greedy longest match: take the longest known piece at the start, then continue with the rest.

# A tiny vocabulary of known pieces; "_" marks the start of a word
vocab = {
    "_the", "_router", "_dropped", "_packet", "_is", "_to", "_un", "_re",
    "reach", "connect", "able", "ing", "_server", "s", "ed",
}


def tokenize_word(word):
    text = "_" + word.lower()
    tokens = []
    while text:
        # Greedy longest match: try the longest known piece first
        for end in range(len(text), 0, -1):
            if text[:end] in vocab:
                piece = text[:end]
                break
        else:
            piece = text[0]              # unknown: fall back to one character
        tokens.append(piece)
        text = text[len(piece):]
    return tokens


for sentence in ["The router dropped the packet",
                 "Servers unreachable",
                 "Reconnecting to Kestrel"]:
    tokens = [t for word in sentence.split() for t in tokenize_word(word)]
    print(f"{sentence!r}: {len(sentence.split())} words, {len(tokens)} tokens")
    print("  ", tokens)

'The router dropped the packet': 5 words, 5 tokens
   ['_the', '_router', '_dropped', '_the', '_packet']
'Servers unreachable': 2 words, 5 tokens
   ['_server', 's', '_un', 'reach', 'able']
'Reconnecting to Kestrel': 3 words, 12 tokens
   ['_re', 'connect', 'ing', '_to', '_', 'k', 'e', 's', 't', 'r', 'e', 'l']

Common words cost one token each; "unreachable" is assembled from meaningful pieces; the unknown name "Kestrel" falls apart into single letters and costs eight tokens. Real tokenizers work the same way in spirit but learn their vocabulary — typically tens of thousands to a few hundred thousand pieces — from data. The most common method, byte-pair encoding (BPE), starts from single bytes and repeatedly merges the most frequent adjacent pair into a new piece. Because every single byte is in the vocabulary, no text is ever "unknown"; it just costs more tokens.

This has practical consequences. API prices and limits are counted in tokens, not words (see Tokens and cost). A common rule of thumb is that a token is roughly ¾ of an English word (100 tokens ≈ 75 words), but it depends on the tokenizer: Anthropic, for example, states that its current tokenizer fits about 555,000 words into a million tokens, against about 750,000 for older ones. Dutch, German and most other languages usually need more tokens per word than English, as do code, log lines and long hexadecimal IDs. Tokenization also explains some odd failures: a model that sees "strawberry" as two or three tokens does not directly see its letters, so counting the r's is harder for it than it looks.

From tokens to embeddings

Each token number is then looked up in a table that gives it a long list of numbers — an embedding, often several thousand numbers per token. These are learned during training, and tokens used in similar ways end up with similar lists: "router" and "switch" lie close together, "router" and "banana" far apart. A second signal is added to record each token's position, since word order matters. Embeddings of whole sentences and documents are also the basis of semantic search and retrieval-augmented generation; that use is explained on Embeddings and Retrieval.

Attention

The meaning of a word depends on the other words around it. In "The router dropped the packet because it was full", what is it? A person knows: the router (its buffer was full). Change the last word to "corrupted" and it becomes the packet. A model needs a way for each word to pull in information from the right other words.

Attention is the mechanism that lets each token in a sequence compute how relevant every other token is to it, and then build its new representation as a weighted mix of those tokens, with the most relevant ones counting most. The weights are not fixed; they are computed fresh for every input.

How it works, in words. From each token's embedding the model computes three vectors with learned weights: a query ("what am I looking for?"), a key ("what do I contain?") and a value ("what do I pass on?"). The query of "it" is compared with the key of every token (a dot product: multiply position by position and add up); the scores are scaled and turned into weights that sum to 1 with the softmax function; and the new representation of "it" is the weighted sum of all the values. Written compactly this is softmax(QK^T / √d_k) · V, "scaled dot-product attention". A model runs many such attention heads side by side (multi-head attention), so one head can track what "it" refers to while another tracks grammar or nearby words.

Two consequences matter in practice. Attention connects any two tokens directly, however far apart, which solved the long-range forgetting of recurrent networks. And every token is compared with every other token, so the work grows with the square of the text length: twice as much text means about four times as many comparisons. That is one reason long contexts are expensive.

The transformer

The transformer is a neural network architecture, introduced in the June 2017 paper "Attention Is All You Need" by Ashish Vaswani and colleagues at Google, that processes a sequence with layers of attention and small fully connected networks instead of recurrence, so all positions are processed in parallel. It was designed for translation and became the basis of nearly all modern language models.

Each transformer layer (block) does two things: an attention step in which tokens exchange information, and a feed-forward step in which each token's numbers are processed on their own by a small neural network. Large models stack dozens of these blocks. Because nothing has to wait for the previous word, training uses GPUs far better than recurrent networks could — which is what made training on internet-scale text practical (see recurrent networks and LSTM for what came before).

The original design has two halves, and models use them differently:

• Encoder: reads the whole input, every token seeing every other token, and produces a representation of it. Encoder-only models such as BERT are used for classification, search and embeddings.
• Decoder: generates output one token at a time, where each token may only look at the tokens before it (it must not peek at the answer). Decoder-only models — the GPT style — are what chat assistants and nearly all current LLMs use.
• Encoder–decoder: the original layout, for tasks that turn one text into another, such as translation.

Pretraining: predicting the next token

Pretraining is the first, largest training phase of a model, in which it learns general patterns from a huge unlabeled dataset; for a language model the task is to predict the next token of real text, over and over. No one has to label anything: the text itself supplies the right answer — the word that actually came next. The loss is the log loss from Neural Networks: how much probability the model gave to the true next token.

The simplest possible next-word predictor just counts which word follows which in its training text — a bigram model. It is nothing like a transformer inside, but it shows the task honestly:

from collections import Counter, defaultdict

# Training text: a few lines from (fictional) incident notes
text = """the router dropped the packet .
the router restarted after the update .
the firewall dropped the connection .
the user restarted the laptop .
the router dropped the session ."""

# Count which word follows which (a bigram model)
follows = defaultdict(Counter)
words = text.split()
for current, nxt in zip(words, words[1:]):
    follows[current][nxt] += 1

print("after 'the':", sorted(follows["the"].items(), key=lambda kv: (-kv[1], kv[0])))
print("after 'dropped':", dict(follows["dropped"]))

# Generate greedily: always pick the most frequent next word
word, output = "the", ["the"]
for _ in range(6):
    candidates = follows[word]
    best = max(sorted(candidates), key=candidates.get)  # ties: alphabetical
    output.append(best)
    word = best
print("generated:", " ".join(output))

after 'the': [('router', 3), ('connection', 1), ('firewall', 1), ('laptop', 1), ('packet', 1), ('session', 1), ('update', 1), ('user', 1)]
after 'dropped': {'the': 3}
generated: the router dropped the router dropped the

The model produces fluent-looking fragments with no idea what a router is, and greedy choice traps it in a loop. Both lessons carry over. An LLM also predicts from statistics of its training text — which is why it can produce confident, plausible text that is wrong (a hallucination). And real systems do not always take the single most likely token: they sample from the probabilities, with a temperature setting that controls how adventurous the choice is.

The difference is scale and context. A bigram model looks at one previous word; a transformer looks at thousands of previous tokens at once through attention, and its billions of weights can capture grammar, facts, styles, code syntax and patterns of reasoning that appear in the text. Predicting the next token well across trillions of tokens of varied text forces a model to pick up a great deal of this — how much that amounts to "understanding" is debated; see Limits and Myths.

Research on scaling laws (Kaplan et al., 2020) found that the loss falls predictably as a power law as parameters, data and compute grow. The "Chinchilla" study (Hoffmann et al., 2022) found that compute-optimal training uses about 20 tokens of text per parameter — so a 70-billion-parameter model would want about 1.4 trillion tokens. GPT-3 (2020) had 175 billion parameters; current frontier models are generally larger, but most providers no longer publish exact sizes.

From text predictor to assistant

A freshly pretrained model continues text; it does not follow instructions. Ask it "How do I reset my VPN password?" and it may continue with three more forum questions. Two further training steps turn it into an assistant.

Fine-tuning on instructions

First the model is fine-tuned on tens of thousands of example conversations — an instruction and a good answer, often written by people — so it learns the format and habit of answering helpfully (supervised fine-tuning, SFT). The same technique lets an organization adapt a model to its own domain, style or output format. Fine-tuning changes how a model behaves much more reliably than what it knows; to give a model access to your own, current documents, retrieval is usually the better tool (see retrieval-augmented generation).

Learning from human feedback

Many qualities of a good answer — helpful, honest, harmless, the right length and tone — are easier to recognize than to write down. Reinforcement learning from human feedback (RLHF) is a training method in which people compare or rate a model's answers, a separate reward model learns to predict those preferences, and the language model is then trained with reinforcement learning to produce answers the reward model scores highly. The method was described by Paul Christiano and colleagues in 2017; OpenAI's InstructGPT paper (Ouyang et al., 2022) made it the standard recipe for instruction-following assistants, in three steps: supervised fine-tuning, a reward model, then optimization with an algorithm called PPO (proximal policy optimization).

Variants and alternatives are now common: preference-optimization methods such as DPO (direct preference optimization) that skip the separate reward model, and feedback generated by AI models against written principles instead of, or alongside, human raters. The general idea is the same — learn from judgments about which answer is better. It shapes behavior; it does not guarantee correctness. (Reinforcement learning in general is explained on How Machines Learn.)

Foundation models and the context window

A foundation model is a large model pretrained on broad data that can be adapted — by prompting, fine-tuning or retrieval — to many different tasks, instead of being built for one. The term was coined by Stanford researchers in 2021. LLMs are the best-known foundation models; others work with images, audio or several kinds of data at once (see Vision, Speech and Multimodal AI). Cloud platforms offer catalogs of them as managed services; see Cloud AI services.

The context window

The context window is the maximum number of tokens a model can take into account at once: the instructions, the conversation so far, any pasted documents and the answer it is writing, together. Anything outside it simply does not exist for the model. As of October 2026, frontier models offered through APIs typically have context windows from about 128,000 to about 1 million tokens; open-weight models are commonly around 128,000.

A large context window is not a memory. A chat model does not remember earlier conversations; the application resends the history each time (see Stateless: the history is sent again), and long conversations are trimmed or summarized when they no longer fit. A large window also costs more per request, and models do not use every part of a very long context equally well — information in the middle of a long document can be overlooked. Retrieving only the relevant passages is often both cheaper and more accurate than pasting everything.

Reasoning models

A reasoning model is a language model trained to generate intermediate reasoning steps — often hidden from the user — before giving its final answer, spending extra tokens and time to do better on multi-step problems such as math, code and planning. The first publicly released one was OpenAI's o1 in September 2024. In January 2025 DeepSeek-R1 showed, with openly published weights and a paper, that such behavior can be trained largely with reinforcement learning on problems whose answers can be checked automatically.

As of 2026, current APIs let you control how much a model thinks, typically through an "effort" setting such as low, medium or high, and some providers turn reasoning on by default and let the model decide how much to use. More thinking usually means better answers on hard tasks, at the cost of more tokens, higher latency and more energy per request; for a simple classification it is wasted.

Whether this amounts to reasoning in the human sense is debated. The intermediate text measurably improves results on many benchmarks, but it is still generated text: it can contain errors, and it is not always a faithful account of how the model reached its answer. Treat the output of a reasoning model like any other: check what matters. See Reasoning and arithmetic.

Model families as of 2026

Model names and versions change every few months, so this site uses family names. An important distinction is how a model is offered. A closed-weight model can only be used through the provider's service or API. An open-weight model can be downloaded and run on your own hardware, under a license that may still restrict how you use it — which is not the same as open source in the sense of the Open Source Initiative, where training code and data would also be open.

Which family "leads" changes from month to month and depends on the task, so rankings are deliberately left out here. For a real choice, test candidate models on your own examples (see Evaluating generative AI) and weigh cost, latency, context window, data location and license. Running open-weight models yourself is covered on AI Infrastructure and Cloud Services.