Limits and Myths

What language models are good and bad at

A large language model (LLM) is trained on a huge amount of text to predict the next piece of text, then tuned to follow instructions. That origin explains a lot of its strengths and weaknesses. The table describes the general pattern as of 2026; models improve quickly, but the underlying reasons have stayed the same.

The left column is about transforming or producing text where "plausible and well-formed" is most of what you need, and where you can check the result. The right column is where the answer must be exactly right and the model has no reliable way to know it.

Hallucination: fluent is not true

A hallucination is confident, plausible output that is false or invented. It is not a glitch that appears now and then; it follows from how the model works. An LLM does not look an answer up in a database of facts. At every step it picks a likely next token given everything before it. A sentence that has the right shape, uses the right vocabulary and fits the context gets a high probability, whether or not it is true.

You can see the mechanism in miniature. The program below trains the simplest possible language model, a bigram model that only learns which word tends to follow which, on three true sentences about network ports. Then it generates sentences.

import random
from collections import Counter, defaultdict

# Three true sentences: the whole "training data"
corpus = [
    "port 22 is used by ssh",
    "port 53 is used by dns",
    "port 443 is used by https",
]

# Training: count which word follows which
follows = defaultdict(Counter)
for sentence in corpus:
    words = ["START"] + sentence.split() + ["END"]
    for current, nxt in zip(words, words[1:]):
        follows[current][nxt] += 1

def probability(sentence):
    """How likely the model finds a sentence: multiply next-word chances."""
    words = ["START"] + sentence.split() + ["END"]
    p = 1.0
    for current, nxt in zip(words, words[1:]):
        counts = follows[current]
        p *= counts[nxt] / sum(counts.values())
    return p

def generate(rng):
    word, output = "START", []
    while True:
        options = sorted(follows[word])
        weights = [follows[word][w] for w in options]
        word = rng.choices(options, weights)[0]
        if word == "END":
            return " ".join(output)
        output.append(word)

print("After 'by':", dict(sorted(follows["by"].items())))
rng = random.Random(1)
for _ in range(5):
    text = generate(rng)
    label = "seen in training" if text in corpus else "NEW, and false"
    print(f"{text:<26} p={probability(text):.3f}  {label}")

After 'by': {'dns': 1, 'https': 1, 'ssh': 1}
port 53 is used by https   p=0.111  NEW, and false
port 22 is used by ssh     p=0.111  seen in training
port 53 is used by dns     p=0.111  seen in training
port 53 is used by dns     p=0.111  seen in training
port 443 is used by https  p=0.111  seen in training

The very first sentence, "port 53 is used by https", is perfectly fluent and completely wrong. Look at the probabilities: the model gives the false sentence exactly the same score (1/3 × 1/3 = 0.111) as the true ones. It learned that "port" is followed by a number and "by" by a protocol name, but nothing ties 53 to DNS, because it only ever looks one word back. Nothing in its training objective says "true"; it only says "likely".

A real LLM is a neural network with billions of parameters, built with libraries such as PyTorch rather than a dictionary of counts, and it looks at thousands of tokens of context instead of one word. It would get this example right, because the true association appears countless times in its training data. But the principle scales: where the training data is thin, contradictory or absent, such as a rare error code, an obscure configuration flag or your company's internal ticket numbers, the model still produces the most plausible-looking continuation. Research from OpenAI in 2025 argued that training and evaluation also tend to reward a confident guess over "I don't know", much like a multiple-choice exam without a penalty for wrong answers.

Grounding the answer in documents you supply (retrieval-augmented generation), asking for sources, and checking the result reduce hallucinations. None of them eliminates them. There is no single "hallucination rate" either: it depends on the model, the task and how it is measured.

Brittle reasoning and missing knowledge

Reasoning and arithmetic

Models can solve many multi-step problems, and reasoning models, which generate intermediate steps before answering, are clearly better at it. But the reasoning can be brittle, meaning it breaks under changes that should not matter. Typical patterns:

• Surface changes. Change the names and numbers in a familiar word problem, or add an irrelevant sentence, and accuracy can drop. A 2024 study by Apple researchers (GSM-Symbolic) showed this on school-level math problems.
• Famous-puzzle traps. Give a well-known riddle with one detail changed so the trick no longer applies, and a model may answer the famous version anyway: the familiar pattern outweighs the details in front of it.
• Arithmetic and counting. Multiplying two long numbers "in its head" can produce an answer with the right number of digits and wrong digits in the middle. Counting letters in a word is hard because the model sees tokens, chunks of characters, not individual letters.
• Compounding errors. In a long chain of steps, one wrong intermediate result is carried forward confidently.

The practical fix is the same as for people: let the model use a tool. An assistant that writes and runs a small script to do the calculation, or that queries the real system instead of remembering it, is far more reliable than one that answers from memory. See AI agents and tool use.

Knowledge cut-off

A model learns from data collected up to a certain date, its knowledge cut-off. It does not know about later software releases, vulnerabilities, price changes or laws, unless the application gives it a search tool or the relevant documents. It may not even know its own cut-off precisely, and it can present outdated information as current. For anything that changes, such as product names, versions and deadlines, check a current source.

A model of the world?

Humans learn from acting in the world: we see, touch, try and get feedback. A text model learns from descriptions of the world. That is why it can explain physics fluently and still get simple everyday situations wrong, or lose track of which object is where in a long story. Whether large models build an internal "world model" is an open research question. Some studies find structured internal representations, for example of a game board in a model trained only on game moves; others find that performance relies on shortcuts that break outside familiar patterns. Both can be true at the same time, to different degrees in different tasks.

Benchmarks and why scores mislead

A benchmark is a fixed, public set of tasks with known answers used to measure and compare AI models, such as a collection of exam-style questions, coding problems or math puzzles. Benchmarks are how the field tracks progress, and AI announcements are full of benchmark scores. They are useful, but a score is a much narrower thing than it sounds:

• Contamination. Benchmarks are published on the internet, and models are trained on the internet. If test questions and answers leak into the training data, the model can recall instead of solve, and the score overstates its ability.
• Goodhart's law. "When a measure becomes a target, it ceases to be a good measure" (the popular phrasing by Marilyn Strathern). Once everyone optimizes for the same leaderboard, gains on the leaderboard stop meaning as much outside it. It is the AI version of a vanity metric.
• Saturation. When top models all score near 100%, the benchmark no longer tells them apart, and a new, harder one is made.
• Setup matters. Scores change with the prompt, the number of attempts allowed and the settings. Vendor-reported scores are not always run the same way as independent ones.
• Narrowness. A high score on multiple-choice questions says little about summarizing your incident tickets in Dutch.

The remedy is a small evaluation of your own: a few dozen real examples from your work with known good answers, run against the candidate models. The page on evaluating models explains how to measure fairly.

Seeing a mind that isn't there

Anthropomorphism is attributing human traits, such as understanding, feelings, intentions or beliefs, to something that is not human, here an AI system. Chat interfaces invite it: the system says "I", apologizes, sounds friendly and remembers what you said a minute ago.

This is not new. In 1966 Joseph Weizenbaum at MIT published ELIZA, a simple program that matched keywords and turned the user's sentences into questions, in the style of a psychotherapist ("You say you are worried about the migration. Why do you think that is?"). It had no understanding at all, yet users confided in it, and Weizenbaum was disturbed by how quickly people attributed insight to it. The tendency to read understanding into fluent output is now called the ELIZA effect.

Why it matters at work: if a tool feels like a knowledgeable colleague, people check it less. Friendly wording, confident tone and an apology followed by a "corrected" answer are features of the text, not evidence of knowledge or remorse. A useful habit is to say "the model generated" rather than "the AI thinks".

The stochastic parrots debate

In 2021 Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell published "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". A stochastic parrot, in their phrase, is a language model that stitches together sequences of words it has seen, based on probabilities, without any reference to meaning. The paper's broader warnings were about the environmental and financial cost of ever-larger models, training data too large to document and full of bias, and people mistaking fluent text for understanding.

The term became a shorthand in a debate that is still open.

• On one side: models are trained only on the form of language, not on its connection to the world; their errors, such as hallucinations and brittle reasoning, look like pattern-matching failing; and fluent output is exactly what would fool us.
• On the other side: large models solve problems that do not appear in their training data, write working code for new specifications, and research into their inner workings finds structured internal representations. Critics of the phrase argue that "just predicting the next word" can require building something like understanding.

Both sides agree on more than the slogans suggest: models are impressive in some ways and unreliable in others, and output needs checking. You do not need to settle the philosophical question to use AI well, but it helps to recognize which side a claim comes from.

The AGI debate

Artificial general intelligence (AGI) means AI that matches people across a broad range of cognitive tasks, not just one. Some AI companies state it as their goal and some of their leaders predict it within years; other researchers argue that today's methods will not get there without fundamental new ideas; others again question whether "general intelligence" is a well-defined target at all. There is no agreed definition and no test everyone accepts, so the same system can be called "close to AGI" by one person and "nowhere near" by another.

This page makes no prediction, and neither should you need one to do your job. When you read a claim about AGI, ask how the speaker defines it, what evidence they give, and whether they have a commercial or reputational stake in the answer.

AI hype and how to read AI news

AI hype is exaggerated or insufficiently supported claims about what AI can do or will soon do, often driven by marketing, investment or media attention. The pattern is old: the field went through two "AI winters" of cut funding after earlier promises were not met (see what is AI). Hype is not the same as progress being fake; real progress and inflated claims exist side by side. Questions to ask about any AI claim:

1. Who says it, and what do they sell? A vendor's launch post is not an independent test.
2. What exactly was shown? A demo, a benchmark score, or results from real use? Demos are selected.
3. Compared with what? Better than the previous model, than a simple rule, or than a skilled person?
4. How often does it fail, and how? A tool that is right 95% of the time can still be unusable if the 5% are silent and costly.
5. Can others reproduce it? Independent replication counts for more than a single impressive result.
6. Is AI doing the work? Some "AI" products are rules, people or plain automation underneath.
7. When is it from? AI news ages fast; check the date.

The opposite error exists too: dismissing a tool because of one funny failure. A fair judgment needs a test on the work you actually do.

Overreliance and automation bias

Overreliance is relying on AI output more than its actual reliability justifies: accepting answers without checking, letting it make decisions it is not suited for, or losing the skills to notice when it is wrong. It is the everyday consequence of automation bias, the human tendency to favor suggestions from an automated system over one's own judgment or contrary evidence.

It grows with success. The first few times, people check every line; after fifty good answers, they skim; the fifty-first answer, the one with a wrong firewall rule, goes straight into production. Countermeasures are mostly about habits and process: decide in advance what must be verified, keep review steps that cannot be skipped for changes with real impact, let juniors still solve problems themselves, and make it normal to say "I checked and the AI was wrong". The page on using AI at work turns this into a checklist.

Myths and reality