# How a language model works

> What a language model actually does when it answers, why that explains its strengths, and why it explains its failures too.

Page: https://schubergphilis.github.io/ai-training/concepts/how-models-work/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

You type a question, and a paragraph comes back that reads like a person
wrote it. It is tempting to picture a mind on the other side that looked up
the answer. Nothing like that happens. In this lesson we replace that
picture with the real one. It is simpler and it explains more: a language
model predicts the next piece of text, one piece at a time.

## Text becomes tokens

A model never sees letters or words. Its input is cut into **tokens**, short
chunks of text that are usually a word, part of a word, or a punctuation
mark. "Unbelievable" might be three tokens, and "the" is one. Each token has a
number, and the model works only with those numbers.

This has consequences you notice. Models are bad at counting letters
in a word, because they never saw the letters. They're priced per token,
not per word. And rare names and non-English text often cost more tokens
per character than common English, because the token vocabulary was built
around the text that appears most often in training data.

## One token at a time

Given the tokens so far, the model outputs a score for every token in its
vocabulary: how likely is each one to come next? Those scores become a
probability distribution. The model picks one token from it, appends it to
the input, and the whole thing runs again. A hundred-word answer is a hundred
or so rounds of "what comes next?".

Try it. The widget below is a toy with five candidate tokens and made-up
scores, but the mechanism is the real one.

The input fixes the distribution, so the randomness is in the *pick*
alone. Along the same line, **temperature** doesn't add knowledge: it
flattens or sharpens the same distribution. At low temperature the model
almost always picks the top token and sounds confident and repetitive. At
high temperature it wanders.

#### Pitfall: Asking the model why

You ask a model to solve a puzzle and it gets it wrong. You ask "why did
you say that?" and it produces a fluent explanation. That explanation is
*also* next-token prediction conditioned on your question. It is a
plausible story about the earlier text. The model can't see what happened
inside. Rule: treat self-explanations as new output to be checked, not as
evidence.

## Where the scores come from

A neural network with billions of adjustable numbers, the **weights**,
produces the scores. During training the network reads enormous
amounts of text and is nudged, each time, so that the token that really
came next scores higher, and it doesn't store facts or rules. The result
is a compressed statistical model of the text it saw: grammar, style,
common facts, common code, common arguments.

So models know a great deal about things that appear often in
writing and are unreliable about things that appear rarely, recently, or
only in your organization's private documents. Knowledge is
frequency-weighted and frozen at the training cutoff.

That first training stage produces a text continuer, and a text continuer
has no idea that you want an answer. Type a question into a raw model and
it may reply with more questions, in the pattern of a worksheet. A second,
shorter stage fixes this: the network is trained on example replies that
people have rated, and it learns that your text is a question to answer.
Those ratings have side effects. People tend to rate agreeable replies and
long replies higher. The finished model leans toward both
(AI capabilities and limitations, Claude Academy).

#### Checkpoint: What did the model do?

A colleague says: "The model looked up the answer in its database and
returned it." Which correction is right?

- It looked the answer up, but the database is called the weights.
- It produced the answer token by token, each time scoring what text is likely to come next.
- It searched the web and summarized the results.
- It picked the closest match from a library of pre-written answers.

## Failure modes, named

Once you see generation as prediction, the famous failures stop being
mysterious.

- **Hallucination.** The most likely continuation of "The 2019 paper by
  Smith et al. showed" is a plausible citation, even if no such paper exists.
  Confidence in the prose isn't confidence in the fact. Made-up details
  concentrate in the specifics, such as a release date, a page number, or a
  quoted line. A common concept has been written about thousands of times
  and the prediction is well constrained, but one exact figure has few or
  no examples behind it, and the model fills the gap with something of the
  right form. A detail the model saw once comes out in the same confident
  prose as a fact it saw ten thousand times
  (AI capabilities and limitations, Claude Academy).
- **Sycophancy.** Agreeable replies were rated well in the second training
  stage, so agreeing is likely text. Push back and the model often folds,
  even when its first answer was right.
- **Instruction dilution.** Everything in the context is just tokens.
  A rule you stated on page one competes with everything said since, and
  long conversations drift. Position in the context window matters too. A
  rule buried on page six of a twelve-page paste weighs less in the
  prediction than the same rule at the top or at the bottom. Liu and
  colleagues measured this. When the one document that holds the answer
  was in the middle of a long context, models answered less accurately than
  when it was at the start or at the end, and models built for long
  contexts showed the same curve (Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics 12 (2024), 157-173).
- **Cutoff and staleness.** Anything after training is invisible unless a
  tool puts it in the context.

#### Checkpoint: Name the failure

You ask for the release date of a library version that came out last
month. The model answers with a specific date, stated confidently, and it
is wrong. Which failure mode is the best name for this?

- Sycophancy triggered by the phrasing of the question
- Hallucination driven by the training cutoff
- Instruction dilution in a long conversation
- A tokenization problem with the version number

## Exercise

Open any chat assistant you have access to. Ask it to count the letter "r"
in "strawberry" and to spell the word backwards. Then ask it for the
opening sentence of a book you know well, and ask that last question a
second time in a fresh chat. Note which answers are right and, for the
wrong ones, which failure mode from this lesson explains it. If the two
answers about the book differ, the difference comes from sampling, the
pick from the distribution you saw in the widget.

A good result names one failure mode per wrong answer and can say, in a
sentence, why prediction rather than lookup produces it.

Stretch: Ask the assistant to explain why it got one of them wrong, then decide whether you believe the explanation and why.

## Recap

1. Input is tokens rather than words, and that alone explains a number of odd weaknesses.
2. Output is one token at a time, sampled from a distribution the model
   scores. Temperature reshapes the distribution and doesn't add knowledge.
3. The scores come from weights fitted to training text, so knowledge is
   frequency-weighted and stops at the cutoff.
4. A second, shorter training stage on rated replies turns the text
   continuer into an assistant, and the ratings have side effects.
5. Hallucination, sycophancy, dilution, and staleness all follow from those
   two stages (How agents think: tokens and context, reasoning strategies, model choice, system prompts, Agent Engineer Course).

## References

- AI capabilities and limitations, Claude Academy, https://academy.claude.com/courses/ai-capabilities-and-limitations
- Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics 12 (2024), 157-173, https://aclanthology.org/2024.tacl-1.9/
- How agents think: tokens and context, reasoning strategies, model choice, system prompts, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/02-how-agents-think/