# How language models work

> A language model turns text into tokens, predicts the next token from everything it has seen so far, and repeats until it stops. This topic covers the pieces behind that loop: tokens, the context window that bounds what the model can see, the split between training and inference, the trade-offs between model families and sizes, and how sampling settings shape the output.

Page: https://schubergphilis.github.io/ai-training/topics/concepts/how-models-work/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

*Concepts* · topic `concepts/how-models-work`

A language model turns text into tokens, predicts the next token from everything it has seen so far, and repeats until it stops. This topic covers the pieces behind that loop: tokens, the context window that bounds what the model can see, the split between training and inference, the trade-offs between model families and sizes, and how sampling settings shape the output.

## Concepts

- **Token**: The unit a language model reads and writes. A token is a chunk of text, often a word fragment, punctuation mark or common word, chosen by a fixed tokenizer. Models are priced, limited and timed in tokens, not words or characters. A rough rule for English is three to four characters per token, so a page of text is a few hundred tokens. [glossary](https://schubergphilis.github.io/ai-training/glossary/#token)
- **Context window**: The maximum number of tokens a model can take into account at once, counting the instructions, the conversation so far, any documents pasted in and the answer it is producing. Anything outside the window does not exist for the model. Larger windows cost more per call and do not guarantee the model uses everything inside them equally well. [glossary](https://schubergphilis.github.io/ai-training/glossary/#context-window)
- **Training vs inference**: Training is the one-off, expensive phase where a model's weights are adjusted on a large body of text until it predicts well. Inference is every later use of the frozen model to produce output. Nothing you type during inference changes the weights; the model only "remembers" what is in the current context window. [glossary](https://schubergphilis.github.io/ai-training/glossary/#training-vs-inference)
- **Model family and size**: Vendors publish models in families with several sizes. Larger models usually reason better and follow complex instructions more reliably, but cost more and respond more slowly. Smaller models are cheaper and faster and are often good enough for narrow, well-specified tasks. Choosing a model means matching the size to the difficulty of the job. [glossary](https://schubergphilis.github.io/ai-training/glossary/#model-family-and-size)
- **Sampling and temperature**: At each step the model has a probability for every possible next token and picks one by sampling. Temperature scales those probabilities: low values make the model pick the most likely token almost every time, high values spread the choice across less likely options. Low temperature gives consistent, conservative output; high temperature gives variety at the cost of more mistakes. [glossary](https://schubergphilis.github.io/ai-training/glossary/#sampling-and-temperature)

## Links

- **Builds on:** nothing
- **Leads to:** [Capabilities and limits](https://schubergphilis.github.io/ai-training/topics/concepts/limits/), [Prompting](https://schubergphilis.github.io/ai-training/topics/concepts/prompting/), [Tokens, cost and effort](https://schubergphilis.github.io/ai-training/topics/concepts/tokens-and-cost/)
- **Competencies drawing on it:** [Explains how a language model produces text and where it fails](https://schubergphilis.github.io/ai-training/competencies/concepts/explains-models/), [Writes prompts that get reliable results](https://schubergphilis.github.io/ai-training/competencies/concepts/prompts-reliably/)

## Lessons

- [What the model can see](https://schubergphilis.github.io/ai-training/concepts/context-window/) (tutorial)
- [How a language model works](https://schubergphilis.github.io/ai-training/concepts/how-models-work/) (explanation)

## Sources

- `AEC-02` [How agents think: tokens and context, reasoning strategies, model choice, system prompts](https://lsimons.github.io/agent-engineer-course/02-how-agents-think/), Agent Engineer Course (course)
- `DLAI-2` [Generative AI for Everyone](https://www.deeplearning.ai/courses/generative-ai-for-everyone), DeepLearning.AI (course)
- `DLAI-6` [Introduction to Generative AI for Software Development](https://www.deeplearning.ai/courses/introduction-to-generative-ai-for-software-development), DeepLearning.AI (course)
- `Academy ai-capabilities-and-limitations` [AI capabilities and limitations](https://academy.claude.com/courses/ai-capabilities-and-limitations), Claude Academy (course)
- `Academy ai-fluency-framework-foundations` [AI Fluency: Framework and foundations](https://academy.claude.com/courses/ai-fluency-framework-foundations), Claude Academy (course)
- `Academy building-with-the-claude-api` [Building with the Claude API](https://academy.claude.com/courses/building-with-the-claude-api), Claude Academy (course)
- `Liu 2024` [Lost in the Middle: How Language Models Use Long Contexts](https://aclanthology.org/2024.tacl-1.9/), Transactions of the Association for Computational Linguistics 12 (2024), 157-173 (paper)