# Explains how a language model produces text and where it fails

> Competency concepts/explains-models

Page: https://schubergphilis.github.io/ai-training/competencies/concepts/explains-models/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

*Concepts* · competency `concepts/explains-models`

**Taught in:** the [Concepts](https://schubergphilis.github.io/ai-training/concepts/) course

**Draws on:** [How language models work](https://schubergphilis.github.io/ai-training/topics/concepts/how-models-work/), [Capabilities and limits](https://schubergphilis.github.io/ai-training/topics/concepts/limits/), [Grounding and memory](https://schubergphilis.github.io/ai-training/topics/concepts/grounding/)

## Learning objectives

### Explains tokens, context and sampling in plain words (base)

| Claim | Why | Example |
| ----- | --- | ------- |
| A model produces text one token at a time, each chosen from a probability distribution over what could come next. | Once the learner sees output as a chain of likely next pieces, both fluent nonsense and surprising variation stop being mysterious. | Asked why the same question gave two different answers, the learner says the model picked a different likely token early on and the rest followed from it, rather than saying the model "changed its mind". |
| The context window is the whole of what the model can see for one response, and anything outside it does not exist for the model. | Most "it forgot" and "it ignored my file" complaints are context problems, and the fix is different from a prompting problem. | When a long chat starts contradicting itself, the learner suspects earlier messages fell out of the window and restates the key facts instead of scolding the model. |
| Temperature and sampling settings trade variety for predictability and do not change what the model knows. | Learners otherwise reach for temperature to fix accuracy, which it cannot do. | For a summary that must be the same on every run the learner asks for low temperature. For brainstorming names they raise it, and in neither case expect it to add facts. |

Served by: [What the model can see](https://schubergphilis.github.io/ai-training/concepts/context-window/), [How a language model works](https://schubergphilis.github.io/ai-training/concepts/how-models-work/), [Where memory comes from](https://schubergphilis.github.io/ai-training/concepts/memory/), [Same prompt, different answer](https://schubergphilis.github.io/ai-training/concepts/same-prompt-twice/), [What every token costs](https://schubergphilis.github.io/ai-training/concepts/what-tokens-cost/)

### Names the common ways output goes wrong and why (base)

| Claim | Why | Example |
| ----- | --- | ------- |
| A hallucination is a fluent, confident statement with nothing behind it, and fluency is no signal of truth. | Confident wording is exactly what the training rewards, so the learner must stop using tone as a check. | Given a citation with a real-looking journal name, author and year, the learner looks it up before trusting it and is not surprised when it does not exist. |
| The knowledge cutoff means the model's built-in facts stop at a date, and anything after that must be supplied or retrieved. | Questions about recent events or versions get plausible but stale answers unless the learner knows to bring the facts. | Asking for the current version of a library, the learner pastes the changelog or asks the tool to look it up rather than accepting the number the model recalls. |
| Sycophancy is the model agreeing with the person in front of it, so the learner's own framing shapes the answer. | A leading question produces a leading answer, which feels like confirmation and is not. | Instead of "this plan is solid, right?" the learner asks "what are the three strongest objections to this plan?" and compares the two responses. |

Served by: [How a language model works](https://schubergphilis.github.io/ai-training/concepts/how-models-work/), [Same prompt, different answer](https://schubergphilis.github.io/ai-training/concepts/same-prompt-twice/), [Getting a straight answer](https://schubergphilis.github.io/ai-training/concepts/straight-answer/), [Where a model makes things up](https://schubergphilis.github.io/ai-training/concepts/thin-path/), [Wrong about the world, or wrong about the text](https://schubergphilis.github.io/ai-training/concepts/wrong-about-the-world/)

### Explains what grounding and retrieval add and what they do not fix (base)

| Claim | Why | Example |
| ----- | --- | ------- |
| Retrieval puts relevant documents into the context window at question time, so the model answers from text it can see rather than from what it memorized. | Learners otherwise expect a model "trained on our documents", and the difference decides what a stale or missing document does to the answer. | When a product assistant gives an outdated price, the learner asks which document it retrieved rather than asking for the model to be retrained. |
| Grounding reduces invented facts about the retrieved material but does not stop the model from misreading it or filling gaps outside it. | A grounded system still produces fluent text, and a wrong summary of a real document looks the same as a right one. | The learner checks the cited passage against the answer and finds the model combined two clauses from different contracts into one. |
| Retrieval finds text that looks similar to the question, so a question that uses other words than the document may miss it, and keyword search misses it most often. | Knowing this turns "the assistant does not know our policy" into "rephrase or fix the index", which is a fixable problem. | The assistant says it has no travel policy, so the learner asks about "expense rules for trips", gets the document, and reports the naming gap. |

Served by: [Where memory comes from](https://schubergphilis.github.io/ai-training/concepts/memory/), [Answers from documents the model never saw](https://schubergphilis.github.io/ai-training/concepts/retrieval/)

## Alignment

| Framework | Code | Asks | Objectives here |
| --------- | ---- | ---- | --------------- |
| AI Fluency 4D (Dakan and Feller) | Discernment | Judge the output, the process and the behavior of the AI critically | names-failure-modes |