# Answers from documents the model never saw

> Read the output of a small keyword search over four policy texts, see it find the leave policy in the document's own words and miss it in a colleague's words, and learn what grounding promises and what it leaves for you to check.

Page: https://schubergphilis.github.io/ai-training/concepts/retrieval/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

A product assistant on a web shop is asked what the yearly plan costs,
and it answers with last year's price. The price went up in January,
and the model was trained before that. The obvious fix seems to be to retrain
the model on the new price list, which is slow, expensive, and out of
date again at the next price change. In this lesson we look at what
assistants do instead. A search step finds the passages that fit the
question and puts them in the prompt, and the model answers from what
it can see. You read the output of a small keyword search over four
policy texts. The same question is asked in the document's own words
and then in a colleague's words, and you judge what the search missed.
You install and run nothing.
By the end you can say what retrieval adds, what grounding promises, and
which two questions to ask of any assistant that answers from documents.

## Search first, then answer

A model knows what was in its training data and what's in its context
window right now. The [lesson on the agent
loop](https://schubergphilis.github.io/ai-training/concepts/agent-loop/#a-model-in-a-loop-with-two-tools) showed a tool that
reads a file into the conversation. Retrieval is a search that runs
before the model answers, so that the right text is already in the
prompt when the model reads the question (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). The whole approach
is called **retrieval (RAG)**, short for retrieval-augmented generation, and it
is how an assistant answers about your files or a company wiki without
anyone retraining anything.

In its basic form, retrieval has a fixed set of steps. The documents are
split into passages. When a question comes in, a search scores each
passage against the question and picks the best few. Those passages are
pasted into the prompt above the question, with an instruction such as
"answer from the passages below". The model then writes an answer from
text it can see. Update the document, and the next answer uses the new
text. The model never changed. Agentic retrieval, where the model itself
decides when to search and searches again after a poor result, builds on
these same steps (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course).

The search is the weak point. A model can only ground its answer in the
passages the search gave it. When the search returns the wrong passage, the model answers from the
wrong passage, and the answer is wrong (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). It reads as sure of
itself as a right answer does. So the first thing to know about any retrieval assistant is
how it searches.

## What a keyword search finds

The simplest search matches words. The question is split into words,
common words such as "how" and "the" are dropped, and each document is
scored by how many of the remaining words it contains. The best-scoring document
becomes the passage in the prompt. Products also use
similarity search, which scores meaning instead of exact words, on its
own or combined with keywords. The keyword version is the one you can
check by eye, so it is the one this lesson uses.

The policy texts are one paragraph each, on annual leave, expenses,
remote work, and laptops. Here is the search on a question that uses
the leave policy's own words. The output below is what the search
printed, line for line.

```text
question: How many days of annual leave do I get?
keywords: annual, days, leave
  expenses.txt: 1 match
  laptops.txt: 0 matches
  leave.txt: 3 matches
  remote-work.txt: 1 match
retrieved: leave.txt
passage: Annual leave. Every employee accrues 25 days of annual leave per year. Unused days carry over until 31 March. Requests go to your manager at least two weeks ahead.
```

Three keywords, and the leave policy contains all three. The expenses
and remote work policies each contain "days" and score one. The search
retrieves the leave policy, and a model that reads that passage answers
"25 days" and is right.

Now the same question in a colleague's words.

```text
question: How much holiday can I take?
keywords: holiday, take
  expenses.txt: 0 matches
  laptops.txt: 0 matches
  leave.txt: 0 matches
  remote-work.txt: 0 matches
retrieved: nothing, no document contains any of the keywords
```

The answer is in the leave policy, the same as before. The policy says
"annual leave" and the colleague said "holiday". The keyword search
scores every document at zero and retrieves nothing. What happens next
depends on the product. A careful assistant says the documents don't
cover the question. Without that care, the model has an empty prompt
and a question about holidays, and answers from training data, which
for a company policy means it makes something up.

#### Checkpoint: Why did the second search find nothing?

The first search retrieved the leave policy and the second retrieved
nothing, although both questions ask the same thing. Why?

- The model forgot the leave policy between the two questions.
- The question used words that appear in no document.
- The leave policy is about leave, and holidays are a different policy.
- Two keywords are too few for a search to work.

## Grounding, and what it doesn't fix

**Grounding** is tying an answer to the sources the model was given,
rather than to its training data. A grounded answer does two
things. It points at the passage it relied on, so you can read the
passage yourself. And it says so when the passages don't cover the
question (AI capabilities and limitations, Claude Academy).
The instruction "answer from the passages below, and say if they don't
answer the question" is what asks the model for both.

Grounding narrows what the model can get wrong. It doesn't remove it.
A grounded answer rests on the passage, and the passage can be the
wrong one, as the third search below shows. The model can also misread
a passage it was given, or answer a question the passage only half
covers by adding a detail from training. And a passage that
is itself out of date grounds an out-of-date answer. Grounding makes an
answer checkable, and checking is still yours to do.

Here is the third search. The question is about the daily allowance
for a meal on a work trip, which the expenses policy sets at 40 euros
per day.

```text
question: What is the daily rate for working away from the office?
keywords: away, daily, office, rate, working
  expenses.txt: 0 matches
  laptops.txt: 0 matches
  leave.txt: 0 matches
  remote-work.txt: 1 match
retrieved: remote-work.txt
passage: Remote work. Staff may work from home up to three days per week. A home office allowance of 30 euros per month is paid with salary. Equipment stays company property.
```

The expenses policy says "per day", and the question said "daily". With no shared word,
the expenses policy scores zero. The remote work policy contains
"office" and wins with one match. A model that grounds its answer in
this passage might say the daily rate is 30 euros, and cite the remote
work policy for it. The citation is real, the passage is real, and the
answer is wrong twice: the 30 euros is a monthly allowance, and it
covers a home office while the question asked about a work trip. A grounded wrong answer
looks the same as a grounded right one until you read the passage.

#### Checkpoint: The assistant quotes last year's price

The web shop's assistant answers from a folder of product documents.
A customer asks what the yearly plan costs and gets last year's price.
What do you do first?

- Ask the vendor to retrain the model on the new price list.
- Check the passage the answer was grounded in, and update that document.
- Add "always give the current price" to the assistant's instructions.
- Turn off retrieval so the model answers from what it knows.

#### Pitfall: Two clauses, one sentence

An assistant is asked what the company pays for working from home. The
search returns two passages, because the remote work policy and the
expenses policy both mention euros and both mention days. The answer
reads: "Staff may work from home up to three days per week, and are
reimbursed up to 40 euros per day with a receipt." Each half is a real
clause from a real policy. Together they say the company pays 40 euros
for every home working day, which no document says. The 40 euros is for
meals on a work trip. The answer cited both passages, and a reader who
trusted the citations without opening them would have filed a claim.
When an answer draws on more than one passage, read the sentence that
joins them, because the join is the model's and the sources don't check
it.

## Exercise

Read the third search on this page again, the one on the daily rate,
together with the four sentences of the expenses policy it should have
found: "Expenses. Travel by train is booked through the travel desk.
Meals on a work trip are reimbursed up to 40 euros per day with a
receipt. Claims are filed within 30 days." Then, on paper or in a note,
write three lines. First, which document holds the answer, and which
document the search retrieved. Second, which word in the question
would have needed to match, and what the document says instead. Third,
the answer a model would give from the retrieved passage, and what a
reader would have to open to notice it is wrong. Ten minutes is enough.

A good result names the expenses policy as the answer and the remote
work policy as what was retrieved, names "daily" against "per day" (or
"working away" against "work trip") as the miss, and says the model's
answer would be a 30 euro figure that a reader can only catch by
reading the passage and seeing "per month" and "home office". If your
own assistant answers from documents, what would you have to open to do
this same check on one of its answers?

Stretch: Rewrite the third question so that the keyword search would retrieve the expenses policy. Which words did you have to borrow from the document, and would the colleague who asked have known them?

## Recap

1. Retrieval is a search that runs before the model answers and pastes
   the best passages into the prompt. The model answers from text it
   can see, and the documents can change without the model changing
   (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course).
2. The search bounds the answer. A keyword search finds a document only
   when the question uses the document's words, and a wrong passage
   produces a wrong answer (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course).
3. A grounded answer points at its passage and says when the passages
   don't cover the question. Grounding makes an answer checkable and
   doesn't make it right: the passage can be wrong, old, or misread
   (AI capabilities and limitations, Claude Academy).
4. Ask two questions of any assistant that answers from documents. What
   did it retrieve, and does the answer match the passage.

## References

- Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/08-agentic-rag/
- AI capabilities and limitations, Claude Academy, https://academy.claude.com/courses/ai-capabilities-and-limitations