# Retrieval as a tool the agent calls

> Build a keyword search as a tool, run it once per question and then in a bounded loop against a twelve-document handbook, and use a six-question test set to decide which version a question needs.

Page: https://schubergphilis.github.io/ai-training/building-agents/retrieval-as-a-tool/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson the agent gets a tool that looks things up. The concepts
course showed retrieval as a search that runs before the model answers,
so that the right passage is in the prompt when the model reads the
question, and it showed what a keyword search gets wrong. Here we build
that search as a tool, run it two ways against a twelve-document company
handbook, and measure each way against a small test set. The basic
version calls the tool once per question. The loop version lets the
agent call it, read the result, judge it and call it again, and the
lesson ends with the rule for choosing between them.

Every example runs against a fixture with a fake model, so you can
predict the output and check yourself. The complete program is
`site/examples/building-agents/retrieval-as-a-tool/agent.py` in the
repository, with the twelve texts under `docs/` next to it. Copy the
folder, and each step below is `python3 agent.py <step>`. The fake model
answers with the sentence of the retrieved passage that shares the most
keywords with the question. That's a deterministic replacement for a
real model reading the passage, and it makes every run print the same lines. The fixture reads
its documents with a glob over `docs/*.txt`, and a stray file in the
folder doesn't change the output.

## The search is a tool

A tool is a function plus a description, as
[Building your first agent](https://schubergphilis.github.io/ai-training/building-agents/agent-loop/) showed. The
search here is one more entry in the same `TOOLS` dictionary. It splits
the query into keywords, drops common words, scores each document by how
many keywords it contains, and returns the best one. The `skip` argument
names documents already read. A second search with `skip` set finds a
different document.

```python
def search_docs(query: str, skip: Optional[list[str]] = None) -> dict:
    terms = keywords(query)
    scores = {
        name: sum(1 for term in terms if term in set(words(text)))
        for name, text in CORPUS.items()
        if name not in (skip or [])
    }
    if not scores:  # every document was skipped, or docs/ is empty
        return {"ok": True, "name": None, "passage": ""}
    best = max(scores, key=lambda name: scores[name])  # the first document wins a tie
    if scores[best] == 0:
        return {"ok": True, "name": None, "passage": ""}
    return {"ok": True, "name": best, "passage": CORPUS[best]}


TOOLS = {
    "search_docs": {
        "fn": search_docs,
        "description": (
            "Search the handbook. Returns the best-matching passage. "
            "Args: query (str), skip (list of str, optional): names of documents already read."
        ),
    },
}
```

With a real model, the description is what the model reads to decide
when to call the tool, and the passage goes back to the model as a
`tool_result` block in the next user message (Define tools, Claude Platform documentation)
(Handle tool calls, Claude Platform documentation). In this fixture the passage goes into a
prompt built by `build_prompt`, with a frame above it, and the question
below it. The frame tells the model where the text came from and what to
do when it doesn't answer the question, and the last section of this
lesson shows what happens without it.

#### Example: The prompt the model gets

Run this, and compare what you see with the output below.

```python
result = TOOLS["search_docs"]["fn"]("How many days of annual leave do I get?")
print(build_prompt(result["passage"], "How many days of annual leave do I get?"))
```

Output:

```text
The passages below were retrieved from the company handbook for this question. Answer from them. Quote the sentence you relied on. If the passages do not cover the question, answer exactly: not found.

Passages:
Annual leave. Every employee accrues 25 days of annual leave per year. Unused days carry over until 31 March. Requests go to your manager at least two weeks ahead.

Question: How many days of annual leave do I get?
```

## Basic retrieval against a test set

The basic version is the fixed pipeline from the concepts course: search
once, put the passage in the prompt, ask (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). The harness calls the
tool before the model sees the question, and the model doesn't decide
anything about retrieval.

```python
def answer_basic(question: str) -> dict:
    result = TOOLS["search_docs"]["fn"](question)
    reply = fake_model(build_prompt(result["passage"], question))
    return {"name": result["name"], "answer": reply, "rounds": 1}
```

A test set decides whether this is good enough. This one has six
questions in the handbook's own words, except the second, which asks
about holiday where the handbook says leave. The concepts course showed
what a keyword search does with that.

```python
TEST_SET = [
    "How many days of annual leave do I get?",
    "How much holiday can I take?",
    "What is the training budget per year?",
    "How many days per week can I work from home?",
    "How many days ahead are flights booked?",
    "Who do I report a lost laptop to?",
]
```

The twelve handbook files are `data-team.txt`, `expenses.txt`,
`laptops.txt`, `leave.txt`, `office-hours.txt`, `onboarding.txt`,
`payments-team.txt`, `remote-work.txt`, `security.txt`,
`service-desk.txt`, `training.txt` and `travel.txt`. Each file holds a
few sentences on the topic its name gives. The `list_sources` function
runs the basic version on each question. For each question it prints a
line in the form `<n>. <file>`, where the file is the source of the
answer. A question without an answer gets `<n>. not found`. The last line
is `answered: <count> of 6`.

#### Checkpoint: Which questions does the basic version answer?

Predict the output first, one `<n>. <file>` or `<n>. not found` line
per question and the `answered: <count> of 6` line. Then run this.

```python
list_sources(TEST_SET)
```

The `run_test_set` function prints each question with the file it
retrieved, or `nothing retrieved`, and its answer, and the same count at
the end.

#### Example: The answers of the basic version

Run this, and compare what you see with the output below.

```python
run_test_set(TEST_SET)
```

Output:

```text
1. How many days of annual leave do I get?
   leave.txt: Every employee accrues 25 days of annual leave per year.
2. How much holiday can I take?
   nothing retrieved: not found
3. What is the training budget per year?
   training.txt: Every employee has a training budget of 1500 euros per year.
4. How many days per week can I work from home?
   remote-work.txt: Staff may work from home up to three days per week.
5. How many days ahead are flights booked?
   travel.txt: Flights are booked through the travel desk at least 14 days ahead.
6. Who do I report a lost laptop to?
   laptops.txt: Report a lost laptop to the service desk the same day.
answered: 5 of 6
```

Five of six, and the miss is the question in a colleague's words. A
better index fixes that one: a similarity search that scores meaning, or
a synonym list that maps holiday to leave. Nothing about the loop is
needed for it, and the next section is about the one failure a better
index doesn't fix.

## One question that needs two searches

Question seven asks who approves a replacement laptop for the payments
team. The laptops text says the cost center owner of the team approves
it and sends the reader to the team pages, and the payments team's text
names its cost center owner. The answer is in two documents, and the
basic version reads one.

#### Example: The basic version on question seven

Run this, and compare what you see with the output below.

```python
show(7, MULTI_HOP, answer_basic(MULTI_HOP))
```

Output:

```text
7. Who approves a replacement laptop for the payments team?
   laptops.txt: The cost center owner of your team approves a replacement laptop before that, see the team pages.
```

The answer is a pointer, and the person asking still has no name.
Retrieval Augmented Generation (RAG) is the full name behind the letters
in the **agentic RAG loop**, and the loop puts the agent in charge of the
search: it
forms a query, reads what comes back, judges whether it answers the
question, and searches again with a new query when it doesn't (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course).
Question seven is a multi-hop question, where the first result says what
to search for next, and that's the case the loop is for.

```python
def answer_with_loop(question: str, max_rounds: int = 3, quiet: bool = False) -> dict:
    query = question
    read: list[str] = []
    for round_number in range(1, max_rounds + 1):
        result = TOOLS["search_docs"]["fn"](query, skip=read)
        if result["name"] is None:
            return {"name": None, "answer": "not found", "rounds": round_number}
        read.append(result["name"])
        sentence = best_sentence(query, result["passage"])
        if not quiet:
            print(f"round {round_number}: search_docs(query={query!r}) -> {result['name']}")
        if is_pointer(sentence):
            # Follow the pointer: keep the question's words and add the sentence's.
            query = " ".join(keywords(question + " " + sentence))
            continue
        if covers(query, sentence):
            return {"name": result["name"], "answer": sentence, "rounds": round_number}
        query = question
    return {"name": None, "answer": "not found", "rounds": max_rounds}
```

The judging step is where a real model reads the passage and decides. In
the fixture, `is_pointer` is that judgment written as a rule: a sentence
that says "see" another page is a pointer, and the agent builds the next
query from the question's keywords and the pointer's. Documents already
read are skipped. The second search returns a document the loop hasn't
read. `max_rounds` bounds the loop, for the same reason every loop in
this course has a step limit.

#### Checkpoint: Predict the search rounds

Predict the number on the `rounds` line, and write the line as the
fixture prints it. With `quiet=True` the loop doesn't print its round
lines, so the `rounds` line is the whole output. Then run it.

```python
result = answer_with_loop(MULTI_HOP, quiet=True)
print(f"rounds: {result['rounds']}")
```

The loop takes two rounds. Call `answer_with_loop(MULTI_HOP)` without
`quiet` and it prints a line per round. The second line shows the second
query, `approves center cost laptop owner pages payments replacement team`,
which is the question's keywords plus the pointer's. The data team's text
also names a cost center owner, and it scores four on that second query
against five for the payments team's text. The word `payments`, carried
over from the question, is the difference. A real loop costs a model call
per round on top of the search, so this question costs about twice what a
basic question does.

## A question the handbook doesn't answer

Question eight asks whether you can bring a dog to the office. No text
covers it, but two texts contain the word `office`. The search
returns one of them, and the model gets a passage about opening hours.
A model that isn't told to ground its answer can answer from a passage
that doesn't cover the question, and the fix is an instruction that tells it to
base the answer on the retrieved text (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). The fixture's `fake_model`
at `min_shared=1` behaves like the model without that instruction: any
overlap between question and passage becomes an answer.

#### Example: Without the frame, and with it

Run this, and compare what you see with the output below.

```python
print("without the frame:")
show(8, NO_ANSWER, answer_basic(NO_ANSWER, min_shared=1))
print("with the frame:")
show(8, NO_ANSWER, answer_basic(NO_ANSWER))
```

Output:

```text
without the frame:
8. Can I bring my dog to the office?
   office-hours.txt: Core hours, when everyone in the office is reachable, are 10 to 15.
with the frame:
8. Can I bring my dog to the office?
   office-hours.txt: not found
```

The first answer is a sentence about core hours, offered as the dog
policy. The frame is the string `FRAME` at the top of the fixture. It
says the passages were retrieved for this question, asks the model to
quote the sentence it relied on, and names the answer to give when the
passages don't cover the question: `not found`. A real model reads that
text. The fake model obeys it through `covers`, which asks for at least
`MIN_SHARED` keywords of the question in the answering sentence, and
that check is what turns the second run into `not found`. An agent that
fails to find an answer should say that it couldn't find one (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). The way
to know yours does is a test set with questions like this one, whose
answer is in no document, with `not found` as the expected answer.

#### Checkpoint: What does the frame add?

The same passage was retrieved twice and the answers differ. What did
the frame change?

- It makes the search return the right document more often.
- It tells the model where the text came from and what to say when it does not cover the question.
- It stops the model from quoting the passage, so the answer is in its own words.

## When basic retrieval is enough

Basic retrieval is the right first version. Search once, put the top
passages in the prompt, answer, and it handles direct questions whose
answer is in one document, at one model call per question (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). The
loop costs a model call per round plus the search and needs a bound, and on
the five questions the basic version already answered it would have
given the same five answers for more money. Add the loop when a test
set shows the basic version failing on a question a better index can't
fix, as question seven did, and not before.

That order matters because the test set is what tells the two failures
apart. Question two failed because of the index, and a synonym or a
similarity search fixes it. Question seven failed because the answer is
in two places, which a second search fixes and a better index doesn't.
Question eight
didn't fail at the search step at all, and the frame fixes it in both
versions. Without the six questions, every failure looks like a reason
to add the loop.

#### Checkpoint: Which failure calls for the loop?

A team is deciding whether to add the retrieval loop. Which of these
failures on their test set is the reason to add it?

- The question uses a word that no document contains, so the search returns nothing.
- No document answers the question, and the model answers from a passage that shares one word with it.
- The first document read says who decides in general and points at a second document for the name.

#### Pitfall: A retrieval system that answers every question

A team ships a handbook assistant and reports that it answered every
question in the demo. The test set had only questions with an answer in
the handbook, so nobody saw what it does with the other kind. The first
week, someone asks about a policy the company doesn't have, the search
returns the closest passage, and the assistant states a policy that
exists nowhere. A retrieval system that answers everything is one that
invents answers when the documents have none. Put questions with no
answer in the test set, expect `not found`, and frame the retrieved text
so the model knows what to do when it doesn't cover the question.

## Exercise

Copy the `retrieval-as-a-tool` folder from the repository. Add the
question "How many days of parental leave do I get?" to `TEST_SET`, run
`python3 agent.py basic_answers`, and read what the basic version answers. The
handbook has no parental leave policy. The leave text shares two
keywords with the question, and two is what `covers` asks for. That's enough for the
answer to go through. Then raise `MIN_SHARED` to 3 and run the step again. Doing
this once shows you how a wrong passage clears a weak coverage check,
and what a stricter one costs.

A good result: before the change, the new question gets the annual leave
sentence as its answer and the count says 6 of 7. After the change, it
gets `not found`, the count says 5 of 7, and the other six lines are the
same as before. Run `python3 agent.py multi_hop_loop` as well and check
that question seven still takes two rounds. How many of the six original
questions would a threshold of 4 break, and what does that say about
thresholds as a replacement for a model's judgment?

Stretch: Add the question 'How many days of unpaid leave can I take per year?' and run python3 agent.py basic_answers again. It shares three keywords with the leave text and still gets the annual leave sentence. Write down what a real model needs from the frame to answer not found here, and why a keyword count can't do it.

## Recap

1. Retrieval is a tool like any other in the `TOOLS` dictionary. The
   basic version has the harness call it once before the model, and the
   agentic version lets the model call it, judge the result, and call it
   again (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course).
2. Start with basic retrieval and a test set. Add the loop only for the
   failures a test set shows and a better index can't fix, such as an
   answer spread over two documents (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course).
3. Frame retrieved text: say where it came from, ask for the sentence
   relied on, and name the answer to give when it doesn't cover the
   question. A model without a grounding instruction can answer from a
   passage that doesn't cover the question (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course).
4. An agent that doesn't find an answer says so (Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course). A test set for
   retrieval holds questions with no answer in the documents, and the
   expected answer for them is `not found`.

## References

- Define tools, Claude Platform documentation, https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools
- Handle tool calls, Claude Platform documentation, https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls
- Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/08-agentic-rag/