# Trying two models on the same task

> Run one reformatting task, one classification task and one ambiguous request on a small and a large model, time each answer, and read a six-cell table to see where the large model earns its price.

Page: https://schubergphilis.github.io/ai-training/using-agents/trying-two-models/
License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/

In this lesson we give the same three small tasks to two models and watch
what each one does with them. Most chat tools have a model picker with at
least two entries, a small one that answers fast and a large one that
answers slower and costs more per message. Many people leave the picker on
the largest model for everything, because "the best" sounds like the safe
choice. By the end of the lesson you have a six-cell table for these
three tasks. It shows which task the large model's extra price and delay
paid for, and you have a way to build the same table for a task of your
own.

You need a chat tool whose model picker offers a small and a large model.
Any pair works, and you don't have to know the names. Pick the smallest
and the largest, and call them "small" and "large" in your notes. The
answers shown on this page are illustrative. The author wrote them to show
the pattern, and your models give a different text.

## Three tasks, one prediction each

The inputs follow. Read each one and, before you run anything,
write down which of the two models you think it needs. That prediction is
the point of the section. Once you have run the tasks you can't recover
what you would have guessed.

The first is reformatting. The text goes in, the same facts come out as
a table, and nothing has to be decided.

#### Prompt (illustrative, not a recorded transcript)

```text
Turn these lines, in the order given, into a table with the columns
Room, Floor, Seats, and Screen.

Harbor room, second floor, seats 8, has a screen
Meadow room, ground floor, seats 4, no screen
Summit room, third floor, seats 12, has a screen
Quarry room, ground floor, seats 6, no screen
```

The second is classification. Each message gets one label from a fixed
list, and the list is short enough that the answer is either right or
wrong.

#### Prompt (illustrative, not a recorded transcript)

```text
Label each of these customer messages as billing, delivery, or other. Reply
with one line per message: the number and the label, nothing else.

1. I was charged twice for the same order last week.
2. The parcel says delivered but nothing has arrived.
3. Can I change the email address on my account?
4. The invoice shows the old company name.
5. Your driver left the box in the rain.
```

The third is an ambiguous request. The useful reply depends on facts the
message doesn't give, and a good answer either asks for them or lays out
how the advice changes with each one.

#### Prompt (illustrative, not a recorded transcript)

```text
Our team offsite is in six weeks and the budget was just cut by a third.
What should we do?
```

Write your three predictions down now. A common guess is "large for all
three, to be safe". Hold on to yours and see how it does.

#### Checkpoint: Name the work before the model

Match each request to what it asks the model to do.

Statements:

- Turn these four lines about rooms into a table with four columns.
- Label each of these five messages as billing, delivery or other.
- Our offsite budget was cut by a third. What should we do?
- What did our main competitor announce this week?

Options:

- Reformatting
- Classification
- Multi-step reasoning
- Fresh facts

## Run them on both models

Now run all three inputs on the small model, then on the large one. Use a
fresh chat for each run, so no earlier answer is in the context. For each
of the six answers, note two things: how long it took to finish, counted
on a clock or by feel, and whether the answer is good enough for the task.
"Good enough" means you would use it as it is. Write the six verdicts in a
table with a row per task and a column per model.

Here is what one run looked like for the author, with the small model's
answers first. This is one example run, written to show the pattern. It is
not a measurement of any model, and your table may differ.

#### Response

````text
```text
| Room   | Floor  | Seats | Screen |
|--------|--------|-------|--------|
| Harbor | Second | 8     | Yes    |
| Meadow | Ground | 4     | No     |
| Summit | Third  | 12    | Yes    |
| Quarry | Ground | 6     | No     |
```
````

The small model reformats in a second or two, and every cell matches the
input. The large model gives the same table a few seconds later. The
task didn't need the extra time.

#### Response

````text
```text
1. billing
2. delivery
3. other
4. billing
5. delivery
```
````

The same on classification. Both models label the five messages the same
way, and the small one is done first. The one place you might see a
difference is message 4, where "invoice" and "company name" pull toward
two labels. Both models in the example run chose billing, which is the
label the fixed list makes most useful.

#### Response

````text
```text
Here are some ways to handle a budget cut for a team offsite:

1. Shorten it. Two days instead of three cuts hotel and meal costs.
2. Move it closer. Travel is often the largest line item.
3. Go remote for part of it. Run the workshops online and meet in
   person for one day.
4. Cut the extras. Drop the outside speaker or the evening activity.
5. Ask for sponsorship. Some vendors will co-fund an event.

Would you like me to expand on any of these?
```
````

The small model's answer to the ambiguous request is a generic list. Every
item could apply to any offsite anywhere. The one fact in the message,
that a third of the budget is gone, appears in none of them. The answer
is fluent and beside the point. As you saw in the lesson on how models
work, the model produced the most likely text for a question like this,
one token at a time, and nothing in that process notices what the
question left out.

#### Response

````text
```text
Before choosing, it helps to know what the missing third was going to
pay for, because that decides which options are open.

- If the cut lands on travel, the offsite can move to a location most
  of the team can reach by train, or go hybrid with one in-person day.
- If it lands on venue and catering, a shorter event keeps the agenda
  and drops one night's hotel and two meals.
- If it lands on the program, the outside facilitator goes and a team
  member runs the sessions, which changes the tone more than the
  content.

Two questions would narrow it down: what was the original budget split,
and which parts of the offsite are the reason it exists at all? Protect
those parts and cut around them.
```
````

The large model's answer takes longer to arrive and is the one you could
act on. It names the missing fact, shows how the choice depends on it, and
asks the two questions whose answers pick the option. This matches what the
concept of model fit describes: a small model handles reformatting and
classification, and a larger one handles reasoning about an ambiguous
request (How agents think: tokens and context, reasoning strategies, model choice, system prompts, Agent Engineer Course). A run like this one is how you find out the fit. The
size of the model on its own tells you nothing about it.

The example run's table then reads like this. "Yes" means good enough
to use as it is.

| Task              | Small model | Large model |
| ----------------- | ----------- | ----------- |
| Reformatting      | Yes, fast   | Yes, slower |
| Classification    | Yes, fast   | Yes, slower |
| Ambiguous request | No          | Yes         |

#### Checkpoint: Where the large model earns its price

Of the three tasks in the run, which one justifies the large model, and why?

- The reformatting, because a large model is more careful with the cells of a table.
- The classification, because a large model knows the difference between billing and delivery.
- The ambiguous request, because the useful answer needed a fact the message left out.
- All three, because a large model is more accurate on every task and speed does not matter in a chat.

## Read the table at volume

For one chat, the difference between the two columns is a few seconds and
a price too small to notice on a monthly bill. That is why the picker set
to "large" feels free. The table starts to matter when the task repeats.

Suppose the classification task is real: your team gets five thousand
customer messages a day, and someone wants a model to label each one. From
your run, the small model labeled the five messages as well as the large
one. Now scale the other two columns. If the large model took four seconds
per answer where the small one took one, that is a difference of three
seconds a message, and three seconds times five thousand is more than four
hours of waiting a day when the messages run one after another. If the
large model costs several times the small
one per message, the cost gap is also multiplied by five thousand, every
day. Cost and speed, per call, become the whole budget of a repeated task,
so for a task that repeats you pick the lightest model that does each part
of it well (How agents think: tokens and context, reasoning strategies, model choice, system prompts, Agent Engineer Course).

So the order of work is the one you followed in this lesson. Say what the
task needs. Run a few real inputs on each candidate and fill the table.
Then read the table at the volume you expect, and measure before you
schedule anything repeated. The large model goes where the table says a
cell changed from no to yes, and nowhere else.

#### Checkpoint: Five thousand messages a day

A colleague wants to label the team's five thousand daily customer
messages with the large model, "because it is the best one". What do you
propose?

- Agree. The large model gives the best labels, and a few extra seconds per message is nothing.
- Run fifty real messages through both models first, compare the labels, and time each run before choosing.
- Use the small model without a test. It handled five messages in the lesson, so it will handle five thousand.
- Use the large model for the first week and switch to the small one once the labels look right.

#### Pitfall: Leaving the picker on the largest model

A team sets every tool to the largest model, because nobody wants to be
the one who chose a worse answer. Six months in, the largest model is
labeling tickets, reformatting exports and answering questions about the
office printer. A test on twenty tickets shows the small model gives the
same labels in a quarter of the time. Nobody had run the comparison,
because "the best" never felt like a decision. Treat the model pick as a
decision, and make it from a table you filled in.

## Exercise

Open your chat tool and find the model picker. Run the three inputs from
the first section on the small model, each in a fresh chat, and then on the
large model, again in fresh chats. For each of the six answers write down
how long it took and whether you would use it as it is. This takes about
ten minutes and leaves you with a six-cell table. A good result is a table
where you can point at each "no" and say what was missing from the answer,
and at each pair of "yes" cells and say how much longer the large model
took. Where did your predictions from the first section turn out wrong,
and what does that tell you about your default?

Stretch: Add a fourth input that asks about something that happened this month, run it on both models, and note whether either one says it cannot know, or whether the answer needs a model with a search tool instead.

## Recap

1. Say what a task needs before you name a model: reformatting, classification, multi-step reasoning, or fresh facts.
2. Find the model fit by running a few representative inputs on each candidate. The size of a model doesn't tell you its fit on your task.
3. A small model is often good enough for reformatting and classification, and a large one pays off on an ambiguous request. Fresh facts need a model with a search tool, whatever its size.
4. Cost and speed are per call. At the volume of a repeated task they become the whole budget, so measure before you schedule.
5. Leaving the picker on the largest model is a choice too, and the table is how you check it.

## References

- How agents think: tokens and context, reasoning strategies, model choice, system prompts, Agent Engineer Course, https://lsimons.github.io/agent-engineer-course/02-how-agents-think/