Skip to content
BENIBIX Learn

Model Selection and Testing

Level 4 · Prompts inside an app

How to pick the AI model that runs a prompt. Build a small test set from real examples, then choose the smallest model that passes it.

Course contentsLesson 12 of 16

After this lesson you can

  • Build a test set from real examples
  • Compare models and prompts on that set
  • Pick the smallest model that passes

12 mincovers 4 glossary entries

Worth reading first: Prompt Caching and Cost

How to choose a model for a prompt

Model selection is choosing which AI model runs a prompt. Makers sell several sizes. Large models handle hard work best, but they are slower and cost more per token. Small ones are fast and cheap. The rule is simple: test a few on real examples, then pick the smallest one that passes.

A frontier model is one of the biggest and newest models from a top lab. Try it when a task is hard, such as careful writing, tricky logic or a long document. A small language model is far smaller inside. It is often good enough to sort, tag and pull out facts.

Do not choose from a leaderboard alone. A model that tops a public test can still stumble on your task, in your format, with your users. The only test that counts is one built from your own examples.

Build that test once. Collect 30 to 50 real inputs, with what a good answer looks like for each. Include the hard and odd cases, not just the easy ones. This set turns prompt engineering from guessing into measuring.

Run every model and every prompt change against the same set. Score the answers by hand or with code. Now the new prompt is better is a number. A change that fixes three cases and breaks five is caught before users see it.

Prompts do not always move between models. A prompt tuned on one model can do worse on another, even from the same maker. When you switch models, run the test set again, and expect to change some wording.

Price and speed decide the rest. If two models both pass, pick the cheaper and faster one. Some apps send simple questions to a small model and hard ones to a big one. That only works if you have tested both on the kind of question each will get.

Test on your own examples, then pick the smallest model that gets them right.

Worked example

Lucy sorts support emails

Lucy's app sorts support emails into five groups. She tests three models on 50 real emails. The small one gets 41 right, and the medium one gets all 50.

The large one also gets all 50, but it is slower and costs more. She picks the medium one, and keeps the 50 emails to test the next model that comes out.

The words for this lesson4

Check yourself

How should you choose an AI model for a task?

Finished this lesson?

Marking it complete also marks its 4 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.