Model Selection and Testing
Level 4 · Prompts inside an app
How to pick the AI model that runs a prompt. Build a small test set from real examples, then choose the smallest model that passes it.
Course contentsLesson 12 of 16
Level 1 · What a prompt is
Level 2 · Writing a better prompt
Level 3 · Harder and bigger tasks
Level 4 · Prompts inside an app
After this lesson you can
- Build a test set from real examples
- Compare models and prompts on that set
- Pick the smallest model that passes
12 mincovers 4 glossary entries
Worth reading first: Prompt Caching and Cost
How to choose a model for a prompt
Model selection is choosing which AI model runs a prompt. Makers sell several sizes. Large models handle hard work best, but they are slower and cost more per token. Small ones are fast and cheap. The rule is simple: test a few on real examples, then pick the smallest one that passes.
A frontier model is one of the biggest and newest models from a top lab. Try it when a task is hard, such as careful writing, tricky logic or a long document. A small language model is far smaller inside. It is often good enough to sort, tag and pull out facts.
Do not choose from a leaderboard alone. A model that tops a public test can still stumble on your task, in your format, with your users. The only test that counts is one built from your own examples.
Build that test once. Collect 30 to 50 real inputs, with what a good answer looks like for each. Include the hard and odd cases, not just the easy ones. This set turns prompt engineering from guessing into measuring.
Run every model and every prompt change against the same set. Score the answers by hand or with code. Now the new prompt is better is a number. A change that fixes three cases and breaks five is caught before users see it.
Prompts do not always move between models. A prompt tuned on one model can do worse on another, even from the same maker. When you switch models, run the test set again, and expect to change some wording.
Price and speed decide the rest. If two models both pass, pick the cheaper and faster one. Some apps send simple questions to a small model and hard ones to a big one. That only works if you have tested both on the kind of question each will get.
Worked example
Lucy sorts support emails
Lucy's app sorts support emails into five groups. She tests three models on 50 real emails. The small one gets 41 right, and the medium one gets all 50.
The large one also gets all 50, but it is slower and costs more. She picks the medium one, and keeps the 50 emails to test the next model that comes out.
The words for this lesson4
Check yourself
How should you choose an AI model for a task?
Finished this lesson?
Marking it complete also marks its 4 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.