Temperature and Sampling
Level 1 · What an LLM is
How a model picks each token from a list of chances. What temperature, top-k, top-p and beam search change about that pick.
Course contentsLesson 3 of 15
Level 1 · What an LLM is
Level 2 · Inside the model
Level 3 · How a model is trained
Level 4 · Choosing and running a model
Level 5 · Where LLMs go wrong
After this lesson you can
- Explain what temperature does to the pick
- Choose a setting for facts or for ideas
- Say why the same request can vary
12 mincovers 8 glossary entries
Worth reading first: Tokens and Tokenizers
How a model picks the next token
After the model scores every possible next token, something has to choose one. That choice is called sampling. The model gives each token a chance, such as 60 in 100 for one word and 5 in 100 for another. Settings like temperature decide how often the less likely tokens get picked.
The simplest rule is to always take the top token. That is greedy decoding. It is fast, and the same request gets the same answer. It can also get stuck, saying one phrase again and again, because the top choice keeps leading back to itself.
Temperature reshapes the chances before the pick. Below 1, it makes the top tokens even more likely, so the text gets safer and plainer. At 1, the chances stay as the model gave them. Above 1, rare tokens get a bigger share, so the text gets more surprising. Near 2, it often stops making sense.
Two more settings trim the list before the pick. Top-k keeps only the k most likely tokens, such as the top 40. Top-p keeps the smallest group whose chances add up to a set share, such as 0.9. Top-p adapts: when the model is sure, the group is tiny, and when it is unsure, the group grows.
Beam search works another way. It keeps several part-written answers at once, often 4 or 5, and grows each of them. At the end it keeps the one that scored best overall. It suits jobs with one right output, such as translation. Chat models rarely use it, because it makes dull text.
Even at temperature 0, two runs can differ. A server handles many requests together. Tiny rounding in that maths can change which token wins when two are nearly tied. One different token early on sends the rest of the reply down a new path.
Some services also return logprobs: the model's chance for each token it wrote, stored as a log number. They are handy for spotting doubt. A low chance on a key word, like a name or a date, is a good sign to check that part.
People often raise the temperature to get better ideas. It adds no knowledge. It only picks rarer tokens from the same list. For facts, code and data, keep it low. For names, slogans and first drafts, where variety helps, a higher setting earns its place.
The sampling settings side by side
| Setting | What it does | A common value |
|---|---|---|
| Greedy decoding | Always takes the top token | Temperature 0 |
| Temperature | Sharpens or flattens the chances | 0 to 1 |
| Top-k | Keeps only the k most likely tokens | 40 |
| Top-p | Keeps tokens until their chances add up | 0.9 |
| Beam search | Grows several answers, keeps the best | 4 or 5 beams |
Worked example
Jack names a coffee shop
Jack asks a model for ten names for a coffee shop. At temperature 0.2, eight of the ten start with the same two words. At 1.0, all ten are different.
He keeps 1.0 for the naming step. For the step that writes the price list from his menu, he sets 0, because there he wants the same answer every time.
The words for this lesson8
Check yourself
Does temperature 0 make an LLM give the same answer every time?
Finished this lesson?
Marking it complete also marks its 8 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.