Skip to content
BENIBIX Learn

Temperature and Sampling

Level 1 · What an LLM is

How a model picks each token from a list of chances. What temperature, top-k, top-p and beam search change about that pick.

Course contentsLesson 3 of 15

After this lesson you can

  • Explain what temperature does to the pick
  • Choose a setting for facts or for ideas
  • Say why the same request can vary

12 mincovers 8 glossary entries

Worth reading first: Tokens and Tokenizers

How a model picks the next token

After the model scores every possible next token, something has to choose one. That choice is called sampling. The model gives each token a chance, such as 60 in 100 for one word and 5 in 100 for another. Settings like temperature decide how often the less likely tokens get picked.

The simplest rule is to always take the top token. That is greedy decoding. It is fast, and the same request gets the same answer. It can also get stuck, saying one phrase again and again, because the top choice keeps leading back to itself.

Temperature reshapes the chances before the pick. Below 1, it makes the top tokens even more likely, so the text gets safer and plainer. At 1, the chances stay as the model gave them. Above 1, rare tokens get a bigger share, so the text gets more surprising. Near 2, it often stops making sense.

Two more settings trim the list before the pick. Top-k keeps only the k most likely tokens, such as the top 40. Top-p keeps the smallest group whose chances add up to a set share, such as 0.9. Top-p adapts: when the model is sure, the group is tiny, and when it is unsure, the group grows.

Beam search works another way. It keeps several part-written answers at once, often 4 or 5, and grows each of them. At the end it keeps the one that scored best overall. It suits jobs with one right output, such as translation. Chat models rarely use it, because it makes dull text.

Even at temperature 0, two runs can differ. A server handles many requests together. Tiny rounding in that maths can change which token wins when two are nearly tied. One different token early on sends the rest of the reply down a new path.

Some services also return logprobs: the model's chance for each token it wrote, stored as a log number. They are handy for spotting doubt. A low chance on a key word, like a name or a date, is a good sign to check that part.

People often raise the temperature to get better ideas. It adds no knowledge. It only picks rarer tokens from the same list. For facts, code and data, keep it low. For names, slogans and first drafts, where variety helps, a higher setting earns its place.

The sampling settings side by side

SettingWhat it doesA common value
Greedy decodingAlways takes the top tokenTemperature 0
TemperatureSharpens or flattens the chances0 to 1
Top-kKeeps only the k most likely tokens40
Top-pKeeps tokens until their chances add up0.9
Beam searchGrows several answers, keeps the best4 or 5 beams
Temperature does not change which token the model likes best. It changes how far ahead the favourite is.

Worked example

Jack names a coffee shop

Jack asks a model for ten names for a coffee shop. At temperature 0.2, eight of the ten start with the same two words. At 1.0, all ten are different.

He keeps 1.0 for the naming step. For the step that writes the price list from his menu, he sets 0, because there he wants the same answer every time.

The words for this lesson8

Check yourself

Does temperature 0 make an LLM give the same answer every time?

Finished this lesson?

Marking it complete also marks its 8 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.