# Temperature and Sampling

> How a model picks each token from a list of chances. What temperature, top-k, top-p and beam search change about that pick.

**Course:** [Large Language Models (LLMs)](https://learn.benibix.com/learn/ai-llms)  
**Time:** 12 min

**After this lesson you can:**

- Explain what temperature does to the pick
- Choose a setting for facts or for ideas
- Say why the same request can vary

## How a model picks the next token

After the model scores every possible next token, something has to choose one. That choice is called sampling. The model gives each token a chance, such as 60 in 100 for one word and 5 in 100 for another. Settings like temperature decide how often the less likely tokens get picked.

The simplest rule is to always take the top token. That is greedy decoding. It is fast, and the same request gets the same answer. It can also get stuck, saying one phrase again and again, because the top choice keeps leading back to itself.

Temperature reshapes the chances before the pick. Below 1, it makes the top tokens even more likely, so the text gets safer and plainer. At 1, the chances stay as the model gave them. Above 1, rare tokens get a bigger share, so the text gets more surprising. Near 2, it often stops making sense.

Two more settings trim the list before the pick. Top-k keeps only the k most likely tokens, such as the top 40. Top-p keeps the smallest group whose chances add up to a set share, such as 0.9. Top-p adapts: when the model is sure, the group is tiny, and when it is unsure, the group grows.

Beam search works another way. It keeps several part-written answers at once, often 4 or 5, and grows each of them. At the end it keeps the one that scored best overall. It suits jobs with one right output, such as translation. Chat models rarely use it, because it makes dull text.

Even at temperature 0, two runs can differ. A server handles many requests together. Tiny rounding in that maths can change which token wins when two are nearly tied. One different token early on sends the rest of the reply down a new path.

Some services also return logprobs: the model's chance for each token it wrote, stored as a log number. They are handy for spotting doubt. A low chance on a key word, like a name or a date, is a good sign to check that part.

People often raise the temperature to get better ideas. It adds no knowledge. It only picks rarer tokens from the same list. For facts, code and data, keep it low. For names, slogans and first drafts, where variety helps, a higher setting earns its place.

## The sampling settings side by side

| Setting | What it does | A common value |
| --- | --- | --- |
| Greedy decoding | Always takes the top token | Temperature 0 |
| Temperature | Sharpens or flattens the chances | 0 to 1 |
| Top-k | Keeps only the k most likely tokens | 40 |
| Top-p | Keeps tokens until their chances add up | 0.9 |
| Beam search | Grows several answers, keeps the best | 4 or 5 beams |

![Three bar charts of the same five next-token chances. At a low temperature of 0.3 one bar towers over the rest. At 1.0 the bars are as the model gave them. At a high temperature of 2.0 the bars are nearly level.](https://learn.benibix.com/diagrams/llm-temperature.svg)

### Example: Jack names a coffee shop

Jack asks a model for ten names for a coffee shop. At temperature 0.2, eight of the ten start with the same two words. At 1.0, all ten are different.

He keeps 1.0 for the naming step. For the step that writes the price list from his menu, he sets 0, because there he wants the same answer every time.

**Words in this lesson:** [LLM Sampling](https://learn.benibix.com/term/ai/sampling), [Greedy Decoding](https://learn.benibix.com/term/ai/greedy-decoding), [LLM Temperature](https://learn.benibix.com/term/ai/temperature), [Top-k Sampling](https://learn.benibix.com/term/ai/top-k-sampling), [Top-p Sampling](https://learn.benibix.com/term/ai/top-p-sampling), [Beam Search](https://learn.benibix.com/term/ai/beam-search), [LLM Non-Determinism](https://learn.benibix.com/term/ai/llm-non-determinism), [Log Probabilities (Logprobs)](https://learn.benibix.com/term/ai/logprobs)

> **The one thing to remember.** Sampling settings change which token gets picked, never what the model knows. Low for facts, higher for ideas.

**Question:** Does temperature 0 make an LLM give the same answer every time?

**Answer:** Nearly always, but server maths can still change it. Temperature 0 means take the top token. But two tokens can be nearly tied. Then tiny rounding on a busy server can flip the winner, and the reply goes a new way.

---

Source: https://learn.benibix.com/learn/ai-llms/temperature-and-sampling
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
