Skip to content
BENIBIX Learn

Top-k vs Top-p Sampling

People think top-k and top-p are the same cut, and they are not. Top-k always keeps a set number of tokens, such as 40. Top-p keeps tokens until their chances add up to a share, such as 0.9. So top-p keeps a few tokens when the model is sure, and many when it is not. Top-k keeps 40 either way.

AI💬 LLMs

Top-k Sampling

Picking from only the top few tokens, like the best 40.

Top-k sampling limits the draw to the k most likely tokens and ignores the rest. With k set to 40, only the 40 top tokens can be picked at each step. It is simple and cuts off rare, odd tokens. But k is fixed: when the model is sure, 40 may let in poor choices, and when it is unsure, 40 may be too few. That is why top-p is often used instead or as well.

Example
Jack sets top-k to 3. For 'I drink', only 'tea', 'coffee' and 'water' can be picked; 'lava' is never on the list.
Why it matters
It is the simplest guard against a model picking a strange word now and then.
Full entry
AI💬 LLMs

Top-p Sampling

Picking only from the top tokens whose chances add up to a set share, like 0.9.

Top-p sampling limits the draw to the smallest group of likely tokens whose chances add up to p. With p set to 0.9, the model sorts tokens from most to least likely and keeps them until their chances reach 90 percent. The long tail of unlikely tokens is dropped. When the model is sure, few tokens are kept; when it is unsure, more are. It is also called nucleus sampling.

Example
For 'The sky is', the top token 'blue' has 0.85 and 'clear' has 0.07. With top-p at 0.9, only those two are kept, and every odd word is out.
Why it matters
It cuts off nonsense words while still allowing variety. That is why many tools turn it on by default.
Full entry

In one line

Top-k Sampling
Picking from only the top few tokens, like the best 40.
Top-p Sampling
Picking only from the top tokens whose chances add up to a set share, like 0.9.