Skip to content
BENIBIX Learn

Top-k Sampling

AI💬 LLMsText Generation

Also known as Top-k

Picking from only the top few tokens, like the best 40.

Think of it like

Think of it like you can only pick one of the top three cakes.

Example

Jack sets top-k to 3. For 'I drink', only 'tea', 'coffee' and 'water' can be picked; 'lava' is never on the list.

Why it matters

It is the simplest guard against a model picking a strange word now and then.

Definition

Top-k sampling limits the draw to the k most likely tokens and ignores the rest. With k set to 40, only the 40 top tokens can be picked at each step. It is simple and cuts off rare, odd tokens. But k is fixed: when the model is sure, 40 may let in poor choices, and when it is unsure, 40 may be too few. That is why top-p is often used instead or as well.
Top-k, on the left, always keeps the same number of bars, here the top three.

Common mistake

Setting k very high and thinking it does anything. With k at 1,000, almost no tokens are cut.

Key takeaways

  1. 1
    Keep the top k. Like 40 tokens.
  2. 2
    Fixed size. Same count every step.
  3. 3
    Simple guard. Cuts rare, odd words.

Compare

Learn it properly