Skip to content
BENIBIX Learn

LLM Sampling

AI💬 LLMsText Generation

Also known as Decoding, Token Sampling, Sampling

How a model picks the next token from its list of chances.

Think of it like

Think of it like a bag with lots of word cards. You pull one out, and likely words have more cards.

Example

Lucy asks for a story opener three times and gets three different first lines. Each time, sampling drew a different token at the start.

Why it matters

It explains why answers change from run to run. It also shows which settings make them steady or varied.

Definition

Sampling is how a language model picks the next token from the chances it gave. Always taking the most likely token gives safe but dull and repetitive text. Picking at random, weighted by the chances, gives more varied text. Settings like temperature, top-k and top-p shape that draw. This is why the same question can get a different answer each time.
Sampling draws from bars like these. The settings change their shape before the draw.

Common mistake

Thinking a new answer to the same question means the model changed. It is the same model drawing a different token.

Key takeaways

  1. 1
    Pick from the chances. One token each step.
  2. 2
    Weighted draw. Likely tokens win more.
  3. 3
    Settings shape it. Temperature, top-k, top-p.

Learn it properly