Skip to content
BENIBIX Learn

Speculative Decoding

AI💬 LLMsText Generation

Also known as Speculative Sampling, Draft Model Decoding

A small model guesses words ahead, and the big model checks them.

Think of it like

It is like a helper who writes a few words ahead. The teacher reads them and keeps the right ones.

Example

A chat service pairs its big model with a tiny one. On easy text, the big model accepts most guesses, so the answer streams out faster.

Why it matters

Big models are slow because they make one token at a time. This lets them confirm several in one go.

Definition

Speculative decoding makes a large language model write faster without changing what it writes. A small, fast draft model guesses the next few tokens. The big model checks all of those guesses in one pass, which takes about as long as making one token. It keeps the guesses it agrees with and fixes the first one it does not. A 2022 paper showed text coming out 2 to 3 times faster, word for word the same.
The draft model guesses four words. The big model keeps three and fixes the fourth in one pass.

Common mistake

Thinking the small model's words slip into the answer. The big model checks every token, so the text is the same.

Key takeaways

  1. 1
    Draft and check. Small guesses, big decides.
  2. 2
    Same output. Only the speed changes.
  3. 3
    Easy text gains most. More guesses are kept.

Learn it properly