💬Speculative Decoding
Also known as Speculative Sampling, Draft Model Decoding
A small model guesses words ahead, and the big model checks them.
Think of it like
It is like a helper who writes a few words ahead. The teacher reads them and keeps the right ones.
Example
A chat service pairs its big model with a tiny one. On easy text, the big model accepts most guesses, so the answer streams out faster.
Why it matters
Big models are slow because they make one token at a time. This lets them confirm several in one go.
Definition
Common mistake
Key takeaways
- 1Draft and check. Small guesses, big decides.
- 2Same output. Only the speed changes.
- 3Easy text gains most. More guesses are kept.