Top-k vs Top-p Sampling
People think top-k and top-p are the same cut, and they are not. Top-k always keeps a set number of tokens, such as 40. Top-p keeps tokens until their chances add up to a share, such as 0.9. So top-p keeps a few tokens when the model is sure, and many when it is not. Top-k keeps 40 either way.
Top-k Sampling
Picking from only the top few tokens, like the best 40.
Top-k sampling limits the draw to the k most likely tokens and ignores the rest. With k set to 40, only the 40 top tokens can be picked at each step. It is simple and cuts off rare, odd tokens. But k is fixed: when the model is sure, 40 may let in poor choices, and when it is unsure, 40 may be too few. That is why top-p is often used instead or as well.
Top-p Sampling
Picking only from the top tokens whose chances add up to a set share, like 0.9.
Top-p sampling limits the draw to the smallest group of likely tokens whose chances add up to p. With p set to 0.9, the model sorts tokens from most to least likely and keeps them until their chances reach 90 percent. The long tail of unlikely tokens is dropped. When the model is sure, few tokens are kept; when it is unsure, more are. It is also called nucleus sampling.
In one line
- Top-k Sampling
- Picking from only the top few tokens, like the best 40.
- Top-p Sampling
- Picking only from the top tokens whose chances add up to a set share, like 0.9.