Top-k Retrieval vs Top-k Sampling
People see top-k in two settings and think it is one knob, and it is not. In retrieval, k is how many matching chunks the search hands to the model. In sampling, k is how many likely next words the model picks from as it writes. One decides what the model reads. The other shapes how it writes.
Top-k Retrieval
Returning only the k best matches from a search. You choose the k.
Top-k retrieval means the search returns only the k best matches. The k is a number you choose, like 5 or 20. Too few, and the needed fact may be left out. Too many, and the prompt fills with noise and costs more. Many systems fetch a larger k, then rerank and keep a smaller number. It is a different idea from top-k sampling, which picks words.
Top-k Sampling
Picking from only the top few tokens, like the best 40.
Top-k sampling limits the draw to the k most likely tokens and ignores the rest. With k set to 40, only the 40 top tokens can be picked at each step. It is simple and cuts off rare, odd tokens. But k is fixed: when the model is sure, 40 may let in poor choices, and when it is unsure, 40 may be too few. That is why top-p is often used instead or as well.
In one line
- Top-k Retrieval
- Returning only the k best matches from a search. You choose the k.
- Top-k Sampling
- Picking from only the top few tokens, like the best 40.