Context Window vs Max Tokens
People think max tokens is the size of the context window, and it is not. The context window is all the text a model can hold at once: what you send and what it writes. Max tokens is a cap you set on the reply alone. The reply has to fit in what is left of the window after your prompt.
Context Window
The most text, in tokens, a model can read at one time.
The context window is the most text a language model can take in at one time. It is counted in tokens. It holds all of one request: the instructions, the chat so far, any pasted files and the answer being written. Sizes run from a few thousand tokens in small models to over a million in some large ones. Text that does not fit is cut off. The model cannot see it.
Max Tokens
The most tokens a model may write in one reply. It stops at the cap.
Max tokens is a setting that caps how many tokens a model may write in one reply. When the reply hits the cap, it stops, even mid-sentence. It guards you from long, costly answers and from runs that never end. The cap counts only the output. But the input and output together must still fit in the context window.
In one line
- Context Window
- The most text, in tokens, a model can read at one time.
- Max Tokens
- The most tokens a model may write in one reply. It stops at the cap.