Skip to content
BENIBIX Learn

LLM Speed and Streaming

Level 4 · Choosing and running a model

The two numbers that decide how fast a model feels. How streaming, the KV cache and speculative decoding each cut the wait.

Course contentsLesson 13 of 15

After this lesson you can

  • Tell the first wait from the output speed
  • Explain what the KV cache saves
  • Cut the wait before the first word

12 mincovers 4 glossary entries

Worth reading first: Self-Attention and Multi-Head

What makes a model reply fast or slow

Two numbers describe how fast a model feels. Time to first token, or TTFT, is the wait before anything appears. Output speed is how many tokens per second follow. A reply of 500 tokens at 50 tokens a second takes 10 seconds to finish, whatever the first wait was.

Streaming sends each token to the screen as soon as it is made, instead of waiting for the whole reply. The total time is the same. But the reader starts reading after half a second, instead of staring at a blank box for 10. That is why chat apps type their answers out.

TTFT grows with the size of what you send. Before it writes anything, the model has to read your whole prompt. A short question starts fast. A document of 100,000 tokens can take many seconds before the first word appears.

The KV cache keeps writing fast after that. Each new token needs the attention work from every token before it. Without a cache, all of that would be done again for every token. The cache keeps the earlier work in memory, so each step only works out the new token.

The cache has a price. It grows with every token in the window and with every request served at once. For long inputs it can take more chip memory than the model itself. Much of the cost of a long context is the memory to hold this cache.

Speculative decoding speeds up the writing itself. A small, fast model guesses the next few tokens. The big model then checks all of them in one pass, which costs about the same as making one. Every guess it agrees with is kept for free. Its makers measured replies 2 to 3 times faster.

The output is exactly what the big model would have written alone. Any guess it disagrees with is thrown away, and it writes that token itself. So the trick changes the speed, never the answer. It works best on text that is easy to guess, like code or set phrases.

People blame the model when an app feels slow. Often the wait is the prompt: a huge system prompt, or a whole chat history sent each time. Trimming what you send cuts the first wait, and that is the part people notice most.

Both replies finish at the same time. The streamed one feels far faster, because reading starts at once.

Worked example

Emma cuts the first wait

Emma's help bot sends a 30,000-token manual with every question. The first word takes 6 seconds to appear.

She sends only the two pages that match the question instead, about 1,500 tokens. The first word now shows in under a second.

The words for this lesson4

Check yourself

What does streaming change about a model's reply?

Finished this lesson?

Marking it complete also marks its 4 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.