💻Tokens per Second
Also known as Tokens/s, Generation Speed, Inference Speed
How fast a model writes, counted in tokens each second.
Think of it like
Think of it like how fast a friend talks. Some talk slowly, some talk very fast.
Example
Lucy tries the same model on two machines. Her old laptop writes about 6 tokens per second, and her new desktop with a graphics card writes about 70.
Why it matters
It decides whether a local model feels like a quick chat or a slow wait.
Definition
Common mistake
Key takeaways
- 1Speed of writing. Tokens out each second.
- 2Memory speed rules. Weights are read each token.
- 3Reading is faster. A separate number.