💬Model Collapse
Also known as AI Model Collapse, Recursive Training Collapse
Models trained on other models' text get worse each round.
Think of it like
It is like a copy of a copy of a copy. Each one is a bit more blurry.
Example
Researchers train each new model only on text from the one before. After a few rounds, the rare words and odd facts are gone, and all the text sounds alike.
Why it matters
More and more of the web is written by AI. Future models trained on it could take in this slow decay.
Definition
Common mistake
Key takeaways
- 1Copies of copies. Each round loses a little.
- 2Rare things go first. The edges of the data fade.
- 3Keep real data. Fresh human data slows the decay.