Skip to content
BENIBIX Learn

Catastrophic Forgetting vs Model Collapse

People think catastrophic forgetting and model collapse are one problem, because both make a model worse after training. They are not. Forgetting happens inside one model: new training writes over skills it had. Collapse builds up across many models, each one trained on text made by the one before. The fix for the first is to test old skills. The fix for the second is to keep real human data.

AI💬 LLMs

Catastrophic Forgetting

A model learns a new task and loses skills it had.

Catastrophic forgetting is when a model learns a new task and gets worse at old ones. Training changes the same weights that held the old skills, so new learning can write over them. In language models it often shows up after tuning on a narrow set of data. A model tuned only on legal text may get worse at math or chat. Mixing some old data back in, or changing only a few weights, can lower the risk.

Example
Jack fine-tunes a chat model on his shop's help tickets. It now answers tickets well, but it starts to fail simple sums it used to get right.
Why it matters
A tuned model can pass its new test and still fail the old ones. You have to check both.
Full entry
AI💬 LLMs

Model Collapse

Models trained on other models' text get worse each round.

Model collapse is the harm that builds up when models learn, round after round, from text made by earlier models. Each model copies the common patterns well but drops some rare ones. Over many rounds, the rare parts of the first data fade away. The output grows narrow and dull. A 2024 paper in Nature showed it in language models and in other kinds of model. Keeping real human data in the mix, and filtering made-up data, lowers the risk.

Example
Researchers train each new model only on text from the one before. After a few rounds, the rare words and odd facts are gone, and all the text sounds alike.
Why it matters
More and more of the web is written by AI. Future models trained on it could take in this slow decay.
Full entry

In one line

Catastrophic Forgetting
A model learns a new task and loses skills it had.
Model Collapse
Models trained on other models' text get worse each round.