Catastrophic Forgetting vs Model Collapse
People think catastrophic forgetting and model collapse are one problem, because both make a model worse after training. They are not. Forgetting happens inside one model: new training writes over skills it had. Collapse builds up across many models, each one trained on text made by the one before. The fix for the first is to test old skills. The fix for the second is to keep real human data.
Catastrophic Forgetting
A model learns a new task and loses skills it had.
Catastrophic forgetting is when a model learns a new task and gets worse at old ones. Training changes the same weights that held the old skills, so new learning can write over them. In language models it often shows up after tuning on a narrow set of data. A model tuned only on legal text may get worse at math or chat. Mixing some old data back in, or changing only a few weights, can lower the risk.
Model Collapse
Models trained on other models' text get worse each round.
Model collapse is the harm that builds up when models learn, round after round, from text made by earlier models. Each model copies the common patterns well but drops some rare ones. Over many rounds, the rare parts of the first data fade away. The output grows narrow and dull. A 2024 paper in Nature showed it in language models and in other kinds of model. Keeping real human data in the mix, and filtering made-up data, lowers the risk.
In one line
- Catastrophic Forgetting
- A model learns a new task and loses skills it had.
- Model Collapse
- Models trained on other models' text get worse each round.