⚙️Training Checkpoint
Also known as Checkpointing (ML), Resume from Checkpoint
A saved copy of a model partway through training. A crash then loses little.
Think of it like
It is like a save point in a video game. If you lose, you start again from there, not from the start.
Example
Jack's training job saves a checkpoint every hour. On day 3 a GPU fails, and the job goes on from the last save, losing 40 minutes, not 3 days.
Why it matters
Long training jobs on many machines will hit failures. Checkpoints turn a lost week into a lost hour.
Definition
Common mistake
Key takeaways
- 1Save often. Every hour, or every few thousand steps.
- 2Save it all. The model, the optimizer and the step.
- 3Resume, not restart. Pick up from the last save.