Skip to content
BENIBIX Learn

Training Checkpoint

AI⚙️ MLOps & DeploymentLarge-Scale Training

Also known as Checkpointing (ML), Resume from Checkpoint

A saved copy of a model partway through training. A crash then loses little.

Think of it like

It is like a save point in a video game. If you lose, you start again from there, not from the start.

Example

Jack's training job saves a checkpoint every hour. On day 3 a GPU fails, and the job goes on from the last save, losing 40 minutes, not 3 days.

Why it matters

Long training jobs on many machines will hit failures. Checkpoints turn a lost week into a lost hour.

Definition

A training checkpoint is a saved copy of a model partway through training. It holds the model's weights and the rest of the state needed to carry on, such as the optimizer's numbers and the step count. Training saves one every so often. If a machine fails, the job starts again from the last checkpoint instead of from zero. Teams also keep checkpoints to compare and pick the best one.
The timeline at the bottom shows checkpoints. After the crash, training goes on from the last save.

Common mistake

Saving only the weights. Without the optimizer state, training does not truly pick up where it stopped.

Key takeaways

  1. 1
    Save often. Every hour, or every few thousand steps.
  2. 2
    Save it all. The model, the optimizer and the step.
  3. 3
    Resume, not restart. Pick up from the last save.