# Training Checkpoint

> A saved copy of a model partway through training. A crash then loses little.

**Field:** Artificial Intelligence  
**Category:** MLOps & Deployment  
**Also known as:** Checkpointing (ML), Resume from Checkpoint

## Think of it like

It is like a save point in a video game. If you lose, you start again from there, not from the start.

## Definition

A training checkpoint is a saved copy of a model partway through training. It holds the model's weights and the rest of the state needed to carry on, such as the optimizer's numbers and the step count. Training saves one every so often. If a machine fails, the job starts again from the last checkpoint instead of from zero. Teams also keep checkpoints to compare and pick the best one.

## Example

Jack's training job saves a checkpoint every hour. On day 3 a GPU fails, and the job goes on from the last save, losing 40 minutes, not 3 days.

## Why it matters

Long training jobs on many machines will hit failures. Checkpoints turn a lost week into a lost hour.

## Common mistake

Saving only the weights. Without the optimizer state, training does not truly pick up where it stopped.

## Three principles

- **Save often.** Every hour, or every few thousand steps.
- **Save it all.** The model, the optimizer and the step.
- **Resume, not restart.** Pick up from the last save.

## Related terms

- [Distributed Training](https://learn.benibix.com/term/ai/distributed-training)
- [Model Weights File](https://learn.benibix.com/term/ai/model-weights)
- [Optimizer (Machine Learning)](https://learn.benibix.com/term/ai/optimizer)
- [Epoch](https://learn.benibix.com/term/ai/epoch)
- [Experiment Tracking](https://learn.benibix.com/term/ai/experiment-tracking)

![Two panels on top. Left: four GPUs, each with a full copy of the model, each fed a quarter of the data. Right: one model cut into four parts, one part on each GPU, with data passing through them in turn. Below, a timeline with save points, a crash, and training going on from the last save.](https://learn.benibix.com/diagrams/ai7-distributed-training.svg)

---

Source: https://learn.benibix.com/term/ai/training-checkpoint
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
