Skip to content
BENIBIX Learn

Checkpointing (Stream Processing)

Data⚙️ ProcessingEvent Time and Windows

Also known as Checkpoint, Stream Checkpoint, Savepoint

It saves a stream job's state and place. Then a crash loses nothing.

Think of it like

It is like saving a video game. If you lose, you start again from the last save, not from the start.

Example

A Flink job saves a checkpoint every 30 seconds. After a server dies, it restarts from the last one and the totals are still right.

Why it matters

Long-running stream jobs will crash at some point. Checkpoints are what make exactly-once results possible.

Definition

Checkpointing saves a stream job's state and its read position at the same moment, again and again. State is what the job remembers, such as running totals and open windows. If the job crashes, it loads the last checkpoint. Then it reads the stream again from the saved position. Because state and position match, nothing is lost or counted twice. Checkpoints go to safe storage, often every few seconds or minutes.

Common mistake

Writing checkpoints to the local disk of the server. When that server dies, the checkpoint dies with it.

Key takeaways

  1. 1
    State and position together. Saved at the same moment.
  2. 2
    Restart from the last. Then replay from that place.
  3. 3
    Durable storage. Not the server's own disk.