Skip to content
BENIBIX Learn

Alerting

Infrastructure📈 Monitoring & ObservabilityMonitoring Basics

Also known as Alert, Monitoring Alert, Alert Rule, Alertmanager

A rule that tells you when a number goes past a set line.

Think of it like

It is like a smoke alarm. It stays quiet until there is smoke, then it is loud.

Example

Lucy's rule fires when the site has not answered for five minutes. At 3 a.m. it pages the engineer on call, who restarts the stuck server.

Why it matters

Alerts are how a problem reaches a human in time. Too few and you miss it, too many and nobody reads them.

Definition

Alerting means sending a message to a person when something needs them. An alert rule watches a number and a limit, like errors above 5% for ten minutes. When the rule fires, it sends a message to chat, email or a phone. A good alert is rare, urgent and tells you what to do. The wait time stops one short blip from waking anyone.

Example code

yaml
groups:
- name: shop
  rules:
  - alert: SiteDown
    expr: up{job="shop"} == 0
    for: 5m
A rule that fires when the shop has not answered for five minutes in a row.
Alerting is the lower branch of the loop. A rule watches the store and calls a person.

Common mistake

Alerting on causes, like high CPU, instead of on what users feel. Alert on slow pages and errors first.

Key takeaways

  1. 1
    Rare and urgent. Every alert should need a person now.
  2. 2
    Wait a little. A short delay filters out blips.
  3. 3
    Say what to do. Link the alert to the next step.