📟Reliability
Uptime, failure and high availability · 26 terms
Service Levels
6How reliable a service must be. It is measured, set as a goal and written down as a promise. Also the hand work that keeps it running.
Running live systems the way you build code. Set goals, and let code do the dull work.
The same hand work, done again and again, that a script could do.
A number you measure. It shows how well a service works for its users.
The goal a team sets for a number it measures. An example is 99.95% of requests working.
A promise in a contract on how well a service works. It says what you get if it fails.
The amount of failure a target allows. Spend it on change, or stop when it is gone.
Resilient Systems
8Ways to build a service so that one broken part does not take it all down.
A service built to stay up nearly all the time. Each part has a spare copy.
A system that keeps going with no break, even when a part has failed.
A small test that asks an app if it is well. Broken copies then get no traffic.
A spare of each key part, so one failure does not stop the system.
Switching work to a standby copy when the main one fails.
The key jobs keep working when one part fails. The extras switch off.
Breaking parts of a system on purpose, to find weak spots first.
A planned team drill. A failure is caused on purpose to practise the response.
Incident Response
8What a team does when something breaks, from the first page to the written review.
A problem no one planned. It breaks a service or makes it worse for users.
A number that says how bad an incident is. SEV1 is usually the worst.
The one person in charge in an incident. They hand out the jobs.
Taking turns to be the one who picks up alerts, by day or by night. Each turn is a set time.
The rule for who gets the page next when no one answers.
Written steps to fix a known problem. Anyone can follow them, even when tired.
A public page that says if a service works. It shows news in an outage.
A written review after an incident. It finds causes and fixes, not someone to blame.
Reliability Metrics
4The times that show how often a service breaks. They also show how fast the team sees it, answers and brings it back.
The average time it takes to fix a service that broke. Lower is better.
How long it takes to spot a break. It is the mean of many.
How long it takes a person to pick up an alert. It is the mean of many.
How long a system runs before it breaks again. It is the mean of many.