Lambda and Kappa Architecture
Level 2 · Pipelines
Two designs for answers that are both live and complete. Lambda runs a batch path and a stream path side by side, and Kappa runs a single stream.
Course contentsLesson 6 of 11
Level 3 · Kinds of data
Level 4 · Models and workloads
After this lesson you can
- Draw the two paths of a Lambda architecture
- Explain how Kappa replays old data
- Say what each design costs to keep running
12 mincovers 2 glossary entries
Worth reading first: Batch and Stream Processing
Why Lambda keeps two paths and Kappa keeps one
Lambda and Kappa are two designs for answers that are both live and complete. Lambda, named in 2011, runs a batch path and a stream path side by side and merges them. Kappa, put forward in 2014, drops the batch path and runs everything through one stream. Both solve the same problem in different ways.
The problem is that batch and stream each give half an answer. A nightly batch is complete and correct, but a day old. A stream is fresh, but it may miss late data or count something twice. A business that wants today's sales live, and exact by tonight, needs a bit of both.
Lambda keeps both. New data goes down two paths at once. The batch layer stores every record and rebuilds full, exact results each night. The speed layer covers the hours since the last batch, quickly and roughly. A serving layer joins the two, so a query sees last night's exact total plus today's live count.
The price is writing the same logic twice. The batch job and the stream job must count sales the same way, in two different tools. Every change is made in both places and tested in both. Sooner or later the two drift apart, and the morning number jumps when the batch result replaces the live one.
Kappa keeps one path. Every event is written to a log that holds a long history, often in Kafka. One stream job reads the log and builds the results. When the logic changes, you start a second copy of the job at the start of the log. It replays the history, and you switch over once it catches up.
Kappa needs the log to keep enough history to replay, which can mean weeks or months of events on disk. Replaying a year of data through a stream job can take hours. It also needs a stream engine strong enough to give exact answers, not rough ones. Modern engines can, which is why Kappa has grown more common.
People often think they must pick one of the two names. Most real systems are simpler than either. A nightly batch plus one small live table for today often does the job. Reach for Lambda or Kappa only when people truly need exact and live numbers on the same screen.
Which one to pick
- 1
Lambda
Pick it when you already trust a batch system and need to add a live view beside it.
- 2
Pick it when you start fresh and one team owns the logic. Your events must sit in a log with a long history.
Worked example
A ride app counts trips
Lucy's ride app shows drivers their trips for the day. Totals were rebuilt each night in one tool and counted live in another. Twice a month the two disagreed by a few trips, and drivers wrote in.
The team moved to one stream job over a log kept for 30 days. When the counting rule changed, they replayed all 30 days in 2 hours. The live and morning numbers now always match.
The words for this lesson2
Check yourself
What is the main cost of a Lambda architecture?
Finished this lesson?
Marking it complete also marks its 2 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.