Skip to content
BENIBIX Learn

Apache Kafka

Also known as Kafka, Kafka Cluster

The most used free tool for event streams.

Think of it like

It is like a big notebook that many people write in. Each line stays. Readers keep a bookmark.

Example

LinkedIn first built Kafka to move its activity data. Today banks, shops and games use it to move clicks, payments and logs.

Why it matters

It is the default backbone for event streaming. Most streaming tools can read from it and write to it.

Definition

Apache Kafka is a free, open source platform for event streaming. It stores events in topics. Topics are split into partitions and spread over servers called brokers. Events are written to the end of a log. They are kept for a set time, seven days by default. Many consumers can read the same events, each at its own pace.

Example code

console
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 \
    --topic orders --from-beginning
{"id": 1, "total": 19.5}
{"id": 2, "total": 42.0}
Read the orders topic from its first event, using the tool that ships with Kafka.
A topic is split into partitions. Each consumer in a group reads its own, and remembers its place.

Common mistake

Using it as a simple job queue. It keeps order only inside a partition, and it does not delete a message when it is read.

Key takeaways

  1. 1
    A log, not a queue. Events are kept after they are read.
  2. 2
    Split to scale. Topics spread over partitions and brokers.
  3. 3
    Many readers. Each group reads at its own pace.

Compare