Let's Learn About Kafka's Architecture
This post was migrated from Tistory. You can find the original here.
Kafka
- zookeeper
- kafka cluster
- broker
- topic
- partition
- topic
- broker
- kafka cluster
- producer
- consumer group
- consumer
Zookeeper
These are the components that play the roles shown above. Let’s look at each of them.
“Kafka” refers to the kafka cluster, and zookeeper is the role that manages things when you run multiple Kafka instances.
Broker
A kafka cluster can have multiple brokers, and a broker distinguishes messages by topic.
A topic is made up of one or more partitions, and a partition is basically an append-only space.
The storage location of a message within a partition is called the offset.
Messages are not deleted even after a consumer consumes a partition.
To store a message in a specific partition of a topic, you can use a key.
If you don’t use a key, messages are stored across the topic’s partitions in round-robin fashion.
Consumer
A consumer belongs to a consumer group,
and a single partition can only be connected to one consumer within a consumer group.
For example, if a consumer group has consumer1 and consumer2, you can’t connect both consumer1 and consumer2 to partition1. However, a consumer from a different group can connect to partition1.
This is why, within a single consumer group, message processing order within a partition is guaranteed.
Producer
The message is converted into a byte array via the Serializer, and the Partitioner decides which partition to send it to.
The converted byte arrays are then grouped into batches and stored in a buffer.
After that, the batch is sent. This part is called the Sender, and it runs on a separate thread from the Serializer/Partitioner.
In other words, it’s not a single sequential flow of Serializer > Partitioner > batch buffer > Sender.
Instead, ‘Serializer > Partitioner > batch buffer’ runs as one flow,
and ‘Sender’ runs as another.
Replication
Kafka has replication as a way to handle failures.
As many replicas of a partition as the replication factor are created on other brokers. Among the resulting partitions, one becomes the leader and the rest become followers. Producers and consumers only process messages through the leader, and the remaining followers read the messages added to the leader and replicate them.
If the broker holding the leader fails, one of the followers becomes the new leader.
Scalability
Kafka scales out horizontally with ease. If a single machine reaches its capacity limit, you can add more brokers and partitions, and if a consumer is too slow, you can add more consumers.