Kafka is an open source distributed messaging system that is been used by many organizations for many use cases. Its use cases include stream processing, log aggregation, metrics collection and so on.
Note: This tutorial is based on Redhat 7 derivative. However, it will work on most Linux systems.
Multi-Node Kafka Cluster Setup
This tutorial will guide you to set up a latest Kafka cluster from scratch.
Prerequisites
1. You need a Zookeeper cluster before setting up a Kafka cluster. Refer this zookeeper cluster setup if you don’t have one.
2. Launch three instances. Make sure you allow the traffic between Zookeeper and Kafka instances in the security groups.
3. Set hostnames for three instances for identification using the following command.
|
1 |
hostnamectl set-hostname (node1,2,3) |
Kafka Installation
Perform the following tasks on all the servers. 1. Update the server.
|
1 |
sudo yum update -y |
2. Install java 8.
|
1 |
sudo yum -y install java-1.8.0-openjdk |
3. Get the latest version of Kafka from here.
|
1 2 |
cd /opt sudo wget http://mirror.fibergrid.in/apache/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz |
4. Untar the Kafka binary.
|
1 |
sudo tar -xvf kafka_2* |
5. Rename the extracted Kafka folder with versions to Kafka.
|
1 |
sudo mv kafka_2.11-0.10.0.0 kafka |
Creating a Kafka Service
6. Open the server.properties file, find zookeeper.connect
at the bottom and enter the zookeeper IPs as shown below. Replace zk1, zk2, and zk3 with the IPs or DNS names of your zookeeper instances.
|
1 |
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181 |
Create a Kafka Service
1. Create a systemd file.
|
1 |
sudo vi /lib/systemd/system/kafka.service |
Copy the following contents on to the kafka.service unit file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[Unit] Description=Kafka Before= After=network.target [Service] User=ec2-user CHDIR= {{ data_dir }} ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties Restart=on-abort [Install] WantedBy=multi-user.target |
2. Reload the daemon.
|
1 |
sudo systemctl daemon-reload |
Managing Kafka Service
Once the Kafka service is created, you can manage the Kafka service using the Linux service module.
1. To start the Kafka service,
|
1 |
sudo service kafka start |
2. To stop and restart,
|
1 2 |
sudo service kafka stop sudo service kafka restart |
Testing The Kafka Cluster
To test the kafka cluter setup, we will create a topic and few messages. Andn we will try to consume it from different node to conform that the cluster is working as intended.
To test, cd in to the kafka bin directory to get access to the kafka scripts.
|
1 |
cd /opt/kafka/bin |
Step 1: Create a Topic from kafka node 1 with replication-factor 3. Replace zk with zookeper IP or DNS name.
./kafka-topics.sh –create –zookeeper zk:2181 –replication-factor 3 –partitions 1 –topic test
Step 2: Describe Topic from kafka node 2.
./kafka-topics.sh –describe –zookeeper zk:2181 –topic test
Step 3: Create a Message using the following command. Enter the message in the terminal with weach message per line. Use Ctrl + C to exit.
./kafka-console-producer.sh –broker-list kafka-node:9092 –topic test
Step 4: From a different node, try to consume the message using the following command.
./kafka-console-consumer.sh –zookeeper zk:2181 –from-beginning –topic test
If you were able to do all the test mentioned above, you have a working kafka cluster.
If you face any issue during the setup, feel free to drop us a comment below.
