Setting up a Multi-Broker Kafka Cluster – Beginners Guide
- Last Updated On: October 25, 2016
- By: devopscube
Kafka is an open source distributed messaging system that is been used by many organizations for many use cases. Its use cases include stream processing, log aggregation, metrics collection and so on.
Note: This tutorial is based on Redhat 7 derivative. However, it will work on most Linux systems.
Multi-Node Kafka Cluster Setup
This tutorial will guide you to set up a latest Kafka cluster from scratch.
Prerequisites
1. You need a Zookeeper cluster before setting up a Kafka cluster. Refer this zookeeper cluster setup if you don’t have one.
2. Launch three instances. Make sure you allow the traffic between Zookeeper and Kafka instances in the security groups.
3. Set hostnames for three instances for identification using the following command.
hostnamectl set-hostname (node1,2,3)
Kafka Installation
Perform the following tasks on all the servers. 1. Update the server.
sudo yum update -y
2. Install java 8.
sudo yum -y install java-1.8.0-openjdk
3. Get the latest version of Kafka from here.
cd /opt sudo wget http://mirror.fibergrid.in/apache/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz
4. Untar the Kafka binary.
sudo tar -xvf kafka_2*
5. Rename the extracted Kafka folder with versions to Kafka.
sudo mv kafka_2.11-0.10.0.0 kafka
Creating a Kafka Service
6. Open the server.properties
file, find zookeeper.connect
at the bottom and enter the zookeeper IPs as shown below. Replace zk1, zk2, and zk3 with the IPs or DNS names of your zookeeper instances.
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
Create a Kafka Service
1. Create a systemd file.
sudo vi /lib/systemd/system/kafka.service
Copy the following contents on to the kafka.service unit file.
[Unit] Description=Kafka Before= After=network.target [Service] User=ec2-user CHDIR= {{ data_dir }} ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties Restart=on-abort [Install] WantedBy=multi-user.target
2. Reload the daemon.
sudo systemctl daemon-reload
Managing Kafka Service
Once the Kafka service is created, you can manage the Kafka service using the Linux service module.
1. To start the Kafka service,
sudo service kafka start
2. To stop and restart,
sudo service kafka stop sudo service kafka restart
Testing The Kafka Cluster
To test the kafka cluter setup, we will create a topic and few messages. Andn we will try to consume it from different node to conform that the cluster is working as intended.
To test, cd in to the kafka bin directory to get access to the kafka scripts.
cd /opt/kafka/bin
Step 1: Create a Topic from kafka node 1 with replication-factor 3. Replace zk
with zookeper IP or DNS name.
./kafka-topics.sh –create –zookeeper zk:2181 –replication-factor 3 –partitions 1 –topic test
Step 2: Describe Topic from kafka node 2.
./kafka-topics.sh –describe –zookeeper zk:2181 –topic test
Step 3: Create a Message using the following command. Enter the message in the terminal with weach message per line. Use Ctrl + C
to exit.
./kafka-console-producer.sh –broker-list kafka-node:9092 –topic test
Step 4: From a different node, try to consume the message using the following command.
./kafka-console-consumer.sh –zookeeper zk:2181 –from-beginning –topic test
If you were able to do all the test mentioned above, you have a working kafka cluster.
If you face any issue during the setup, feel free to drop us a comment below.
devopscube
Other Interesting Blogs
How to Setup Custom UI Theme For Jenkins
If you are bored with the old Jenkins UI, its font, and icons, you can give your Jenkins a makeover using custom
Vagrant Tutorial For Beginners: Getting Started Guide
In this Vagrant tutorial, I will teach you to set up Vagrant on your workstation to create and manage Virtual machines for
Jenkins Automated Build Trigger On Github Pull Request
Building projects based on pull request is something you cannot avoid in CI/CD pipelines. Nowadays every team does several deployments/operations per day
Linux VI Editor Shortcuts, Tips and Productivity Hacks For Beginners
In our last blog post, Â we have covered Linux CLI Productivity tips. When you work on Linux systems, vi editor is something
Automating Code Reviews on GitHub
Code reviews are part of the daily activities of software engineers and a key process in release management. With engineers spending 10%
Comments
you should fix your title , this is a single broker , not multiples