Setting up a Multi-Broker Kafka Cluster – Beginners Guide

multi-broker kafka cluster setup
Kafka is an open source distributed messaging system that is been used by many organizations for many use cases. Its use cases include stream processing, log aggregation, metrics collection and so on.

Note: This tutorial is based on Redhat 7 derivative. However, it will work on most Linux systems.

Multi-Node Kafka Cluster Setup

This tutorial will guide you to set up a latest Kafka cluster from scratch.

Prerequisites

1. You need a Zookeeper cluster before setting up a Kafka cluster. Refer this zookeeper cluster setup if you don’t have one.

2. Launch three instances. Make sure you allow the traffic between Zookeeper and Kafka instances in the security groups.

3. Set hostnames for three instances for identification using the following command.

Kafka Installation

Perform the following tasks on all the servers. 1. Update the server.

2. Install java 8.

3. Get the latest version of Kafka from here.

4. Untar the Kafka binary.

5. Rename the extracted Kafka folder with versions to Kafka.

Creating a Kafka Service

6. Open the server.properties file, find zookeeper.connect

at the bottom and enter the zookeeper IPs as shown below. Replace zk1, zk2, and zk3 with the IPs or DNS names of your zookeeper instances.

READ  How to Setup a Replicated GlusterFS Cluster on AWS EC2

Create a Kafka Service

1. Create a systemd file.

Copy the following contents on to the kafka.service unit file.

2. Reload the daemon.

Managing Kafka Service

Once the Kafka service is created, you can manage the Kafka service using the Linux service module.

1. To start the Kafka service,

2. To stop and restart,

Testing The Kafka Cluster

To test the kafka cluter setup, we will create a topic and few messages. Andn we will try to consume it from different node to conform that the cluster is working as intended.

To test, cd in to the kafka bin directory to get access to the kafka scripts.

Step 1: Create a Topic from kafka node 1 with replication-factor 3. Replace zk with zookeper IP or DNS name.

./kafka-topics.sh –create –zookeeper zk:2181 –replication-factor 3 –partitions 1 –topic test

Step 2: Describe Topic from kafka node 2.

READ  List of Best Docker Container Clustering and Orchestration Tools/Services

./kafka-topics.sh –describe –zookeeper zk:2181 –topic test

Step 3: Create a Message using the following command. Enter the message in the terminal with weach message per line. Use Ctrl + C to exit.

./kafka-console-producer.sh –broker-list kafka-node:9092 –topic test

Step 4: From a different node, try to consume the message using the following command.

./kafka-console-consumer.sh –zookeeper zk:2181 –from-beginning –topic test

If you were able to do all the test mentioned above, you have a working kafka cluster.

If you face any issue during the setup, feel free to drop us a comment below.

Free DevOps Resources

Get DevOps news, tutorials and resources in your inbox. A perfect way If you want to get started with devops. Like you, we dont like spam.

Leave a Reply

Your email address will not be published. Required fields are marked *