Rebalance Your Kafka Cluster Using Cruise Control

Cluster Skewness is the major problem in today’s date. uneven Kafka partition load on broker node or unbalanced cluster can generate unnecessary disk, CPU problems, or even the need to add another broker to handle unexpected traffic.

To solve this real-time problem, Linked-in came up with a solution called Linkedin Cruise-control. It is an open-sourced Kafka Tool that helps automate and manage Kafka clusters, up to a certain level. Added a REST interface to help with the administration of Kafka clusters remotely, a centralised dashboard to operate and check the status of any Kafka cluster at LinkedIn. Below is the diagram where cruise-control sits in the architecture.

Cruise Control features include:

  1. Kafka broker resource utilization tracking
  2. The ability to query the latest replica state (offline, URP, out of sync) from brokers
  3. Goal-based resource distribution
  4. Anomaly detection with self-healing
  5. Admin operations on Kafka (add/remove/demote brokers, rebalance cluster, run PLE)

For more details, you can refer to [https://github.com/linkedin/cruise-control]. Bellow’s example shows how cruise control redistributed the load if a broker is under-utilized.

Cruise-control Deployment

What we are trying to achieve lets see with a flow chart

Dependencies needed

  1. Kafka Cluster: Fully operation cluster with prometheus node and jmx expoter enabled.
  2. Prometheus server: Prometheus server should be running and the target file must be configured to monitor the Kafka broker nodes

Different ways we tried deploying cruise-control were

  1. As a tool on AWS EC2
  2. As docker image

Deploying on AWS EC2

  1. Create a ec2 instance ( taking ubuntu as instance type)
  2. Install git :

3. Install Java and set the Home paths: sudo yum install java-1.8.0

4. clone cruise control repo:

5. cd in cruise-control

run ./gradlew jar

6. Edit only the option mentioned bellow in cruisecontrol.property file for simple connection.

Also make changes in the capacityCore.json with DISK, CPU,NW_IN,NW_OUT of the actual value of the broker node.

7 Setting Up Cruise Control Frontend.

8. Security group for the AWS EC2 instance where cruise-control is deployed must be allowed for

•Kafka broker server
•Prometheus server
•Zookeeper
•Your Ip for SSH into the host and for UI access

9. To access the UI:

http://<instance dns>:9091

Deploying as Docker image

The docker image is created out of the steps mentioned above and is pushed into dockerhub

  1. You can pull docker image using below command:

2. You need to provide your config files (ex: capacityCores.json and cruisecontrol.properties). Create a directory
mkdir ~/cc
mkdir ~/cc/config

3. Download the config files from https://github.com/linkedin/cruise-control/tree/master/config.

4. Make sure the Prometheus server is running. If not you can use the Prometheus docker image and edit the Prometheus config file with Kafka broker node targets.
mkdir ~/prometheus

create a file ~/prometheus/prometheus.yml and ~/prometheus/targets.json with proper value . ( for more details check Prometheus monitoring for Kafka)

5. Run Prometheus docker image and then cruise-control docker image:

web ui config location
webserver.ui.diskpath=./cruise-control-ui

Add /cruise-control-ui/ui-config.csv if you want to adjust web ui config.

6. Security group remains the same. Access it using http://<instance dns>:9091. You will see the homescreen

Cruise-control with SASL/SCRAM Authentication

If you have Enabled the Kafka Authentication with SASL/SCRAM you need to add the below option to the cruisecontrol.properties files:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ username=’’ \ password=’’;

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Neha Singh

Infrastructure Consultant by profession and artist by weekend