Rebalance Your Kafka Cluster Using Cruise Control

Photo from
  1. Kafka broker resource utilization tracking
  2. The ability to query the latest replica state (offline, URP, out of sync) from brokers
  3. Goal-based resource distribution
  4. Anomaly detection with self-healing
  5. Admin operations on Kafka (add/remove/demote brokers, rebalance cluster, run PLE)
Photo: Example of partition reassignment

Cruise-control Deployment

Photo: Cruise-control with Kafka cluster and Prometheus

Dependencies needed

  1. Kafka Cluster: Fully operation cluster with prometheus node and jmx expoter enabled.
  2. Prometheus server: Prometheus server should be running and the target file must be configured to monitor the Kafka broker nodes
  1. As a tool on AWS EC2
  2. As docker image

Deploying on AWS EC2

  1. Create a ec2 instance ( taking ubuntu as instance type)
  2. Install git :
sudo yum install -y git 
sudo yum install java-1.8.0-openjdk-devel
git clone or wget “"
# If using TLS encryption, use 9094; use 9092 if using plaintext bootstrap.servers= <Bootstrap servers endpoint string from kafka>
zookeeper.connect= <zookeeper endpoint string from kafka>
# Use the Prometheus Metric Sampler metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.prometheus.PrometheusMetricSampler
# Prometheus Metric Sampler specific configuration prometheus.server.endpoint= # Replace with your Prometheus IP and port
# Change the capacity config file and specify its path; details below
“brokerId”: “-1”,
“capacity”: {
“DISK”: “10000”,
“CPU”: { “num.cores”: “2” },
“NW_IN”: “5000000”,
“NW_OUT”: “5000000”
}, “doc”: “This is the default capacity. Capacity unit used for disk is in MB, cpu is in number of cores, network throughput is in KB.”
wget -xvzf cruise-control-ui-0.3.4.tar.gz
mv cruise-control-ui ~/cruise-control-2.4.25/

Deploying as Docker image

  1. You can pull docker image using below command:
docker pull 11nehas/cruise-control:latest
sudo docker run -d -p 9090:9090 — name=prometheus -v $PWD/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v $PWD/prometheus/targets.json:/etc/prometheus/targets.json prom/prometheus — config.file=/etc/prometheus/prometheus.ymldocker run -p 9091:9091  --link prometheus:prometheus -v $PWD/cc/config/ -v $PWDc/cc/config/capacityCores.json:/cc/config/capacityCores.json 11nehas/cruise-control:latest

Cruise-control with SASL/SCRAM Authentication



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store