Rebalance Your Kafka Cluster Using Cruise Control

Photo from https://softwareengineeringdaily.com/2020/02/20/linkedin-kafka/
  1. Kafka broker resource utilization tracking
  2. The ability to query the latest replica state (offline, URP, out of sync) from brokers
  3. Goal-based resource distribution
  4. Anomaly detection with self-healing
  5. Admin operations on Kafka (add/remove/demote brokers, rebalance cluster, run PLE)
Photo: Example of partition reassignment

Cruise-control Deployment

Photo: Cruise-control with Kafka cluster and Prometheus

Dependencies needed

  1. Kafka Cluster: Fully operation cluster with prometheus node and jmx expoter enabled.
  2. Prometheus server: Prometheus server should be running and the target file must be configured to monitor the Kafka broker nodes
  1. As a tool on AWS EC2
  2. As docker image

Deploying on AWS EC2

  1. Create a ec2 instance ( taking ubuntu as instance type)
  2. Install git :
sudo yum install -y git 
sudo yum install java-1.8.0-openjdk-devel
git clone https://github.com/linkedin/cruise-control.git or wget “https://github.com/linkedin/cruise-control/archive/refs/tags/2.5.42.zip"
# If using TLS encryption, use 9094; use 9092 if using plaintext bootstrap.servers= <Bootstrap servers endpoint string from kafka>
zookeeper.connect= <zookeeper endpoint string from kafka>
# Use the Prometheus Metric Sampler metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.prometheus.PrometheusMetricSampler
# Prometheus Metric Sampler specific configuration prometheus.server.endpoint=1.2.3.4:9090 # Replace with your Prometheus IP and port
# Change the capacity config file and specify its path; details below
capacity.config.file=config/capacityJBOD.json
{ 
“brokerCapacities”:
[
{
“brokerId”: “-1”,
“capacity”: {
“DISK”: “10000”,
“CPU”: { “num.cores”: “2” },
“NW_IN”: “5000000”,
“NW_OUT”: “5000000”
}, “doc”: “This is the default capacity. Capacity unit used for disk is in MB, cpu is in number of cores, network throughput is in KB.”
}
]
}
wget https://github.com/linkedin/cruise-control-ui/releases/download/v0.3.4/cruise-control-ui-0.3.4.tar.gztar -xvzf cruise-control-ui-0.3.4.tar.gz
mv cruise-control-ui ~/cruise-control-2.4.25/

Deploying as Docker image

  1. You can pull docker image using below command:
docker pull 11nehas/cruise-control:latest
sudo docker run -d -p 9090:9090 — name=prometheus -v $PWD/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v $PWD/prometheus/targets.json:/etc/prometheus/targets.json prom/prometheus — config.file=/etc/prometheus/prometheus.ymldocker run -p 9091:9091  --link prometheus:prometheus -v $PWD/cc/config/cruisecontrol.properties:/cc/config/cruisecontrol.properties -v $PWDc/cc/config/capacityCores.json:/cc/config/capacityCores.json 11nehas/cruise-control:latest

Cruise-control with SASL/SCRAM Authentication

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store