Install Kafka Cluster and Zookeeper with High Availability

kafka multi broker

What is apache kafka ?

What is apache zookeeper?

Setup Zookeeper Cluster with High Availability

sudo apt update
apt install -y openjdk-11-jdk
vi /etc/hosts
10.20.20.51 ha-zoo1
10.20.20.52 ha-zoo2
10.20.20.53 ha-zoo3
cd /opt
sudo wget https://downloads.apache.org/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz
sudo tar -xvf apache-zookeeper-3.6.2-bin.tar.gz
sudo mv apache-zookeeper-3.6.2 zookeeper
cd zookeeper
sudo vi conf/zoo.cfg
tickTime=2000
dataDir=/data/zookeeper
clientPort=2181
initLimit=10
syncLimit=5
server.1=ha-zoo1:2888:3888
server.2=ha-zoo2:2888:3888
server.3=ha-zoo3:2888:3888

set multiple zookeeper, setup each node

vi /data/zookeeper/myid
1
vi /data/zookeeper/myid
2
vi /data/zookeeper/myid
3
java -cp lib/zookeeper-3.6.2.jar:lib/*:conf org.apache.zookeeper.server.quorum.QuorumPeerMain conf/zoo.cfg
cd /opt/zookeeper
bin/zkCli.sh -server ha-zoo1:2181
[zk: ha-zoo3:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: ha-zoo3:2181(CLOSED) 3] quit

Setup Kafka Cluster Multi Broker with High Availability

apt update
apt install -y openjdk-11-jdk
vi /etc/hosts
10.20.20.41 ha-kafka1
10.20.20.42 ha-kafka2
10.20.20.43 ha-kafka3
10.20.20.51 ha-zoo1
10.20.20.52 ha-zoo2
10.20.20.53 ha-zoo3
mkdir /opt/kafka
curl https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz -o /opt/kafka/kafka.tgz
cd /opt/kafka
tar xvfz kafka.tgz --strip 1
sudo mkdir -p /data/kafka/log
chown -R ubuntu:ubuntu /data/kafka/
vi bin/config/server.propertieslog.dirs=/data/kafka/log
num.partitions=3
zookeeper.connect=ha-zoo1:2181,ha-zoo2:2181,ha-zoo3:2181
vi bin/config/server.properties
broker.id=0
vi bin/config/server.properties
broker.id=1
vi bin/config/server.properties
broker.id=2
vi /etc/systemd/system/kafka.service
[Unit]
Description=Kafka
Before=
After=network.target
[Service]
User=ubuntu
CHDIR= {{ data_dir }}
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Restart=on-abort
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl start kafka.service
sudo systemctl enable kafka.service
sudo systemctl status kafka.service
root@ha-kafka1:/opt/kafka# bin/kafka-topics.sh --create --bootstrap-server ha-kafka1:9092 ha-kafka2:9092 ha-kafka3:9092 --topic test-multibroker
Created topic test-multibroker.
root@ha-kafka1:/opt/kafka# bin/kafka-topics.sh --list --bootstrap-server ha-kafka1:9092 ha-kafka2:9092 ha-kafka3:9092 
test-multibroker
root@ha-kafka1:/opt/kafka# ls /data/kafka/log/
cleaner-offset-checkpoint log-start-offset-checkpoint meta.properties recovery-point-offset-checkpoint replication-offset-checkpoint test-multibroker-0 test-multibroker-2

This is my little notes based my experiment and my learning. Keep learning and keep calm.