Skip to content
This repository has been archived by the owner on Feb 8, 2024. It is now read-only.

Kafka Configuration

Jump to bottom
Ajay Paratmandali edited this page Jul 28, 2021 · 2 revisions

Kafka Configuration

Prerequisite

  • Configure cortx-py-utils RPM

Basic configuration

  1. Basic Configuration
yum install -y yum-utils
yum-config-manager --add-repo "${CORTX_RELEASE_REPO}/3rd_party/"
yum-config-manager --add-repo "${CORTX_RELEASE_REPO}/cortx_iso/"
rpm --import "${CORTX_RELEASE_REPO}/cortx_iso/RPM-GPG-KEY-Seagate"
  1. Install Kafka
yum install -y kafka*  --nogpgcheck

SingleNode

  1. /etc/hosts
<NODE-1 eth2>  srvnode-1
  1. message bus conf
  • Create file /etc/cortx/message_bus.conf
{
  "message_broker": {
    "type": "kafka",
    "cluster": [
      {
        "server": "srvnode-1",
        "port": "9092"
      }
    ]
  }
}
  1. Check configuration of zookeeper
  • Check /opt/kafka/config/zookeeper.properties
clientPort=2181
tickTime=2000
initLimit=10
syncLimit=5
dataLogDir=/var/log/zookeeper
dataDir=/var/lib/zookeeper
server.1=<CURRENT_NODE>:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
4lw.commands.whitelist=*
  1. Check configuration of kafka
  • Check /opt/kafka/config/server.properties
broker.id=1
listeners=PLAINTEXT://srvnode-1:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/log/kafka
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=srvnode-1:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
  1. Add broker id (same as in point 4)
echo "1" > /var/lib/zookeeper/myid
  1. Add Permission
mkdir -p /var/log/zookeeper /var/lib/zookeeper /var/log/kafka

chown -R kafka:kafka /var/log/zookeeper
chown -R kafka:kafka /var/lib/zookeeper
chown -R kafka:kafka /var/log/kafka
  1. Enable service
systemctl daemon-reload
systemctl enable kafka-zookeeper
systemctl enable kafka
  1. Start service
systemctl start kafka-zookeeper
sleep 5 # (kafka service needs zookeeper service to be up and running.)
systemctl start kafka

MultiNode (Run on all node)

  1. /etc/hosts
<NODE-1 eth2>  srvnode-1
<NODE-2 eth2>  srvnode-2
<NODE-3 eth2>  srvnode-3
  1. message bus conf
  • Create file /etc/cortx/message_bus.conf
{
  "message_broker": {
    "type": "kafka",
    "cluster": [
      {
        "server": "srvnode-1",
        "port": "9092"
      },
      {
        "server": "srvnode-2",
        "port": "9092"
      },
      {
        "server": "srvnode-3",
        "port": "9092"
      }
    ]
  }
}
  1. Check configuration of zookeeper
  • Check /opt/kafka/config/zookeeper.properties
clientPort=2181
tickTime=2000
initLimit=10
syncLimit=5
dataLogDir=/var/log/zookeeper
dataDir=/var/lib/zookeeper
server.1=srvnode-1:2888:3888
server.2=srvnode-2:2888:3888
server.3=srvnode-3:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
4lw.commands.whitelist=*
  1. Check configuration of kafka
  • Check /opt/kafka/config/server.properties
  • Keep broker.id different for each node
  • Update listeners name to current node name
broker.id=1
listeners=PLAINTEXT://srvnode-1:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/log/kafka
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=srvnode-1:2181,srvnode-2:2181,srvnode-3:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
default.replication.factor=3
  1. Add broker id (same as in point 4)
echo "1" > /var/lib/zookeeper/myid
  1. Add Permission
mkdir -p /var/log/zookeeper /var/lib/zookeeper /var/log/kafka

chown -R kafka:kafka /var/log/zookeeper
chown -R kafka:kafka /var/lib/zookeeper
chown -R kafka:kafka /var/log/kafka
  1. Enable service
systemctl daemon-reload
systemctl enable kafka-zookeeper
systemctl enable kafka
  1. Start service
systemctl start kafka-zookeeper
sleep 5 # (kafka service needs zookeeper service to be up and running.)
systemctl start kafka

Cleanup

  1. Cleanup
rm -rf /opt/kafka/config/server.properties
rm -rf /opt/kafka/config/zookeeper.properties
rm -rf /etc/cortx/message_bus.conf
rm -rf /var/log/zookeeper /var/lib/zookeeper /var/log/kafka
yum remove -y kafka* --nogpgcheck

Ref command:

/opt/kafka/bin/kafka-topics.sh --create --topic alerts --bootstrap-server srvnode-1:9092

/opt/kafka/bin//kafka-topics.sh --delete --topic alerts --bootstrap-server srvnode-1:9092

/opt/kafka/bin/kafka-console-consumer.sh --topic alerts --from-beginning  --bootstrap-server srvnode-1:9092
Clone this wiki locally