kafka和zk的集群安装和ssl通信

目录

  • 准备工作
  • zk的集群安装
  • kafka的集群安装
  • kafka的ssl通信

写在前面

zk和kafka的集群安装和ssl通信,安装了很多次,这次把傻瓜式安装文档贴出来

环境

ip 内存 系统 安装目录
192.168.109.139 4c8g centos7.4 /app/kafkaZK
192.168.109.140 4c8g centos7.4 /app/kafkaZK
192.168.109.141 4c8g centos7.4 /app/kafkaZK

准备工作:

临时关闭SELINUX
setenforce 0
因为是openstack环境,所以绑定内网ip

cat >> /etc/hosts <<END
1.1.1.31 kafka-1
1.1.1.30 kafka-2
1.1.1.32 kafka-3
END

1、安装oracle jdk1.8

略。快速安装jdk可参考我的另一篇文档
https://www.jianshu.com/p/91be48fbc7d4

2、安装zk

zookeeper版本 3.4.10

mkdir -p /app/kafkaZK && cd /app/kafkaZK
wget http://apache.claz.org/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

tar zxvf zookeeper-3.4.10.tar.gz
mkdir -p /app/kafkaZK/zookeeper-3.4.10/data
cd zookeeper-3.4.10/conf
cp zoo_sample.cfg zoo.cfg

修改配置项

sed -i "s/#autopurge.purgeInterval=1/autopurge.purgeInterval=24/g" zoo.cfg
sed -i "s/dataDir=\/tmp\/zookeeper/dataDir=\/app\/kafkaZK\/zookeeper-3.4.10\/data/g" zoo.cfg

cat >> zoo.cfg <<END
server.1=kafka-1:2888:3888
server.2=kafka-2:2888:3888
server.3=kafka-3:2888:3888
END

赋权
chmod 755 zoo.cfg

id文件

节点一:
echo 1 > /app/kafkaZK/zookeeper-3.4.10/data/myid
节点二:
echo 2 > /app/kafkaZK/zookeeper-3.4.10/data/myid
节点三:
echo 3 > /app/kafkaZK/zookeeper-3.4.10/data/myid
启动zk:
/app/kafkaZK/zookeeper-3.4.10/bin/zkServer.sh start
注意查看日志有无报错,日志在执行命令的当前目录下

3、安装kafka

kafka版本 0.10
cd /app/kafkaZK
国内的清华的源:
wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz

tar xzvf kafka_2.12-0.10.2.1.tgz
mv kafka_2.12-0.10.2.1 kafka

修改配置文件
sed -i "s/dataDir=\/tmp\/zookeeper/dataDir=\/app\/kafkaZK\/zookeeper-3.4.10\/data/g" /app/kafkaZK/kafka/config/zookeeper.properties
节点一上:
sed -i "s/broker.id=0/broker.id=1/g" /app/kafkaZK/kafka/config/server.properties
节点二上:
sed -i "s/broker.id=0/broker.id=2/g" /app/kafkaZK/kafka/config/server.properties
节点三上:
sed -i "s/broker.id=0/broker.id=3/g" /app/kafkaZK/kafka/config/server.properties
所有节点:

sed -i "s/#delete.topic.enable=true/delete.topic.enable=true/g" /app/kafkaZK/kafka/config/server.properties

sed -i "/num.network.threads=3/i\port=9092" /app/kafkaZK/kafka/config/server.properties
sed -i "s|log.dirs=/tmp/kafka-logs|log.dirs=/app/kafkaZK/kafka/logs|g" /app/kafkaZK/kafka/config/server.properties
sed -i "s|num.partitions=1|num.partitions=3|g" /app/kafkaZK/kafka/config/server.properties
sed -i "/num.partitions=3/i\default.replication.factor=3" /app/kafkaZK/kafka/config/server.properties
sed -i "s|#log.flush.interval.messages=10000|log.flush.interval.messages=10000|g" /app/kafkaZK/kafka/config/server.properties
sed -i "s|#log.flush.interval.ms=1000|log.flush.interval.ms=1000|g" /app/kafkaZK/kafka/config/server.properties
sed -i "s|zookeeper.connect=localhost:2181|zookeeper.connect=kafka-1:2181,kafka-2:2181,kafka-3:2181|g" /app/kafkaZK/kafka/config/server.properties

4、生成密钥

mkdir -p /var/private/ssl/kafka/

#!/bin/bash
# 生成服务器keystore(密钥和证书)
keytool -keystore server.keystore.jks -alias localhost -validity 3650 -keyalg RSA -storepass 123456 -keypass 123456 -genkey -dname "C=CN,ST=FC,L=FZ,O=LEON,OU=LEON,CN=123456.COM"
# 生成客户端keystore(密钥和证书)
keytool -keystore client.keystore.jks -alias localhost -validity 3650 -keyalg RSA -storepass 123456 -keypass 123456 -genkey -dname "C=CN,ST=FJ,L=FZ,O=LEON,OU=LEON,CN=123456.COM"
# 创建CA证书
openssl req -new -x509 -keyout ca.key -out ca.crt -days 3650 -passout pass:123456 -subj "/C=CN/ST=FJ/L=FZ/O=LEON/OU=LEON/CN=123456.COM"
# 将CA证书导入到服务器truststore
keytool -keystore server.truststore.jks -alias CARoot -import -file ca.crt -storepass 123456
# 将CA证书导入到客户端truststore
keytool -keystore client.truststore.jks -alias CARoot -import -file ca.crt -storepass 123456
# 导出服务器证书
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file -storepass 123456
keytool -keystore client.keystore.jks -alias localhost -certreq -file client-cert-file -storepass 123456
# 用CA证书给服务器证书签名
openssl x509 -req -CA ca.crt -CAkey ca.key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:123456
openssl x509 -req -CA ca.crt -CAkey ca.key -in client-cert-file -out client-cert-signed -days 365 -CAcreateserial -passin pass:123456
# 将CA证书导入服务器keystore
keytool -keystore server.keystore.jks -alias CARoot -import -file ca.crt -storepass 123456
keytool -keystore client.keystore.jks -alias CARoot -import -file ca.crt -storepass 123456
# 将已签名的服务器证书导入服务器keystore
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed -storepass 123456
keytool -keystore client.keystore.jks -alias localhost -import -file client-cert-signed -storepass 123456

验证ssl
openssl s_client -debug -connect ip:9093 -tls1
openssl s_client -debug -connect ip:9092 -tls1

注意ip,继续修改配置文件

cat >> /app/kafkaZK/kafka/config/server.properties <<END
security.inter.broker.protocol=SSL
listeners=SSL://0.0.0.0:9093
advertised.listeners=SSL://192.168.109.141:9093 
ssl.keystore.location=/var/private/ssl/kafka/server.keystore.jks
ssl.keystore.password=123456
ssl.key.password=123456
ssl.truststore.location=/var/private/ssl/kafka/server.truststore.jks
ssl.truststore.password=123456
ssl.client.auth=required
END

mkdir -p /app/kafkaZK/kafka/logs
启动kafka,打开bin目录
nohup ./kafka-server-start.sh ../config/server.properties > /app/kafkaZK/kafka/logs/kafka-zk.log 2>&1 &

把这两个文件copy到其他两个节点相同的目录下

/var/private/ssl/kafka/server.keystore.jks
/var/private/ssl/kafka/server.truststore.jks

验证ssl

openssl s_client -debug -connect ip:9093 -tls1
openssl s_client -debug -connect ip:9092 -tls1

加入开机自启动

vi /etc/rc.d/init.d/zkkafka.sh

#!/bin/bash
# chkconfig: - 85 15
#description:zk kafka
nohup /app/kafkaZK/kafka/bin/kafka-server-start.sh /app/kafkaZK/kafka/config/server.properties > /app/kafkaZK/kafka/logs/kafka-zk.log 2>&1 &
/app/kafkaZK/zookeeper-3.4.10/bin/zkServer.sh start
chmod +x /etc/rc.d/init.d/zkkafka.sh
chkconfig --add zkkafka.sh

总结

具体原理和配置项的含义这里就不一一说明了,官网都有。后续会加上zk和kafka的监控。

推荐阅读更多精彩内容

  • Spring Cloud为开发人员提供了快速构建分布式系统中一些常见模式的工具(例如配置管理,服务发现,断路器,智...
    卡卡罗2017阅读 67,152评论 12 114
  • 姓名:周小蓬 16019110037 转载自:http://blog.csdn.net/YChenFeng/art...
    aeytifiw阅读 18,628评论 10 364
  • Kafka入门经典教程-Kafka-about云开发 http://www.aboutyun.com/threa...
    葡萄喃喃呓语阅读 5,298评论 4 52
  • 一、入门1、简介Kafka is a distributed,partitioned,replicated com...
    HxLiang阅读 1,421评论 0 8
  • kafka的定义:是一个分布式消息系统,由LinkedIn使用Scala编写,用作LinkedIn的活动流(Act...
    时待吾阅读 2,944评论 1 15