zabbix高可用设计目标
1、keepalived服务优先级选择切换机制：对于zabbix服务器来说，只要zabbix存活和mysql存活，就能够正常记录数据，不会丢失数据，php和httpd只是web页面的访问而已，所以我在这里定义mysql和zabbix为主要服务，php和httpd为次要服务，为了实现主要服务存在，次要服务挂了；次要服务器存在，主要服务器挂了，keepalived会优先选择主要服务存在的一方作为master，让keepalived切换更为更合理。
2、数据库主主同步：不管切到哪一边都需要保持数据一致性，不可出现丢数据或者数据重复，保证数据库的高可用。
3、文件双向同步：
（1）web文件同步：任何做web文件得配置和修改，保证两边一致，确保切换不会发生任何变化。
（2）zabbix服务文件同步：对zabbix_servr的配置文件做的任何修改，和脚本的修改进行同步，也是为了保证两边的服务一致性，没有落后情况，达到无需人工干预自动切换服务可正常时候。
（3）到此完成了zabbix得无缝切换，无故障时间！
一、实验环境
操作系统： CentOS7.2 Minial
zabbixA： 192.168.1.102
zabbixB： 192.168.1.103
vip： 192.168.1.120
二、软件安装
在zabbixA和zabbixB
1.安装zabbix组件
# yum -y install epel-release
# rpm -hiv http://repo.zabbix.com/zabbix/3.5/rhel/7/x86_64/zabbix-release-3.5-1.el7.noarch.rpm
# yum -y install zabbix-server-mysql zabbix-web.noarch zabbix-web-mysql.noarch zabbix-java-gateway zabbix-get
# yum -y install zabbix-agent zabbix-sender
2. 安装数据库组件
# rpm -ivh https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
# yum repolist enabled | grep "mysql.*-community.*"
# yum -y install mysql-community-server mysql-community-client
3.安装php httpd组件
# yum -y install php php-gd php-mysql php-bcmath php-mbstring php-xml httpd
4.安装文件同步组件
# yum -y install rsync inotify-tools
5. 双机互信，实现免密登录
# ssh-keygen -t rsa -P '' -q -f ~/.ssh/id_rsa
# ssh-copy-id root@$ip
6. 设置selinux模式和防火墙
# sysctl -w net.ipv4.ip_nonlocal_bind=1
# echo "net.ipv4.ip_nonlocal_bind=1" >> /etc/sysctl.conf
# setenforce 0
# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config
# iptables -F
三、zabbixA软件配置
1.数据库配置
# systemctl start mysqld.service
# systemctl enable mysqld.service
# cat /var/log/mysqld.log | grep password
# mysql -u root -p"XXX"
mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=1;
mysql> set password = password('mysql123');
mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbix123';
2.zabbix-server配置
# sed -i '/^# SourceIP/s/=.*$/=192.168.1.120/' /etc/zabbix/zabbix_server.conf
# sed -i '/^# DBHost/s/=.*$/=192.168.1.120/' /etc/zabbix/zabbix_server.conf
# sed -i '/^DBName/s/=.*$/=zabbix/' /etc/zabbix/zabbix_server.conf
# sed -i '/^# DBPassword/s/.*$/DBPassword=zabbix123/' /etc/zabbix/zabbix_server.conf
3.php配置
# sed -i 's/post_max_size = 8M/post_max_size = 32M/g' /etc/php.ini
# sed -i 's/upload_max_filesize = 2M/upload_max_filesize = 50M/g' /etc/php.ini
# sed -i 's/\;date.timezone =/date.timezone = Asia\/Shanghai/' /etc/php.ini
# sed -i 's/max_execution_time = 30/max_execution_time = 600/g' /etc/php.ini
# sed -i 's/max_input_time = 60/max_input_time = 600/g' /etc/php.ini
# sed -i 's/memory_limit = 128M/memory_limit = 256M/g' /etc/php.ini
# systemctl restart zabbix-server
# systemctl enable zabbix-server
# systemctl restart httpd
# systemctl enable httpd
4.keepalived配置
# vim /etc/keepalived/keepalived.conf
############################################
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check {
script "/etc/keepalived/check.sh"
interval 5
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 100
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
check
}
virtual_ipaddress {
192.168.1.120
}
#notify_master "/etc/keepalived/zabbix.sh start"
#notify_backup "/etc/keepalived/zabbix.sh stop"
}
##############################################
# vim /etc/keepalived/check.sh
##############################################
#!/bin/bash
zabbix_server_status1=$(ps -C zabbix_server --no-heading|wc -l)
if [ "${zabbix_server_status1}" = "0" ]; then
systemctl start zabbix_server.service
sleep 3
zabbix_server_status2=$(ps -C zabbix_server --no-heading|wc -l)
if [ "${zabbix_server_status2}" = "0" ]; then
systemctl restart keepalived.service
fi
fi
mysqld_status1=$(ps -C mysqld --no-heading|wc -l)
if [ "${mysqld_status1}" = "0" ]; then
systemctl start mysqld.service
sleep 3
mysqld_status2=$(ps -C mysqld --no-heading|wc -l)
if [ "${mysqld_status2}" = "0" ]; then
systemctl restart keepalived.service
fi
fi
httpd_status1=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status1}" = "0" ]; then
systemctl start httpd.service
sleep 3
httpd_status2=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status2}" = "0" ]; then
systemctl restart keepalived.service
fi
fi
##########################################
# chmod +x /etc/keepalived/check.sh
5. 文件同步设置
# vim /opt/data_realtime_rsync.sh
####################################################################
#!/bin/bash
# File name:data_realtime_sync.sh
# File path: /opt/data_realtime_sync.sh
src_dir="/etc/zabbix"
dest_dir="/etc/zabbix"
src_ip="192.168.1.102"
dest_ip="192.168.1.103"
. /etc/init.d/functions
cd ${src_dir}
inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |
while read file;do
INO_EVENT=$(echo $file | awk '{print $1}')
INO_FILE=$(echo $file | awk '{print $2}')
if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
if [[ $INO_EVENT =~ 'ATTRIB' ]];then
if [ ! -d "$INO_FILE" ];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
fi
if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then
for ip in ${dest_ip};do
rsync -avzcR --delete -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
done
#####################################################################
# chmod +x /opt/data_realtime_rsync.sh
# vim /etc/systemd/system/rsync-inotity.service
###############################################
[Unit]
Description=File instant rsync script
[Service]
ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh &
[Install]
WantedBy=multi-user.target
#################################################
6.数据库主主同步配置
# vim /etc/my.cnf
######################################################
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
server-id=1
log-bin=mysql-bin
auto-increment-increment = 2
auto-increment-offset = 1
binlog_format = mixed
sync_binlog=1
binlog-do-db=zabbix
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
binlog-ignore-db=information_schema
replicate-do-db=zabbix
replicate-ignore-db=mysql
replicate-ignore-db=sys
replicate-ignore-db=performance_schema
replicate-ignore-db=information_schema
######################################################
# systemctl restart mysqld
登录zabbixB的MySQL
创建数据同步账号
# mysql -u root -p"mysql123"
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repluser'@'%' IDENTIFIED BY 'Repl@123';
mysql> grant all privileges on zabbix.* to zabbix@192.168.1.120 identified by 'zabbix123';
mysql> FLUSH PRIVILEGES;
# exit
获取zabbixB二进制日志名和位置
#mysql -uroot -p"mysql123" --execute='show master status;
# mysql -uroot -p$"mysql123" --execute='show master status;'
在zabbixA 上
# mysql -uroot -p"mysql123"
mysql> CHANGE MASTER TO
-> MASTER_HOST='192.168.1.103',
-> MASTER_USER='repluser',
-> MASTER_PASSWORD='Repl@123',
-> MASTER_LOG_FILE='mysql-bin.000002',
-> MASTER_LOG_POS=50334;
mysql> START SLAVE;
mysql> SHOW SLAVE STATUS\G
四、zabbixB软件配置
1.数据库配置
# systemctl start mysqld.service
# systemctl enable mysqld.service
# cat /var/log/mysqld.log | grep password
# mysql -u root -p"XXX"
mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=1;
mysql> set password = password('mysql123');
mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbix123';
2.zabbix-server配置
# sed -i '/^ SourceIP/s/=.*$/=192.168.1.120/' /etc/zabbix/zabbix_server.conf
# sed -i '/^# DBHost/s/=.*$/=192.168.1.120/' /etc/zabbix/zabbix_server.conf
# sed -i '/^DBName/s/=.*$/=zabbix/' /etc/zabbix/zabbix_server.conf
# sed -i '/^# DBPassword/s/.*$/DBPassword=zabbix123/' /etc/zabbix/zabbix_server.conf
3.php配置
# sed -i 's/post_max_size = 8M/post_max_size = 32M/g' /etc/php.ini
# sed -i 's/upload_max_filesize = 2M/upload_max_filesize = 50M/g' /etc/php.ini
# sed -i 's/\;date.timezone =/date.timezone = Asia\/Shanghai/' /etc/php.ini
# sed -i 's/max_execution_time = 30/max_execution_time = 600/g' /etc/php.ini
# sed -i 's/max_input_time = 60/max_input_time = 600/g' /etc/php.ini
# sed -i 's/memory_limit = 128M/memory_limit = 256M/g' /etc/php.ini
# systemctl restart zabbix-server
# systemctl enable zabbix-server
# systemctl restart httpd
# systemctl enable httpd
4.keepalived配置
# vim /etc/keepalived/keepalived.conf
############################################
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check {
script "/etc/keepalived/check.sh"
interval 5
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 90
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
check
}
virtual_ipaddress {
192.168.1.120
}
#notify_master "/etc/keepalived/zabbix.sh start"
#notify_backup "/etc/keepalived/zabbix.sh stop"
}
##############################################
# vim /etc/keepalived/check.sh
##############################################
#!/bin/bash
zabbix_server_status1=$(ps -C zabbix_server --no-heading|wc -l)
if [ "${zabbix_server_status1}" = "0" ]; then
systemctl start zabbix_server.service
sleep 3
zabbix_server_status2=$(ps -C zabbix_server --no-heading|wc -l)
if [ "${zabbix_server_status2}" = "0" ]; then
systemctl restart keepalived.service
fi
fi
mysqld_status1=$(ps -C mysqld --no-heading|wc -l)
if [ "${mysqld_status1}" = "0" ]; then
systemctl start mysqld.service
sleep 3
mysqld_status2=$(ps -C mysqld --no-heading|wc -l)
if [ "${mysqld_status2}" = "0" ]; then
systemctl restart keepalived.service
fi
fi
httpd_status1=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status1}" = "0" ]; then
systemctl start httpd.service
sleep 3
httpd_status2=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status2}" = "0" ]; then
systemctl restart keepalived.service
fi
fi
##########################################
# chmod +x /etc/keepalived/check.sh
5. 文件同步设置
# vim /opt/data_realtime_rsync.sh
####################################################################
#!/bin/bash
# File name:data_realtime_sync.sh
# File path: /opt/data_realtime_sync.sh
src_dir="/etc/zabbix"
dest_dir="/etc/zabbix"
src_ip="192.168.1.103"
dest_ip="192.168.1.102"
. /etc/init.d/functions
cd ${src_dir}
inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |
while read file;do
INO_EVENT=$(echo $file | awk '{print $1}')
INO_FILE=$(echo $file | awk '{print $2}')
if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
if [[ $INO_EVENT =~ 'ATTRIB' ]];then
if [ ! -d "$INO_FILE" ];then
for ip in ${dest_ip};do
rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
fi
if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then
for ip in ${dest_ip};do
rsync -avzcR --delete -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
done
fi
done
#####################################################################
# chmod +x /opt/data_realtime_rsync.sh
# vim /etc/systemd/system/rsync-inotity.service
###############################################
[Unit]
Description=File instant rsync script
[Service]
ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh &
[Install]
WantedBy=multi-user.target
#################################################
6.数据库主主同步配置
# vim /etc/my.cnf
######################################################
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
server-id=2
log-bin=mysql-bin
auto-increment-increment = 2
auto-increment-offset = 2
binlog_format = mixed
sync_binlog=1
binlog-do-db=zabbix
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
binlog-ignore-db=information_schema
replicate-do-db=zabbix
replicate-ignore-db=mysql
replicate-ignore-db=sys
replicate-ignore-db=performance_schema
replicate-ignore-db=information_schema
######################################################
# systemctl restart mysqld
登录zabbixA的MySQL
创建数据同步账号
# mysql -u root -p"mysql123"
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repluser'@'%' IDENTIFIED BY 'Repl@123';
mysql> grant all privileges on zabbix.* to zabbix@192.168.1.120 identified by 'zabbix123';
mysql> FLUSH PRIVILEGES;
# exit
获取zabbixA二进制日志名和位置
#mysql -uroot -p"mysql123" --execute='show master status;
# mysql -uroot -p$"mysql123" --execute='show master status;'
在zabbixB上
# mysql -uroot -p"mysql123"
mysql> CHANGE MASTER TO
-> MASTER_HOST='192.168.1.102',
-> MASTER_USER='repluser',
-> MASTER_PASSWORD='Repl@123',
-> MASTER_LOG_FILE='mysql-bin.000002',
-> MASTER_LOG_POS=50334;
mysql> START SLAVE;
mysql> SHOW SLAVE STATUS\G
五、zabbixA zabbixB服务启动
# systemctl start zabbix-server
# systemctl enable zabbix-server
# systemctl start rsync-inotify
# systemctl enable rsync-inotify
# cat /etc/zabbix/zabbix_server.conf | grep 120
# cat /etc/zabbix/zabbix_server.conf | grep -Ev "^#|^$"
# vim /etc/zabbix/web/zabbix.conf.php
六、访问测试
七、参考
inotify+rsync+mysql主主复制+keepalived实现zabbix高可用
http://blog.51cto.com/3241766/2137303
Keepalived中Master和Backup主备切换机制浅析
http://blog.51cto.com/3241766/2097483
Zabbix高可用，实现zabbix的无缝切换，无故障时间
http://blog.51cto.com/yigemeng/1738174
实现Zabbix的高可用
https://www.longlong.asia/2016/10/29/zabbix-ha.html
keepalived + rsync +inotify 实现真正的高效数据实时同步
https://fandenggui.com/post/keepalived-rsync-inotify.html
真正的inotify+rsync实时同步 彻底告别同步慢
http://www.ttlsa.com/web/let-infotify-rsync-fast
Zabbix HA Requirment
https://kasperdeng.github.io/zabbix-ha-on-aws
Configuring the Zabbix server for high availability
