20171029 KeepAlived

一、高可用集群

(一)提升系统高可用性的解决方案:冗余(redundant)

  • 工作模式

    • active/passive:主备
    • active/active:双主
  • 以心跳方式通告

    • active --> HEARTBEAT --> passive
    • active <--> HEARTBEAT <--> active
  • 故障处理

    • failover:故障切换,即某资源的主节点故障时,将资源转移至其它节点的操作
    • failback:故障移回,即某资源的主节点故障后重新修改上线后,将之前已转移至其它节点的资源重新切回的过程

(二)HA Cluster实现方案

  • ais:应用接口规范完备复杂的HA集群
    RHCS:Red Hat Cluster Suite红帽集群套件
    heartbeat
    corosync

  • vrrp协议实现:虚拟路由冗余协议
    keepalived

二、KeepAlived基本介绍

(一)VRRP(Virtual Router Redundancy Protocol)协议术语

  • 虚拟路由器:Virtual Router,多个物理路由器对外以一个IP地址提供服务,仿佛一台路由器

    • 虚拟路由器标识:VRID(0-255),唯一标识虚拟路由器
    • VIP:Virtual IP,虚拟IP
    • VMAC:Virutal MAC (00-00-5e-00-01-VRID),虚拟MAC
  • 物理路由器
    master:主设备
    backup:备用设备
    priority:优先级

(二)KeepAlived的工作特性

  • 通告:心跳,优先级等;周期性

  • 工作方式:抢占式,非抢占式

  • 安全认证:

    • 无认证
    • 简单字符认证:预共享密钥
    • MD5
  • 工作模式:

    • 主/备:单虚拟路由器
    • 主/主:主/备(虚拟路由器1),备/主(虚拟路由器2)

(三)KeepAlived的功能

  • vrrp协议完成地址流动

  • 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)

  • 为ipvs集群的各RS做健康状态检测

  • 基于脚本调用接口通过执行脚本完成脚本中定义的功能,进而影响集群事务,以此支持nginx, haproxy等服务

三、KeepAlived的配置

(一)HA Cluster配置准备:

  • 各节点时间必须同步:ntp服务(CentOS 6), chrony(CentOS 7)

    // 由于ntp/chrony服务不能同步差距过大的时间,需要先使用ntpdate命令同步一次,再开启服务
    ntpdate ntp_server_ip
    // 开启chronyd服务(CentOS 7)
    vim /etc/chrony.conf
    server 172.18.0.1 iburst
    systemctl enable chronyd
    systemctl start chronyd
    // 开启ntp服务(CentOS 6)
    vim /etc/ntp.conf
    server 172.18.0.1 iburst
    chkconfig ntpd on
    service ntpd start
    
  • 确保iptables及selinux不会成为阻碍

  • 各节点之间可通过主机名互相通信(对KA并非必须),建议使用/etc/hosts文件实现

  • 各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信(对KA并非必须)

    ssh-keygen
    ssh-copy-id destination_ip
    

(二)KeepAlived的程序环境

  • 主配置文件:/etc/keepalived/keepalived.conf

  • 主程序文件:/usr/sbin/keepalived

  • Unit File:/usr/lib/systemd/system/keepalived.service

  • Unit File的环境配置文件:/etc/sysconfig/keepalived

(三)KeepAlived的配置文件结构

  • GLOBAL CONFIGURATION:全局设置
    Global definitions
    Static routes/addresses

  • VRRPD CONFIGURATION:VRRP设置
    VRRP synchronization group(s):vrrp同步组
    VRRP instance(s):即一个vrrp虚拟路由器

  • LVS CONFIGURATION:LVS设置
    Virtual server group(s)
    Virtual server(s):ipvs集群的vs和rs

(四)配置虚拟路由器

  • 语法:

    vrrp_instance <STRING> {
    ....
    }
    
  • 专用参数:

    • state MASTER | BACKUP
      当前节点在此虚拟路由器上的初始状态;只能有一个是MASTER,余下的都应该为BACKUP
    • interface IFACE_NAME
      绑定为当前虚拟路由器使用的物理接口
    • virtual_router_id VRID
      当前虚拟路由器惟一标识,范围是0-255
    • priority 100
      当前物理节点在此虚拟路由器中的优先级;范围1-254
    • advert_int 1
      vrrp通告的时间间隔,默认1s
    • authentication:认证机制
    authentication {
    auth_type AH|PASS
    auth_pass <PASSWORD> 仅前8位有效
    }
    
    • virtual_ipaddress:虚拟IP
    virtual_ipaddress { 
    <IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL>
    }
    
    • track_interface:配置监控网络接口,一旦出现故障,则转为FAULT状态实现地址转移
    track_interface {
    eth0
    eth1
    …
    }
    
    • nopreempt:定义工作模式为非抢占模式
    • preempt_delay 300:抢占式模式,节点上线后触发新选举操作的延迟时长,默认模式
    • 定义通知脚本:
      notify_master <STRING> | <QUOTED-STRING>:
      当前节点成为主节点时触发的脚本
      notify_backup <STRING> | <QUOTED-STRING>:
      当前节点转为备节点时触发的脚本
      notify_fault <STRING> | <QUOTED-STRING>:
      当前节点转为“失败”状态时触发的脚本
      notify <STRING> | <QUOTED-STRING>:
      通用格式的通知触发机制,一个脚本可完成以上三种状态的转换时的通知
  • 实验1:实现主/备虚拟路由器

    • 实验环境:
      物理路由器1:ip: 192.168.136.230, 主机名: node1, MASTER
      物理路由器2:ip: 192.168.136.130, 主机名: node2, BACKUP
      VIP:192.168.136.100
    // 配置物理路由器1
    vim /etc/keepalived/keepalived.conf
    global_defs {
       notification_email {
         root@localhost
       }
       notification_email_from node1@localhost
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node1                  // vrrp中的路由器主机名
       vrrp_mcast_group4 224.0.0.58     // 设置组播ip地址
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface ens37
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6          // openssl rand -hex 4 生成8位16进制密码
        }
        virtual_ipaddress {
            192.168.136.100/24
        }
    }
    
    systemctl start keepalived
    
    // 配置物理路由器2
    vim /etc/keepalived/keepalived.conf
    global_defs {
       notification_email {
         root@localhost
       }
       notification_email_from node2@localhost
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node2
       vrrp_mcast_group4 224.0.0.58
    }
    
    vrrp_instance VI_1 {
        state BACKUP
        interface ens37
        virtual_router_id 51
        priority 90                    //作为BACKUP优先级比MASTER要低
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6         // 密码与node1相同
        }
        virtual_ipaddress {
           192.168.136.100/24
        }
    }
    
    systemctl start keepalived
    
    • 测试
      node1的ip地址已经出现VIP

    监听组播地址的tcp连接tcpdump -i ens37 -nn host 224.0.0.58,此时关闭node1的keepalived服务systemctl stop keepalived,自动由node2接管并开始声明自身拥有虚拟路由器的IP

    VIP此时已经被node2接管

  • 实验2:实现keepalived日志
vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 3"    // -D:详细日志,-S 3: 设置日志facility为local3
vim /etc/rsyslog.conf 
local3.*               /var/log/keepalived.log    // 设置日志存储路径
systemctl restart rsyslog
systemctl restart keepalived
tail -f  /var/log/keepalived.log
  • 实验3:实现主/主虚拟路由器,并且当节点发生变化时主动发送邮件

    • 实验环境
      物理路由器1:ip: 192.168.136.230, 主机名: node1
      物理路由器2:ip: 192.168.136.130, 主机名: node2
      虚拟路由器1:MASTER: node1, BACKUP: node2, VIP: 192.168.136.100
      虚拟路由器2:MASTER: node2, BACKUP: node1, VIP: 192.168.136.200
    // 配置物理路由器1(虚拟路由器1的MASTER,虚拟路由器2的BACKUP)
    vim /etc/keepalived/keepalived.conf
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node1
       vrrp_mcast_group4 224.0.0.58
    }
    // 虚拟路由器1的设置
    vrrp_instance VI_1 {
        state MASTER
        interface ens37
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6
        }
        virtual_ipaddress {
            192.168.136.100/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    }
    // 虚拟路由器2的设置
    vrrp_instance VI_2 {
        state BACKUP
        interface ens37
        virtual_router_id 61
        priority 80
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass a56c19be
        }
        virtual_ipaddress {
            192.168.136.200/24
       }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    }
    systemctl restart keepalived
    
    // 配置物理路由器2(虚拟路由器1的BACKUP,虚拟路由器2的MASTER)
    vim /etc/keepalived/keepalived.conf
    global_defs {
       notification_email {
       root@localhost
       }
       notification_email_from node2@localhost
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node2
       vrrp_mcast_group4 224.0.0.58
    }
    // 虚拟路由器1的设置
    vrrp_instance VI_1 {
        state BACKUP
        interface ens37
        virtual_router_id 51
        priority 90
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6
        }
        virtual_ipaddress {
           192.168.136.100/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    }
    // 虚拟路由器2的设置
    vrrp_instance VI_2 {
        state MASTER
        interface ens37
        virtual_router_id 61
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass a56c19be
        }
        virtual_ipaddress {
            192.168.136.200/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    }
    
    // 在物理路由器1,2上添加脚本文件
    vim /etc/keepalived/notify.sh
    #! /bin/bash
    
    contact='root@localhost'
    notify() {
            mailsubject="$(hostname) to be $1, vip floating"
            mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
            echo "$mailbody" | mail -s "$mailsubject" $contact
    }
    
    case $1 in
    master)
            notify master
            ;;
    backup)
            notify backup
            ;;
    fault)
            notify fault
            ;;
    *)
            echo "Usage: $(basename $0) {master|backup|fault}"
            exit 1
            ;;
    esac
    chmod +x /etc/keepalived/notify.sh
    
    • 测试
      监听组播地址的tcp连接tcpdump -i ens37 -nn host 224.0.0.58,可以看到node1, node2分别声明拥有虚拟路由器1(vrid 51)、虚拟路由器2(vrid61)的IP地址

    分别查看node1和node2的网卡IP地址,进一步确认上述结果

    此时,断开node1的网络连接
    虚拟路由器1的VIP立即由node2的网卡接管

    恢复node1的网络连接,在node1和node2上都可以看到相应的邮件通知:
    node1上通知出错,很快通知自身被切换为BACKUP,恢复网络连接后通知自身重新变为MASTER;

    node2上通知自身切换为MASTER,恢复网络连接后通知自身切换为BACKUP

(五)Keepalived支持IPVS

  • 语法:
virtual_server {IP port | fwmark int}
{
    ...
    real_server{
        ...
    }
    ...
}
  • virtual_server常用参数

    • delay_loop <INT>
      检查后端服务器的时间间隔
    • lb_algo rr|wrr|lc|wlc|lblc|sh|dh
      定义调度方法
    • lb_kind NAT|DR|TUN
      集群的类型
    • persistence_timeout <INT>
      持久连接时长
    • protocol TCP
      服务协议,仅支持TCP
    • sorry_server<IPADDR> <PORT>
      所有RS故障时,备用服务器地址
  • real_server <IPADDR> <PORT>常用参数

    • weight <INT>
      RS权重
    • notify_up <STRING>|<QUOTED-STRING>
      RS上线通知脚本
    • notify_down <STRING>|<QUOTED-STRING>
      RS下线通知脚本
    • HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK { ... }
      定义当前主机的健康状态检测方法
  • HTTP_GET|SSL_GET:应用层健康状态检测

    HTTP_GET|SSL_GET {
    url {
    path <URL_PATH>               // 定义要监控的URL
    status_code <INT>             // 判断上述检测机制为健康状态的响应码
    digest <STRING>               // 判断为健康状态的响应的内容的校验码
    }
    connect_timeout <INTEGER>     // 连接请求的超时时长
    nb_get_retry <INT>            // 重试次数
    delay_before_retry <INT>      // 重试之前的延迟时长
    connect_ip <IP ADDRESS>       // 向当前RS哪个IP地址发起健康状态检测请求
    connect_port <PORT>           // 向当前RS的哪个PORT发起健康状态检测请求
    bindto <IP ADDRESS>           // 发出健康状态检测请求时使用的源地址
    bind_port <PORT>              // 发出健康状态检测请求时使用的源端口
    }
    
  • TCP_CHECK参数

    • connect_ip <IP ADDRESS>
      向当前RS的哪个IP地址发起健康状态检测请求
    • connect_port <PORT>
      向当前RS的哪个PORT发起健康状态检测请求
    • bindto <IP ADDRESS>
      发出健康状态检测请求时使用的源地址
    • bind_port <PORT>
      发出健康状态检测请求时使用的源端口
    • connect_timeout <INTEGER>
      连接请求的超时时长
  • 实验4:实现主/备模型的IPVS集群

    • 实验环境:
      LB1(master)/VS:IP: 192.168.136.230
      LB2(backup)/VS:IP: 192.168.136.130
      VIP:192.168.136.100
      RS1:IP: 192.168.136.229
      RS2:IP: 192.168.136.129
    // 配置LB1的keepalived设置
    vim /etc/keepalived/keepalived.conf
    global_defs {
       notification_email {
         root@localhost
       }
       notification_email_from node1@localhost
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node1
       vrrp_mcast_group4 224.0.0.58
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface ens37
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6
        }
        virtual_ipaddress {
            192.168.136.100/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    }
    
    virtual_server 192.168.136.100 80{
        delay_loop 3
        lb_algo wrr
        lb_kind DR
        protocol TCP
        sorry_server 127.0.0.1 80
        real_server 192.168.136.229 80{
            weight 2
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
            }
        }
        real_server 192.168.136.129 80{
            weight 1
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
            }
        }
    }
    
    // 配置LB2的keepalived设置
    vim /etc/keepalived/keepalived.conf
    global_defs {
       notification_email {
         root@localhost
       }
       notification_email_from node2@localhost
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node2
       vrrp_mcast_group4 224.0.0.58
    }
    
    vrrp_instance VI_1 {
        state BACKUP
        interface ens37
        virtual_router_id 51
        priority 90
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6
        }
        virtual_ipaddress {
           192.168.136.100/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    }
    
    virtual_server 192.168.136.100 80{
        delay_loop 3
        lb_algo wrr
        lb_kind DR
        protocol TCP
        sorry_server 127.0.0.1 80
        real_server 192.168.136.229 80{
            weight 2
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
            }
        }
        real_server 192.168.136.129 80{
            weight 1
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
            }
        }
    }
    
    // 配置LB1, LB2的sorry server服务
    echo sorry on LB1 > /var/www/html/index.html     // LB1上操作
    echo sorry on LB2 > /var/www/html/index.html     // LB2上操作
    systemctl start httpd
    
    // 配置RS1, RS2的Web服务
    echo RS1 homepage > /var/www/html/index.html     // RS1上操作
    echo RS2 homepage > /var/www/html/index.html     // RS2上操作
    systemctl start httpd
    
    // 编辑脚本实现:禁止RS响应ARP请求,并将网卡绑定VIP
    vim lvs_dr_rs.sh
    #! /bin/bash
    vip='192.168.136.100'
    mask='255.255.255.255'
    dev=lo:1
    rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
    service httpd start &> /dev/null && echo "The httpd Server is Ready!"
    
    case $1 in
    start)
        echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
        echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
        ifconfig $dev $vip netmask $mask broadcast $vip up
        echo "The RS Server is Ready!"
        ;;
    stop)
        ifconfig $dev down
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo "The RS Server is Canceled!"
        ;;
    *)
        echo "Usage: $(basename $0) start|stop"
        exit 1
        ;;
    esac
    
    chmod +x lvs_dr_rs.sh
    bash lvs_dr_rs.sh start
    
    // LB1, LB2启动KeepAlived服务,进行测试
    systemctl start keepalived
    

    访问VIP(192.168.136.100)的Web服务,正常工作

    停止RS2的Web服务,自动进行健康检查,全部调度至RS1

    停止RS1的Web服务,自动进行健康检查,调度至LB1的sorry server

    停止LB1的KeepAlived服务,自动切换至LB2

  • 实验5:实现主/主模型的IPVS集群

    • 实验环境:
      LB1/VS1:IP: 192.168.136.230,后端RS: RS1, RS2
      LB2/VS2:IP: 192.168.136.130,后端RS: RS3, RS4
      LB1 VIP:192.168.136.100
      LB2 VIP:192.168.136.200
      RS1:IP: 192.168.136.229
      RS2:IP: 192.168.136.129
      RS3:IP: 192.168.136.240
      RS4:IP: 192.168.136.250
      LB之间互为MASTER与BACKUP的关系
      MASTER:LB1,BACKUP:LB2
      MASTER:LB2,BACKUP:LB1
    // 配置LB1, LB2的keepalived设置
    global_defs {
       notification_email {
         root@localhost
       }
       notification_email_from node1@localhost     // LB1上操作
       notification_email_from node1@localhost     // LB2上操作
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node1                             // LB1上操作
       router_id node2                             // LB2上操作
       vrrp_mcast_group4 224.0.0.58
    }
    vrrp_instance VI_1 {
        state MASTER                               // LB1上操作
        state BACKUP                               // LB2上操作
        interface ens37
        virtual_router_id 51
        priority 100                               // LB1上操作
        priority 90                                // LB2上操作
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6
        }
        virtual_ipaddress {
            192.168.136.100/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    }
    vrrp_instance VI_2 {
        state BACKUP                               // LB1上操作
        state MASTER                               // LB2上操作
        interface ens37
        virtual_router_id 61
        priority 80                                // LB1上操作
        priority 100                               // LB2上操作
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass a56c19be
        }
        virtual_ipaddress {
            192.168.136.200/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    
    }
    virtual_server 192.168.136.100 80{
        delay_loop 3
        lb_algo wrr
        lb_kind DR
        protocol TCP
        sorry_server 127.0.0.1 80
        real_server 192.168.136.229 80{
            weight 2
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
            }
        }
    real_server 192.168.136.129 80{
            weight 1
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
            }
        }
    }
    virtual_server 192.168.136.200 80{
        delay_loop 3
        lb_algo wrr
        lb_kind DR
        protocol TCP
        sorry_server 127.0.0.1 80
        real_server 192.168.136.240 80{
            weight 2
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
            }
        }
        real_server 192.168.136.250 80{
            weight 1
            HTTP_GET {
                url {
                  path /
                  status_code 200
                }
                connect_timeout 1
                nb_get_retry 3
                delay_before_retry 1
           }
       }
    }
    
    // 配置LB1, LB2的sorry server服务
    echo sorry on LB1 > /var/www/html/index.html     // LB1上操作
    echo sorry on LB2 > /var/www/html/index.html     // LB2上操作
    systemctl start httpd
    
    // 配置RS1, RS2, RS3, RS4的Web服务
    echo RS1 homepage > /var/www/html/index.html     // RS1上操作
    echo RS2 homepage > /var/www/html/index.html     // RS2上操作
    echo RS3 homepage > /var/www/html/index.html     // RS3上操作
    echo RS4 homepage > /var/www/html/index.html     // RS4上操作
    systemctl start httpd
    
    // 编辑脚本实现:禁止RS响应ARP请求,并将网卡绑定VIP
    vim lvs_dr_rs.sh
    #! /bin/bash
    vip='192.168.136.100'                            // RS1, RS2上操作
    vip='192.168.136.200'                            // RS3, RS4上操作
    mask='255.255.255.255'
    dev=lo:1
    rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
    service httpd start &> /dev/null && echo "The httpd Server is Ready!"
    
    case $1 in
    start)
        echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
        echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
        ifconfig $dev $vip netmask $mask broadcast $vip up
        echo "The RS Server is Ready!"
        ;;
    stop)
        ifconfig $dev down
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo "The RS Server is Canceled!"
        ;;
    *)
        echo "Usage: $(basename $0) start|stop"
        exit 1
        ;;
    esac
    
    chmod +x lvs_dr_rs.sh
    bash lvs_dr_rs.sh start
    
    // LB1, LB2启动KeepAlived服务,进行测试
    systemctl start keepalived
    

使用ipvsadm -Ln命令查看ipvs调度策略,与KeepAlived的配置吻合

访问VIP1, VIP2(192.168.136.100, 192.168.136.200)的Web服务,正常工作

停止RS1的Web服务,自动进行健康检查,全部调度至RS2

停止RS2的Web服务,自动进行健康检查,调度至LB1的sorry server

停止LB1的KeepAlived服务,自动切换至LB2

停止RS3的Web服务,自动进行健康检查,全部调度至RS4

停止RS4的Web服务,自动进行健康检查,调度至LB2的sorry server

(六)Keepalived调用脚本进行资源监控

  • keepalived调用外部的辅助脚本进行资源监控,并根据监控的结果状态实现优先动态调整

  • vrrp_script:自定义资源监控脚本,vrrp实例根据脚本返回值,公共定义,可被多个实例调用,定义在vrrp实例之外

  • track_script:调用vrrp_script定义的脚本去监控资源,定义在实例之内,调用事先定义的vrrp_script

    • 分两步:(1) 先定义一个脚本;(2) 调用此脚本
      格式:
    // 定义脚本,定义在实例外
    vrrp_script <SCRIPT_NAME> {
        script ""     // 引号内为脚本命令
        interval INT
        weight -INT
    }
    // 调用脚本,定义在实例内
    track_script {
        SCRIPT_NAME_1
        SCRIPT_NAME_2
    }
    
  • 实验6:实现主/主模型的高可用Nginx反向代理

    • 实验环境:
      LB1/VS1:IP: 192.168.136.230,后端RS: RS1, RS2
      LB2/VS2:IP: 192.168.136.130,后端RS: RS3, RS4
      LB1 VIP:192.168.136.100
      LB2 VIP:192.168.136.200
      RS1:IP: 192.168.136.229
      RS2:IP: 192.168.136.129
      RS3:IP: 192.168.136.240
      RS4:IP: 192.168.136.250
      LB之间互为MASTER与BACKUP的关系
      MASTER:LB1,BACKUP:LB2
      MASTER:LB2,BACKUP:LB1
    // 配置LB1, LB2的KeepAlived设置
    vim /etc/keepalived/keepalived.conf
    global_defs {
       notification_email {
         root@localhost
       }
       notification_email_from node1@localhost     // LB1上操作
       notification_email_from node2@localhost     // LB2上操作
       smtp_server 127.0.0.1
       smtp_connect_timeout 30
       router_id node1                             // LB1上操作
       router_id node2                             // LB2上操作
       vrrp_mcast_group4 224.0.0.58
    }
    
    vrrp_script chk_nginx {
            script "killall -0 nginx && exit 0 || exit 1;"
            interval 1
            weight -20
            fall 3
            rise 3
    }
    vrrp_instance VI_1 {
        state MASTER                               // LB1上操作
        state BACKUP                               // LB2上操作
        interface ens37
        virtual_router_id 51
        priority 100                               // LB1上操作
        priority 90                                // LB2上操作
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass dd73f9d6
        }
        virtual_ipaddress {
            192.168.136.100/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    // 下面的脚本引用仅在LB1的配置文件出现
        track_script {
            chk_nginx
        }
    }
    vrrp_instance VI_2 {
        state BACKUP                               // LB1上操作
        state MASTER                               // LB2上操作
        interface ens37
        virtual_router_id 61
        priority 90                                // LB1上操作
        priority 100                               // LB2上操作
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass a56c19be
        }
        virtual_ipaddress {
            192.168.136.200/24
        }
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
    // 下面的脚本引用仅在LB2的配置文件出现
        track_script {
            chk_nginx
        }
    }
    
    // 配置LB1,LB2的nginx反向代理
    vim /etc/nginx/nginx.conf
    http {
        upstream websrvs1 {
            server 192.168.136.229:80 weight=2;
            server 192.168.136.129:80 weight=1;
        }
        upstream websrvs2 {
            server 192.168.136.240:80 weight=2;
            server 192.168.136.250:80 weight=1;
        }
        server {
            listen  192.168.136.100:80;
            location / {
                    proxy_pass http://websrvs1;
            }
        }
        server {
            listen  192.168.136.200:80;
            location / {
                    proxy_pass http://websrvs2;
            }
        }
    }
    nginx -t
    systemctl start nginx
    
    // 配置RS1, RS2, RS3, RS4的Web服务
    echo RS1 homepage > /var/www/html/index.html     // RS1上操作
    echo RS2 homepage > /var/www/html/index.html     // RS2上操作
    echo RS3 homepage > /var/www/html/index.html     // RS3上操作
    echo RS4 homepage > /var/www/html/index.html     // RS4上操作
    systemctl start httpd
    
    // LB1, LB2启动KeepAlived服务,进行测试
    systemctl start keepalived
    

    登录192.168.136.100和192.168.136.200的web服务,确实按照设置要求调度

    停止RS2的httpd服务,全部调度至RS1

    停止RS3的httpd服务,全部调度至RS4

    关闭LB2的nginx反向代理服务,通过tcpdump -i ens37 -nn host 224.0.0.58查看组播情况。三个红框依次表达:
    (1)未关闭nginx前的组播状态
    (2)关闭nginx后,LB2的vrid 61权重减去20变作80,而LB1vrid 61的权重为90
    (3)由于LB1的权重高,VIP2的所有权被LB1接管

    关闭LB1的nginx反向代理服务,通过tcpdump -i ens37 -nn host 224.0.0.58查看组播情况。三个红框依次表达:
    (1)未关闭nginx前的组播状态
    (2)关闭nginx后,LB1的vrid 51权重减去20变作80,而LB2的vrid 51权重为90
    (3)由于LB2的权重高,VIP1的所有权被LB2接管

    由于此时两个nginx反向代理均关闭,故访问192.168.136.100和192.168.136.200的web服务全部失败

    打开LB2的nginx反向代理服务,通过tcpdump -i ens37 -nn host 224.0.0.58查看组播情况。三个红框依次表达:
    (1)未打开nginx前的组播状态
    (2)打开nginx后,LB2的vrid 61权重增加20变作100,而LB1的vrid 61权重为90
    (3)由于LB2的权重高,VIP2的所有权被LB2接管

    此时VIP1和VIP2均由LB2上的nginx服务器进行反向代理,192.168.136.100和192.168.136.200的web服务全部恢复

(七)Keepalived同步组

  • LVS NAT模型VIP和DIP需要同步,需要同步组

  • 格式:

    vrrp_sync_group VG_1 {
      group {
          VI_1 # name of vrrp_instance(below)
          VI_2 # One for each moveable IP.
      }
    }
    vrrp_instance VI_1 {
      eth0
      vip
    }
    vrrp_instance VI_2 {
      eth1
      dip
    }
    

推荐阅读更多精彩内容