HA相关概念：

可用性=平均无故障时间/(平均无故障时间+修复时间) //一般要到达5个9至6个9的可用性；
可通过缩短修复时间提供可用性；
缩短修复时间的方式：提供备用主机，实现Failover(故障转移)
1.转移ip地址，即loating ip(流动的ip地址)
2.转移服务

特定情况下需要转移追踪信息和数据
A.基于rsync+inotify同步
B.共享存储
备用主机如何知道主节点不可用？
主节点周期性向备用节点发送heartbeat(心跳信息)
补充：
SAN:块级别接口(一块硬盘)，后端有两台主机都挂载了该SAN，当两台主机同一时刻操作的是同一个文件时，文件系统可能会崩溃；
NAS:文件系统级别
当后端主机使用的SAN的共享存储时，一台主机发生问题，为了防止其自动修复后会争用存储，通常需要用补刀设备，确保出问题的主机无法正常工作；
STONITH：shooting the other node in the head 切断电源
主机都是通过交换机接入到共享存储的，所以在交换机上切断其连接存储的网线也可实现；

Failback：故障夺回；备用主机的性能可能很差，当主节点修复后，应该立即夺回资源；

HA Cluster实现方案：

vrrp协议的实现：keepalived
ais完备HA集群：RHCS(cman)，heartbeat，corosync

Vrrp协议：virtual redundant routing protocol 虚拟冗余路由协议

什么是vrrp协议？
将多个物理路由器虚拟成一个或多个虚拟路由器来使用，每个虚拟路由有自己的标识(VRID)，虚拟路由内部的物理设备有主节点和备用节点之分，主节点不停的向备用节点发送心跳和优先级信息，备用节点通过心跳信息判定主节点是否有问题，根据优先级决定是否强主节点的资源，抢还是不抢取决于备用节点工作在抢占模式还是非抢占模式下；
抢占式：自己的优先级比别人高，就抢；
非抢占式：自己的优先级比别人高，只要对方能正常工作，就不抢
Vrrp协议仅转移ip地址；

术语：
虚拟路由器：virtual router
虚拟路由器标识：VRID(0-255)
物理路由器：
Master:主设备
Backup：备用设备
Priority：优先级
VIP:virtual ip
VMC:virtual MAC
Keepalived：
vrrp协议的软件实现，只能工作在linux上，实现ip地址漂移；
根据配置生成ipvs规则；
后端主机的健康状态监测；
原生目的高可用lvs；
组件：
Vrrp stack,ipvs wrapper,checkers,配置文件分析器，io复用器，内存管理组件；

配置HA Cluster配置前提：

1.各节点时间同步
2.确保iptables和selinux不会成为阻碍
3.各节点可通过主机名互相通信(对KA并非必须)，本地hosts文件
4.各节点基于秘钥认证的ssh服务完成通信(并非必须)

Keepalived安装配置：

CentOS6.4+ 随base仓库提供；
程序环境：
主配置文件：/etc/keepalived/keepalived.conf ``
GLOBAL CONFIGURATION
全局定义
静态路由和地址的相关配置
VRRPD CONFIGURATION
Vrrp同步组；
vrrp instance 对应一个虚拟路由实例
LVS CONFIGURATION
Ipvs集群的vs和rs

配置文件示例：

全局配置段：
global_defs {
   notification_email {
root@localhost
     //定义收件人地址
   }
   notification_email_from Alexandre.Cassen@firewall.loc //定义发件人地址
   smtp_server 192.168.200.1 //邮件服务器地址
   smtp_connect_timeout 30 //连接邮件服务器的超时时常
   router_id LVS_DEVEL //当前物理路由器的标识
   vrrp_mcast_group4 224-239.x.x.x //组播地址
}

虚拟路由配置段：
vrrp_instance VI_1 {  //定义一个虚拟路由实例
    state MASTER 
    interface eth0 //发送heartbeat的网卡
    virtual_router_id 51 //虚拟路由器的id
    priority 100 //优先级
    advert_int 1 //通告时间间隔
    authentication {   //认证
        auth_type PASS
        auth_pass 1111
    }   
    virtual_ipaddress {    //vip地址
        192.168.200.16/24 dev eno16777736 //vip地址配置在哪块网卡的别名 
    }   
}   
其他可用的配置指令：
nopreempt 非抢占模式
preempt_delay 300 抢占模式下，节点上线后触发新选举操作的延迟时常；
定义通知脚本：
#notify_master 当前节点成为主节点触发的脚本
notify_backup 当前节点转为备用节点时触发的脚本
notify_fault 当前节点转为失败状态时触发的脚本

实践：Vrrp主备模型实现ip地址漂移：

环境：
各节点时间同步，确保iptables及selinux不会成为阻碍
Node1(CentOS 7)172.18.20.7 MASTER
Node2(CentOS 7)172.18.20.8 BACKUP
VIP 172.18.20.100/16
vrrp_mcast_group4 224.0.0.100

配置node 1：

[root@localhost keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
    root@localhost
   }
    
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id node1
   vrrp_mcast_group4 224.0.0.100
}

vrrp_instance myroute {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1 
    authentication { 
        auth_type PASS
        auth_pass ck2384
    }
    virtual_ipaddress {
    172.18.20.100/16 dev ens33
}
}

配置node 2

[root@localhost keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
    root@localhost
   }
    
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id node2
   vrrp_mcast_group4 224.0.0.100
}

vrrp_instance myroute {
    state BACKUP
    interface eno16777736
    virtual_router_id 51
    priority 98
    advert_int 1 
    authentication { 
        auth_type PASS
        auth_pass ck2384
    }
    virtual_ipaddress {
    172.18.20.100/16 dev eno16777736
}
}

测试：

1.首先启动node2
[root@node2 ~]# systemctl status keepalived 
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-05-30 15:00:36 CST; 6s ago
  Process: 5004 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 5005 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─5005 /usr/sbin/keepalived -D
           ├─5006 /usr/sbin/keepalived -D
           └─5007 /usr/sbin/keepalived -D

May 30 15:00:36 node2 Keepalived_vrrp[5007]: Opening file '/etc/keepalived/keepalived.conf'.
May 30 15:00:36 node2 Keepalived_vrrp[5007]: Configuration is using : 62909 Bytes
May 30 15:00:36 node2 Keepalived_vrrp[5007]: Using LinkWatch kernel netlink reflector...
May 30 15:00:36 node2 Keepalived_vrrp[5007]: VRRP_Instance(myroute) Entering BACKUP STATE
May 30 15:00:36 node2 Keepalived_vrrp[5007]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 30 15:00:40 node2 Keepalived_vrrp[5007]: VRRP_Instance(myroute) Transition to MASTER STATE
May 30 15:00:41 node2 Keepalived_vrrp[5007]: VRRP_Instance(myroute) Entering MASTER STATE
May 30 15:00:41 node2 Keepalived_vrrp[5007]: VRRP_Instance(myroute) setting protocol VIPs.
May 30 15:00:41 node2 Keepalived_vrrp[5007]: VRRP_Instance(myroute) Sending gratuitous ARPs on eno16777736 for 172.18.20.100
May 30 15:00:41 node2 Keepalived_healthcheckers[5006]: Netlink reflector reports IP 172.18.20.100 added
[root@node2 ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:cf:ba:90 brd ff:ff:ff:ff:ff:ff
    inet 172.18.20.8/16 brd 172.18.255.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 172.18.20.100/16 scope global secondary eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fecf:ba90/64 scope link 
       valid_lft forever preferred_lft forever

2.启动node1，vip配置在了master node1上

[root@node1 ~]# systemctl start keepalived 
[root@node1 ~]# systemctl status keepalived 
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-05-30 03:02:18 EDT; 4s ago
  Process: 49646 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 49647 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─49647 /usr/sbin/keepalived -D
           ├─49648 /usr/sbin/keepalived -D
           └─49649 /usr/sbin/keepalived -D

May 30 03:02:18 node1 Keepalived_vrrp[49649]: Opening file '/etc/keepalived/keepalived.conf'.
May 30 03:02:18 node1 Keepalived_vrrp[49649]: Configuration is using : 62887 Bytes
May 30 03:02:18 node1 Keepalived_vrrp[49649]: Using LinkWatch kernel netlink reflector...
May 30 03:02:18 node1 Keepalived_vrrp[49649]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 30 03:02:19 node1 Keepalived_vrrp[49649]: VRRP_Instance(myroute) Transition to MASTER STATE
May 30 03:02:19 node1 Keepalived_vrrp[49649]: VRRP_Instance(myroute) Received lower prio advert, forcing new election
May 30 03:02:20 node1 Keepalived_vrrp[49649]: VRRP_Instance(myroute) Entering MASTER STATE
May 30 03:02:20 node1 Keepalived_vrrp[49649]: VRRP_Instance(myroute) setting protocol VIPs.
May 30 03:02:20 node1 Keepalived_vrrp[49649]: VRRP_Instance(myroute) Sending gratuitous ARPs on ens33 for 172.18.20.100
May 30 03:02:20 node1 Keepalived_healthcheckers[49648]: Netlink reflector reports IP 172.18.20.100 added
[root@node1 ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e4:53:15 brd ff:ff:ff:ff:ff:ff
    inet 172.18.20.7/16 brd 172.18.255.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 172.18.252.52/16 brd 172.18.255.255 scope global secondary dynamic ens33
       valid_lft 71521sec preferred_lft 71521sec
    inet 172.18.20.100/16 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::87aa:10b2:67e3:70d0/64 scope link 
       valid_lft forever preferred_lft forever

3.在node2上抓包，可以收到主节点的通告信息
[root@node2 ~]# tcpdump -i eno16777736 -nn host 224.0.0.100
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16777736, link-type EN10MB (Ethernet), capture size 65535 bytes
15:04:21.217999 IP 172.18.20.7 > 224.0.0.100: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:04:22.221621 IP 172.18.20.7 > 224.0.0.100: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
15:04:23.227296 IP 172.18.20.7 > 224.0.0.100: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20