- FileName：
KeepAlived+Haproxy集群.txt- Function:
Implement the load balancer cluster via KeepAlived and Haproxy- Version:
V1.0(trial version)- ChangeLog:
2015/08/27 yzhantong.com(Internal test passed)
0、规划
网络环境：
192.168.146.220 VIP
eth0:192.168.146.221 node1.mycluster.com
eth1:10.0.0.221(暂时未用上，如果涉及到集群内网环境，可能数据库、文件服务器可能会用上)
eth0:192.168.146.222 node2.mycluster.com
eth1:10.0.0.222
操作系统：
CentOS release 6.6 (Final) x86_64
1、在两个节点上设置hosts文件
[root@node1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.146.221 node1.mycluster.com
192.168.146.222 node2.mycluster.com
[root@node2 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.146.221 node1.mycluster.com
192.168.146.222 node2.mycluster.com
2、在两个节点上安装HAProxy and Keepalived
[root@node1 ~]# yum -y install haproxy keepalived
[root@node2 ~]# yum -y install haproxy keepalived
已加载插件：fastestmirror
设置安装进程
Loading mirror speeds from cached hostfile
- base: mirrors.btte.net
- extras: mirrors.btte.net
- updates: mirrors.btte.net
解决依赖关系
--> 执行事务检查
---> Package haproxy.i686 0:1.5.2-2.el6 will be 安装
---> Package keepalived.i686 0:1.2.13-5.el6_6 will be 安装
--> 处理依赖关系 libnl.so.1，它被软件包 keepalived-1.2.13-5.el6_6.i686 需要
--> 执行事务检查
---> Package libnl.i686 0:1.1.4-2.el6 will be 安装
--> 完成依赖关系计算
依赖关系解决
====================================================================================================================================
软件包 架构 版本 仓库 大小
====================================================================================================================================
正在安装:
haproxy i686 1.5.2-2.el6 base 787 k
keepalived i686 1.2.13-5.el6_6 updates 209 k
为依赖而安装:
libnl i686 1.1.4-2.el6 base 124 k
事务概要
Install 3 Package(s)
总下载量：1.1 M
Installed size: 3.4 M
下载软件包：
(1/3): haproxy-1.5.2-2.el6.i686.rpm | 787 kB 00:00
(2/3): keepalived-1.2.13-5.el6_6.i686.rpm | 209 kB 00:00
(3/3): libnl-1.1.4-2.el6.i686.rpm | 124 kB 00:00
总计 5.7 MB/s | 1.1 MB 00:00
运行 rpm_check_debug
执行事务测试
事务测试成功
执行事务
正在安装 : libnl-1.1.4-2.el6.i686 1/3
正在安装 : keepalived-1.2.13-5.el6_6.i686 2/3
正在安装 : haproxy-1.5.2-2.el6.i686 3/3
Verifying : libnl-1.1.4-2.el6.i686 1/3
Verifying : keepalived-1.2.13-5.el6_6.i686 2/3
Verifying : haproxy-1.5.2-2.el6.i686 3/3
已安装:
haproxy.i686 0:1.5.2-2.el6 keepalived.i686 0:1.2.13-5.el6_6
作为依赖被安装:
libnl.i686 0:1.1.4-2.el6
完毕！
3、两节点确认两服务随系统启动
[root@node1 ~]# chkconfig haproxy on && chkconfig keepalived on
[root@node1 ~]# chkconfig | egrep 'haproxy|keepalived'
haproxy 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭
keepalived 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭
[root@node2 ~]# chkconfig haproxy on && chkconfig keepalived on
[root@node2 ~]# chkconfig | egrep 'haproxy|keepalived'
haproxy 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭
keepalived 0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭
4、允许non-local Virtual IPs在所有节点上
[root@node1 ~] vi /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[root@node1 ~]# sysctl -p
[root@node2 ~] vi /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[root@node2 ~]# sysctl -p
5、配置haproxy
[root@node1 haproxy]# pwd
/etc/haproxy
[root@node1 haproxy]# cat haproxy.cfg
---------------------------------------------------------------------
Example configuration for a possible web application. See the
full configuration options online.
http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
---------------------------------------------------------------------
---------------------------------------------------------------------
Global settings
---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
---------------------------------------------------------------------
common defaults that all the 'listen' and 'backend' sections will
use if not designated in their block
---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
---------------------------------------------------------------------
main frontend which proxys to the backends
---------------------------------------------------------------------
frontend main *:80
default_backend webservers
---------------------------------------------------------------------
static backend for serving up images, stylesheets and such
---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check
---------------------------------------------------------------------
round robin balancing between the various backends
---------------------------------------------------------------------
backend webservers
mode http
cookie webservers insert
stats enable
stats auth admin:admin
stats uri /haproxy?stats
balance roundrobin
option httpclose
option forwardfor
#server webserver1 10.0.0.222:8000 check
server webserver1 10.0.0.222:8000 cookie webserver1 check
#server webserver2 10.0.0.221:8000 check
server webserver2 10.0.0.221:8000 cookie webserver2 check
listen stats :8888
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth admin:Pa55wd@CM
[root@node2 haproxy]# pwd
/etc/haproxy
[root@node2 haproxy]# cat haproxy.cfg
---------------------------------------------------------------------
Example configuration for a possible web application. See the
full configuration options online.
http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
---------------------------------------------------------------------
---------------------------------------------------------------------
Global settings
---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
---------------------------------------------------------------------
common defaults that all the 'listen' and 'backend' sections will
use if not designated in their block
---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
---------------------------------------------------------------------
main frontend which proxys to the backends
---------------------------------------------------------------------
frontend main *:80
default_backend webservers
---------------------------------------------------------------------
static backend for serving up images, stylesheets and such
---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check
---------------------------------------------------------------------
round robin balancing between the various backends
---------------------------------------------------------------------
backend webservers
mode http
cookie webservers insert
stats enable
stats auth admin:admin
stats uri /haproxy?stats
balance roundrobin
option httpclose
option forwardfor
#server webserver1 10.0.0.222:8000 check
server webserver1 10.0.0.222:8000 cookie webserver1 check
#server webserver2 10.0.0.221:8000 check
server webserver2 10.0.0.221:8000 cookie webserver2 check
listen stats :8888
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth admin:Pa55wd@CM
6、配置keepalved
[root@node1 haproxy]# cd /etc/keepalived/
[root@node1 keepalived]# pwd
/etc/keepalived
[root@node1 keepalived]# ll
total 8
-rw-r--r-- 1 root root 1172 Jun 10 15:28 keepalived.conf
-rw-r--r--. 1 root root 3562 Jun 2 14:50 keepalived.conf.default
[root@node1 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_haproxy {
script "killall -0 haproxy" # verify the pid existance
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface eth0 # interface to monitor
state MASTER
virtual_router_id 51 # Assign one ID for this route
priority 101 # 101 on master, 100 on backup
authentication {
auth_type PASS
auth_pass VI_1
}
virtual_ipaddress {
192.168.146.220 # the virtual IP
}
track_script {
chk_haproxy
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 58
priority 92
advert_int 1
authentication {
auth_type PASS
auth_pass VI_2
}
virtual_ipaddress {
192.168.146.223
}
}
[root@node2 keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_haproxy {
script "killall -0 haproxy" # verify the pid existance
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface eth0 # interface to monitor
state BACKUP
virtual_router_id 51 # Assign one ID for this route
priority 100 # 101 on master, 100 on backup
authentication {
auth_type PASS
auth_pass VI_1
}
virtual_ipaddress {
192.168.146.220 # the virtual IP
}
track_script {
chk_haproxy
}
}
vrrp_instance VI_2 {
state MASTER
interface eth0
virtual_router_id 58
priority 92
advert_int 1
authentication {
auth_type PASS
auth_pass VI_2
}
virtual_ipaddress {
192.168.146.223
}
track_script {
chk_haproxy
}
}
7、两节点启动haproxy、keepalived服务
service haproxy start
service keepalived start
[root@node1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:19:b9:f1:d2:25 brd ff:ff:ff:ff:ff:ff
inet 192.168.146.221/24 brd 192.168.146.255 scope global eth0
inet 192.168.146.220/32 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:19:b9:f1:d2:27 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.221/8 brd 10.255.255.255 scope global eth1
[root@node2 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 84:2b:2b:19:f5:ca brd ff:ff:ff:ff:ff:ff
inet 192.168.146.222/24 brd 192.168.146.255 scope global eth0
inet 192.168.146.223/32 scope global eth0
inet6 fe80::862b:2bff:fe19:f5ca/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 84:2b:2b:19:f5:cb brd ff:ff:ff:ff:ff:ff
inet 10.0.0.222/8 brd 10.255.255.255 scope global eth1
inet6 fe80::862b:2bff:fe19:f5cb/64 scope link
valid_lft forever preferred_lft forever
8、两台安装nginx，测试HA与LB
两节点配置文件内容一样，在主目录放测试文件
[root@node1 conf.d]# cat default.conf
server {
listen 8000;
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/log/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.php index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
root /usr/share/nginx/html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
#fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
WEB:
http://192.168.146.220/
haproxy:
http://192.168.146.220:8888/
登录信息在配置文件中
9、日常维护
9.1 Keepalive
服务管理
[root@node1 ~]# service keepalived {start|stop|status|restart|condrestart|try-restart|reload|force-reload}
or
[root@node1 ~]# /etc/init.d/keepalived {start|stop|status|restart|condrestart|try-restart|reload|force-reload}VIP绑定
[root@node1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:19:b9:f1:d2:25 brd ff:ff:ff:ff:ff:ff
inet 192.168.146.221/24 brd 192.168.146.255 scope global eth0
inet 192.168.146.220/32 scope global eth0keepalived运行日志
[root@node1 ~]# cat /var/log/messages|grep -i Keepalived
Aug 27 16:19:17 node1 Keepalived_vrrp[1620]: VRRP_Instance(VI_1) sending 0 priority
Aug 27 16:19:17 node1 Keepalived_vrrp[1620]: VRRP_Instance(VI_1) removing protocol VIPs.
Aug 27 16:19:17 node1 Keepalived[1617]: Stopping Keepalived v1.2.13 (03/19,2015)
Aug 27 16:19:18 node1 Keepalived[27675]: Starting Keepalived v1.2.13 (03/19,2015)
Aug 27 16:19:18 node1 Keepalived[27676]: Starting Healthcheck child process, pid=27678
Aug 27 16:19:18 node1 Keepalived[27676]: Starting VRRP child process, pid=27679
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Netlink reflector reports IP 192.168.146.221 added
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Netlink reflector reports IP 10.0.0.221 added
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Registering Kernel netlink reflector
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Registering Kernel netlink command channel
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Registering gratuitous ARP shared channel
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Configuration is using : 69407 Bytes
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: Using LinkWatch kernel netlink reflector...
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: VRRP_Instance(VI_2) Entering BACKUP STATE
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: VRRP_Script(chk_haproxy) succeeded
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 27 16:19:18 node1 Keepalived_vrrp[27679]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Aug 27 16:19:19 node1 Keepalived_vrrp[27679]: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 27 16:19:19 node1 Keepalived_vrrp[27679]: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 27 16:19:19 node1 Keepalived_vrrp[27679]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.146.220
Aug 27 16:19:24 node1 Keepalived_vrrp[27679]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.146.220
9.2 Haproxy
- 服务管理
service haproxy {start|stop|status|restart|try-restart|reload|force-reload}
or
/etc/init.d/haproxy {start|stop|status|restart|try-restart|reload|force-reload}
- 运行日志
[root@node1 ~]# tail -f /var/log/haproxy.log
Netlink reflector reports IP 192.168.146.221 added
Netlink reflector reports IP 10.0.0.221 added
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Opening file '/etc/keepalived/keepalived.conf'.
Configuration is using : 8029 Bytes
Using LinkWatch kernel netlink reflector...
Netlink reflector reports IP 192.168.146.220 added
[root@node1 ~]# tail -f /var/log/haproxy-status.log
Server webservers/node1 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 3ms. 5 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Server webservers/node1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server webservers/node4 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Proxy main started.
Proxy webservers started.
Proxy stats started.
Server webservers/node1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server webservers/node2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server webservers/node4 is DOWN, reason: Layer7 timeout, check duration: 10002ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server webservers/node4 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 22ms. 4 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
10、Q&A
- message日志中报错：
Jun 2 15:02:39 node1 modprobe: FATAL: Error inserting ip_vs (/lib/modules/2.6.32-504.16.2.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Jun 2 15:02:39 node1 Keepalived_healthcheckers[12807]: IPVS: Can't initialize ipvs: Protocol not available
Jun 2 15:02:39 node1 kernel: ip_vs: Unknown symbol icmpv6_send
Jun 2 15:02:39 node1 Keepalived[14868]: Healthcheck child process(12807) died: Respawning
Jun 2 15:02:39 node1 Keepalived[14868]: Starting Healthcheck child process, pid=12810
Jun 2 15:02:39 node1 kernel: ip_vs: Unknown symbol ip6_local_out
Jun 2 15:02:39 node1 kernel: ip_vs: Unknown symbol ip6_route_me_harder
Jun 2 15:02:39 node1 kernel: ip_vs: Unknown symbol ipv6_dev_get_saddr
Jun 2 15:02:39 node1 kernel: ip_vs: Unknown symbol ip6_route_output
查看是否禁用了IPV6
[root@node2 keepalived]# cat /etc/modprobe.d/ipv6.conf
install ipv6 /bin/true
[root@node2 keepalived]# vi /etc/modprobe.d/ipv6.conf
install ipv6 /bin/true
11、引用
