haproxy+keepalived反向代理,查看状态,后端主机上下线配置

本文描述的是haproxy做代理,负载均衡,haproxy状态查看,后端主机的上下线,keepalived高可用

一、前提条件

1.1 至少两台服务器 操作系统为centos7

IP:192.168.238.138/24机器名:ceph4

IP:192.168.238.139/24机器名:ceph5

1.2 部署软件(两台都有)

Keepalived高可用

Haproxy1.7.9反向代理

Apache HTTP后端主机

1.3 在两台主机设置分别机器名

#hostnamectl ceph4

#hostnamectl ceph4

1.4编辑vi /etc/hosts中加入(两台都有)

#加入本机的域名解析

192.168.238.138 ceph4

192.168.238.139 ceph5

1.5 关闭防火墙,selinux(两台都有)

#systemctl stop firewalld#停止正在运行的防火墙

#systemctl disabled firewalld#禁止开机启动

#setenforce 0#临时禁止selinux

#sed -i "s/^SELINUX\=enforcing/SELINUX\=disabled/g"/etc/selinux/config#永久禁止


二、安装配置后端软件apache http

2.1 安装http软件(两台都有)

#yum install httpd -y

2.2 编辑配置文件改监听的端口

安装完编辑vi /etc/httpd/conf/httpd.conf

Listen 8080 #监听端口改为8080,可以不修改,我的是主机的80端口已经被占用

2.3 配置访问的页面

Ceph4

#echo 'ceph4' >/var/www/html/index.html

Ceph5:

#echo 'ceph5' >/var/www/html/index.html

2.4 启动http并测试

#systemctl start httpd

#curl ceph4:8080

ceph4#结果不同,以方便之后的测试

#curl ceph5:8080

ceph5


三、Haproxy安装部署

下边的操作两个节点都需要安装

3.1 相关系统包的安装

#yum install -y gcc glibc gcc-c++ make screen tree lrzsz

3.2 Haproxy安装

#mkdir /soft#创建目录

#cd /soft/

#wgethttp://www.haproxy.org/download/1.7/src/haproxy-1.7.9.tar.gz#下载安装包

#tar xf haproxy-1.7.9.tar.gz#解压

#cd haproxy-1.7.9

#make TARGET=linux2628 PREFIX=/usr/local/haproxy1.7.9#编译

#make install#安装

install -d "/usr/local/sbin"

install haproxy"/usr/local/sbin"

install -d "/usr/local/share/man"/man1

install -m 644 doc/haproxy.1 "/usr/local/share/man"/man1

install -d "/usr/local/doc/haproxy"

for x in configuration management architecture cookie-options luaWURFL-device-detection proxy-protocol linux-syn-cookies network-namespacesDeviceAtlas-device-detection 51Degrees-device-detectionnetscaler-client-ip-insertion-protocol close-options SPOE intro; do \

install -m 644doc/$x.txt "/usr/local/doc/haproxy" ; \

#cp /usr/local/sbin/haproxy /usr/sbin/#启动文件

#haproxy-v#查看安装结果

HA-Proxy version 1.7.9 2017/08/18

Copyright 2000-2017 Willy Tarreauwilly@haproxy.org

创建haproxy启动脚本

#cp examples/haproxy.init /etc/init.d/haproxy

#/etc/init.d/haproxy start#启动

创建需要的相关的目录

#useradd -r haproxy

#mkdir /etc/haproxy

#mkdir /var/lib/haproxy

#mkdir /var/run/haproxy

编辑haproxy配置文件

#vi /etc/haproxy/haproxy.cfg

global

log 127.0.0.1 local3 info

chroot /var/lib/haproxy

maxconn10000#设置允许的最大连接数,需要考虑ulimit -n的限制

user haproxy

group haproxy

daemon

defaults

log global

mode http

option httplog

option dontlognull

timeout connect 5000

timeout client 50000

timeout server 50000

frontend front#前端

mode http

bind *:8088#这里的端口为8088,也可以是其他为占用的端口

stats uri /haproxy?stats

default_backend default_backend

backend default_backend#后端

#source cookie SERVERID

option forwardfor header X-REAL-IP

option httpchk GET /index.html#检查的url

balance roundrobin

server ceph5 192.168.238.139:8080 check inter 2000 rise 3 fall 3 weight1

server ceph4 192.168.238.138:8080 check inter 2000 rise 3 fall 3 weight1

日志设置

#sed -i 's@\#\$ModLoad imudp@\$ModLoad imudp@g' /etc/rsyslog.conf

#sed-i 's@\#\$UDPServerRun514@\$UDPServerRun 514@g' /etc/rsyslog.conf

#echo "local3.*/var/log/haproxy.log" >> /etc/rsyslog.conf

启动:

#/etc/init.d/haproxy start

Startinghaproxy (via systemctl):[OK]

测试:

# ceph5的haproxy配置正常

[root@ceph4 ~]# curlceph5:8088

ceph5

[root@ceph4 ~]# curlceph5:8088

ceph4

# ceph4的haproxy配置正常

[root@ceph4 ~]# curlceph4:8088

ceph5

[root@ceph4 ~]# curlceph4:8088

ceph4

看到访问url的结果是两台服务器轮换相应

状态管理页面

在浏览器访问http://192.168.238.138:8088/haproxy?stats,查看状态

3.3 Haproxy动态维护(两点都需要)

在配置文件的global下添加socket文件

stats socket /var/lib/haproxy/haproxy.sockmode 600 level admin

stats timeout 2m

安装socat

#yum install -y socat

查看haproxy的帮助

#echo "help" |socat stdio /var/lib/haproxy/haproxy.sock

查看info状态信息,可以通过zabbix来监控相关状态值

#echo "show info" |socat stdio/var/lib/haproxy/haproxy.sock

Name: HAProxy

Version: 1.7.9

Release_date: 2017/08/18

Nbproc: 1

Process_num: 1

Pid: 5145

Uptime: 0d 0h03m34s

Uptime_sec: 214

Memmax_MB: 0

PoolAlloc_MB: 0

PoolUsed_MB: 0

PoolFailed: 0

Ulimit-n: 20033

Maxsock: 20033

Maxconn: 10000

Hard_maxconn: 10000

CurrConns: 0

CumConns: 4

CumReq: 4

Maxpipes: 0

PipesUsed: 0

PipesFree: 0

ConnRate: 0

ConnRateLimit: 0

MaxConnRate: 0

SessRate: 0

SessRateLimit: 0

MaxSessRate: 0

CompressBpsIn: 0

CompressBpsOut: 0

CompressBpsRateLim: 0

Tasks: 7

Run_queue: 1

Idle_pct: 100

node: ceph4

haproxy维护模式(主机上下线)


在ceph4上做测试,下线default_backend下的ceph4主机

#echo "disable server default_backend/ceph4 " |socat stdio/var/lib/haproxy/haproxy.sock

注:ceph4已经不在线

上线default_backend下的ceph4

#echo "enable server default_backend/ceph4 " |socat stdio/var/lib/haproxy/haproxy.sock

注:ceph4恢复

3.4 Haproxy生产使用建议

haproxy的本地端口会出现用尽情况,解决方案如下4条

1.更改local的端口范围,调整内核参数

#cat /proc/sys/net/ipv4/ip_local_port_range

3276861000

2.调整timewait的端口复用,设置为1

#cat /proc/sys/net/ipv4/tcp_tw_reuse

1

3.调整tcp_wait的时间,不建议修改

#cat /proc/sys/net/ipv4/tcp_fin_timeout

60

4.最佳方案:增加多个ip,端口数量就足够




四、Keepalived

Mail配置使用

4.1安装mailx邮件服务

yum install mailx -y

#配置文件追加信息(/etc/mail.rc)

vim /etc/mail.rc

#发件人信息

set from=zhouguanjie2005@163.com#发件人邮箱地址(163设置得开起允许代理)

set smtp=smtp.163.com#smtp地址

setsmtp-auth-user=zhouguanjie2005@163.com#邮箱用户名,不用加域名

set smtp-auth-password=******#邮箱密码(邮件密码是smtp代理授权码)

set smtp-auth=login#邮箱验证方式

#测试发送

echo "hello world" | mail -s"hello"18706768942@163.comzhuguanjie@qq.com#会看到测试邮件信息,可以发送多个邮件

#echo "邮件内容"

| mail -s "标题"邮箱地址

#最好把你的发送邮件地址加入你接收邮箱的白名单,不然发多了可能被认为发送垃圾邮件而被163拒绝,这是真的

4.2.安装配置keepalived(两节点都需要)

# yum install -y keepalived

# keepalived -v#查看版本

Keepalived v1.3.5 (03/19,2017), git commitv1.3.5-6-g6fa32f2

在/etc/keepalived下建立文件如下(两节点):

# ls

check_haproxy.shcheck_haproxy_url.shdown.shkeepalived.confvrrp.sh

#主要是一些脚本和keepalived配置文件

# vi check_haproxy.sh#检测haproxy进程是否村子,不存在的话重启

#!/bin/bash

counter=$(ps -C haproxy --no-heading|wc -l)

if [ "${counter}" = "0"]; then

/etc/init.d/haproxy start

fi

exit 0

# vi check_haproxy_url.sh#通过url检测如果不成功返回非0,待达到次数后,keepalived会降权值变为backup节点

#!/bin/bash

# curl -ILhttp://localhost/member/login.htm

# curl --data"memberName=fengkan&password=22" http://localhost/member/login.htm

count=0

for (( k=0; k<2; k++ ))

do

check_code=$( curl --connect-timeout 3 -sL -w"%{http_code}\\n" http://localhost:8088/index.html -o /dev/null )

if [ "$check_code" != "200" ]; then

# count = count +1

let "count += 1"

continue

else

count=0

break

fi

done

if [ "$count" != "0" ];then

#/etc/init.d/keepalived stop

exit 1

else

exit 0

fi

# vi down.sh#维护用的脚本,不需要手动关闭keepalived

#!/bin/bash

#判断down文件是否存在,在需要维护的时候,建立一个down文件,虚拟地址会自动转移走

if [-f /etc/keepalived/down ]; then

exit 1

else

exit 0

fi

# vi vrrp.sh (ceph4)#状态发生变换,邮件提醒

#!/bin/bash

#当状态发生变换的时候,发送邮件提醒

echo "192.168.238.138 ceph4$1状态被激活,请确认HAProxy服务运行状态"|mail -s "HAProxy状态切换警告"15063176713@139.com

# vi vrrp.sh (ceph5)#状态发生变换,邮件提醒

#!/bin/bash

echo "192.168.238.139 ceph5$1状态被激活,请确认HAProxy服务运行状态"|mail -s "HAProxy状态切换警告"15063176713@139.com

建立完脚本后不要忘记赋予可执行的权限

#chmod +x check_haproxy.sh check_haproxy_url.sh vrrp.sh down.sh

Keepalived主配置文件

这里ceph4为master节点,ceph5为backup节点

ceph4:

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

acassen

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_script chk_haproxy_url {

script "/etc/keepalived/check_haproxy_url.sh"#查看链接是否能正常访问,不正常两次后降级,看下边的配置

interval 2# check every 2 seconds

weight -5

fall 2#失败两次后,触发weight减5操作,想有降级操作必须有

rise 2#成功两次后,恢复

}

vrrp_script chk_haproxy {

script "/etc/keepalived/check_haproxy.sh"#查看haproxy进程是否存在,不存在的话启动,无降权

interval 2#check every 2 seconds,执行的时间间隔

}

vrrp_script chk_mantaince_down {

script "/etc/keepalived/down.sh"

interval 2# check every 2 seconds

weight -5

fall 2#维护操作命令,在/etc/keepalived建立down文件开始维护

rise 2

}

vrrp_instance VI_1 {

state MASTER#这里主备不一样,注意

interface ens33#根据自己的网卡修改

virtual_router_id 50

#nopreempt

priority 101#设置优先级

advert_int 1

virtual_ipaddress {

192.168.238.200#虚拟IP地址

}

track_script {

chk_haproxy_url#与上边的执行vrrp_script脚本对应

chk_haproxy

chk_mantaince_down

}

#状态转换的时候,邮件告警

notify_backup "/etc/keepalived/vrrp.shBACKUP"

notify_master "/etc/keepalived/vrrp.shMASTER"

notify_fault"/etc/keepalived/vrrp.shFAULT"

}

Ceph5

backup节点

#这里只标出与master不一样的地方,其他同上

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

acassen

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_script chk_haproxy_url {

script "/etc/keepalived/check_haproxy_url.sh"# cheaper than pidof

interval 2#check every 2 seconds

weight -5

fall 2

rise 2

}

vrrp_script chk_haproxy {

script "/etc/keepalived/check_haproxy.sh"# cheaper than pidof

interval 2#check every 2 seconds

}

vrrp_script chk_mantaince_down {

script "/etc/keepalived/down.sh"

interval 2#check every 2 seconds

weight -5

fall 2

rise 2

}

vrrp_instance VI_1 {

state BACKUP#这里为BACKUP

interface ens33

virtual_router_id 50

#nopreempt

priority 100#设置级别

advert_int 1

virtual_ipaddress {

192.168.238.200

}

track_script {

chk_haproxy_url

chk_haproxy

chk_mantaince_down

}

notify_backup "/etc/keepalived/vrrp.shBACKUP"

notify_master "/etc/keepalived/vrrp.shMASTER"

notify_fault"/etc/keepalived/vrrp.shFAULT"

}

测试:

分别启动keepalived

会看到

Ceph4日志:

# tailf /var/log/messages

Sep 21 15:09:55 ceph4 Keepalived[50677]:Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

Sep 21 15:09:55 ceph4 Keepalived[50677]:Unable to resolve default script username 'keepalived_script' - ignoring

Sep 21 15:09:55 ceph4 Keepalived[50677]:Opening file '/etc/keepalived/keepalived.conf'.

Sep 21 15:09:55 ceph4 systemd: PID file/var/run/keepalived.pid not readable (yet?) after start.

Sep 21 15:09:55 ceph4 Keepalived[50678]:Starting Healthcheck child process, pid=50679

Sep 21 15:09:55 ceph4 Keepalived[50678]:Starting VRRP child process, pid=50680

Sep 21 15:09:55 ceph4 systemd: Started LVSand VRRP High Availability Monitor.

Sep 21 15:09:55 ceph4Keepalived_healthcheckers[50679]: Opening file '/etc/keepalived/keepalived.conf'.

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering Kernel netlink reflector

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering Kernel netlink command channel

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering gratuitous ARP shared channel

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Opening file '/etc/keepalived/keepalived.conf'.

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) removing protocol VIPs.

Sep 21 15:09:55 ceph4 Keepalived_vrrp[50680]:Unsafe permissions found for script '/etc/keepalived/check_haproxy_url.sh'.

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: SECURITY VIOLATION - scripts are being executed butscript_security not enabled. There are insecure scripts.

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Using LinkWatch kernel netlink reflector...

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP sockpool: [ifindex(2), proto(112), unicast(0),fd(10,11)]

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) succeeded

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_haproxy) succeeded

Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_haproxy_url) succeeded

Sep 21 15:09:56 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Transition to MASTER STATE

Sep21 15:09:57 ceph4 Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Entering MASTERSTATE#现在虚拟地址在主节点上

Sep 21 15:09:57 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) setting protocol VIPs.

Sep 21 15:09:57 ceph4Keepalived_vrrp[50680]: Sending gratuitous ARP on ens33 for 192.168.238.200

测试一、现在我在ceph4(模拟不关机维护)

# touch /etc/keepalived/down#创建down文件

Sep 21 15:12:49 ceph4Keepalived_vrrp[50680]: /etc/keepalived/down.sh exited with status 1

Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: /etc/keepalived/down.sh exited with status 1

Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) failed

Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Changing effective priority from101 to 96

Sep 21 15:12:52 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Received advert with higherpriority 100, ours 96

Sep 21 15:12:52 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1)Entering BACKUP STATE#在创建down文件后,weight降级了,变为了BACKUP节点

Sep 21 15:12:52 ceph4 Keepalived_vrrp[50680]:VRRP_Instance(VI_1) removing protocol VIPs.

邮箱收到信息如下:

Ceph4变为backup



Ceph5变为激活master


Ceph5变为激活master

说明测试成功,地址已经漂移到ceph5,ceph4可以维护了

将down文件删掉后

# rmdown

rm:remove regular empty file ‘down’? y

ceph4

Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) succeeded

Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Changing effective priority from 96to 101

Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) forcing a new MASTER election

Sep 21 15:17:19 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Transition to MASTER STATE

Sep21 15:17:20 ceph4 Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Entering MASTERSTATE#变为主的状态

Sep 21 15:17:20 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) setting protocol VIPs.

Sep 21 15:17:20 ceph4Keepalived_vrrp[50680]: Sending gratuitous ARP on ens33 for 192.168.238.200

Ceph5:

Sep 21 15:17:18 ceph5Keepalived_vrrp[11531]: VRRP_Instance(VI_1) Received advert with higherpriority 101, ours 100

Sep 21 15:17:18 ceph5Keepalived_vrrp[11531]: VRRP_Instance(VI_1) Entering BACKUP STATE

Sep21 15:17:18 ceph5 Keepalived_vrrp[11531]: VRRP_Instance(VI_1) removing protocolVIPs.#地址已经漂移走了,回到了ceph4





推荐阅读更多精彩内容