Docker方式部署redis-cluster

前面我们手工部署了redis-cluster集群,可以看到步骤烦多,现在使用docker的方式部署,非常简单,共分为二步。
参考镜像https://github.com/publicisworldwide/docker-stacks/tree/master/oracle-linux/environments/storage/redis-cluster

Docker方式部署redis-cluster步骤
1、redis容器初始化
2、redis容器集群配置

一、redis容器初始化
容器初始化,使用docker-compose方式,先创建一个docker-compose.yml文件,内容如下:

version: '3'

services:
 redis1:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /data/redis/8001/data:/data
  environment:
   - REDIS_PORT=8001

 redis2:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /data/redis/8002/data:/data
  environment:
   - REDIS_PORT=8002

 redis3:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /data/redis/8003/data:/data
  environment:
   - REDIS_PORT=8003

 redis4:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /data/redis/8004/data:/data
  environment:
   - REDIS_PORT=8004

 redis5:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /data/redis/8005/data:/data
  environment:
   - REDIS_PORT=8005

 redis6:
  image: publicisworldwide/redis-cluster
  network_mode: host
  restart: always
  volumes:
   - /data/redis/8006/data:/data
  environment:
   - REDIS_PORT=8006

这里引用了别人的一个镜像publicisworldwide/redis-cluster,方便快捷。
这里使用host(主机)网络模式,把redis数据挂载到本机目录/data/redis/800*下。

若不想使用host模式,也可以把network_mode去掉,但就要加ports映射。
redis-cluster的节点端口共分为2种,一种是节点提供服务的端口,如6379;一种是节点间通信的端口,固定格式为:10000+6379。

version: '3'

services:
 redis1:
  image: publicisworldwide/redis-cluster
  restart: always
  volumes:
   - /data/redis/8001/data:/data
  environment:
   - REDIS_PORT=8001
  ports:
    - '8001:8001'       #服务端口
    - '18001:18001'   #集群端口

 redis2:
  image: publicisworldwide/redis-cluster
  restart: always
  volumes:
   - /data/redis/8002/data:/data
  environment:
   - REDIS_PORT=8002
  ports:
    - '8002:8002'
    - '18002:18002'

 redis3:
  image: publicisworldwide/redis-cluster
  restart: always
  volumes:
   - /data/redis/8003/data:/data
  environment:
   - REDIS_PORT=8003
  ports:
    - '8003:8003'
    - '18003:18003'

 redis4:
  image: publicisworldwide/redis-cluster
  restart: always
  volumes:
   - /data/redis/8004/data:/data
  environment:
   - REDIS_PORT=8004
  ports:
    - '8004:8004'
    - '18004:18004'

 redis5:
  image: publicisworldwide/redis-cluster
  restart: always
  volumes:
   - /data/redis/8005/data:/data
  environment:
   - REDIS_PORT=8005
  ports:
    - '8005:8005'
    - '18005:18005'

 redis6:
  image: publicisworldwide/redis-cluster
  restart: always
  volumes:
   - /data/redis/8006/data:/data
  environment:
   - REDIS_PORT=8006
  ports:
    - '8006:8006'
    - '18006:18006'

创建文件后,直接启动服务

窗口模式
docker-compose up
后台进程
docker-compose up -d

查看启动的进程

[root@localhost redis-cluster]# docker-compose ps
        Name                       Command               State   Ports
----------------------------------------------------------------------
rediscluster_redis1_1   /usr/local/bin/entrypoint. ...   Up           
rediscluster_redis2_1   /usr/local/bin/entrypoint. ...   Up           
rediscluster_redis3_1   /usr/local/bin/entrypoint. ...   Up           
rediscluster_redis4_1   /usr/local/bin/entrypoint. ...   Up           
rediscluster_redis5_1   /usr/local/bin/entrypoint. ...   Up           
rediscluster_redis6_1   /usr/local/bin/entrypoint. ...   Up           

状态为Up,说明服务均已启动,镜像无问题。
注意:以上镜像不能设置永久密码,其实redis一般是内网访问,可以不需密码。

二、redis容器集群配置
上面只是启动了6个redis容器,并没有设置集群,通过下面的命令可以设置集群。

docker run --rm -it inem0o/redis-trib create --replicas 1 172.19.165.222:8001 172.19.165.222:8002 172.19.165.222:8003 172.19.165.222:8004 172.19.165.222:8005 172.19.165.222:8006

这里同样使用了另一个镜像inem0o/redis-trib,执行时会自动下载。
日志如下:

[root@localhost disconf]# docker run --rm -it inem0o/redis-trib create --replicas 1 172.19.165.222:8001 172.19.165.222:8002 172.19.165.222:8003 172.19.165.222:8004 172.19.165.222:8005 172.19.165.222:8006
Unable to find image 'inem0o/redis-trib:latest' locally
latest: Pulling from inem0o/redis-trib
a2b2998a36ab: Pull complete 
a3ed95caeb02: Pull complete 
46ab6b64c08e: Pull complete 
3d82c3ac2025: Pull complete 
Digest: sha256:0b89d25b387f70ef1c54605bdf061dd86e0833dbc0e2149390570b8b372278f8
Status: Downloaded newer image for inem0o/redis-trib:latest
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.19.165.222:8001
172.19.165.222:8002
172.19.165.222:8003
Adding replica 172.19.165.222:8004 to 172.19.165.222:8001
Adding replica 172.19.165.222:8005 to 172.19.165.222:8002
Adding replica 172.19.165.222:8006 to 172.19.165.222:8003
M: 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950 172.19.165.222:8001
   slots:0-5460 (5461 slots) master
M: 206626063f31dcd7e69010ce13c258e786197f1e 172.19.165.222:8002
   slots:5461-10922 (5462 slots) master
M: e9924018d95772b8ff535f6bc0605a6630837069 172.19.165.222:8003
   slots:10923-16383 (5461 slots) master
S: 548f4e65fbab8dcde8aac187849d50983d68599d 172.19.165.222:8004
   replicates 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950
S: 0a5c799c1f8fed083c50902639fc354e4c25aa8c 172.19.165.222:8005
   replicates 206626063f31dcd7e69010ce13c258e786197f1e
S: 94e2530ddd05b0e9eb3e71a9616342bd6647a5e6 172.19.165.222:8006
   replicates e9924018d95772b8ff535f6bc0605a6630837069
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 172.19.165.222:8001)
M: 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950 172.19.165.222:8001
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 94e2530ddd05b0e9eb3e71a9616342bd6647a5e6 172.19.165.222:8006@18006
   slots: (0 slots) slave
   replicates e9924018d95772b8ff535f6bc0605a6630837069
S: 0a5c799c1f8fed083c50902639fc354e4c25aa8c 172.19.165.222:8005@18005
   slots: (0 slots) slave
   replicates 206626063f31dcd7e69010ce13c258e786197f1e
M: e9924018d95772b8ff535f6bc0605a6630837069 172.19.165.222:8003@18003
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 206626063f31dcd7e69010ce13c258e786197f1e 172.19.165.222:8002@18002
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 548f4e65fbab8dcde8aac187849d50983d68599d 172.19.165.222:8004@18004
   slots: (0 slots) slave
   replicates 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

可以看到设置了3个Master,3个Slave,若想测试下集群,可以参考【Redis-Cluster集群

只需二步就完成了集群的部署,但上面是使用了别人的镜像,若是要自己创建镜像该如何呢?

手动创建镜像
编写Dockerfile

#基础镜像
FROM redis
#修复时区
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' >/etc/timezone
#环境变量
ENV REDIS_PORT 8000
#ENV REDIS_PORT_NODE 18000
#暴露变量
EXPOSE $REDIS_PORT
#EXPOSE $REDIS_PORT_NODE
#复制
COPY entrypoint.sh /usr/local/bin/
COPY redis.conf /usr/local/etc/
#for config rewrite
RUN chmod 777 /usr/local/etc/redis.conf
RUN chmod +x /usr/local/bin/entrypoint.sh
#入口
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
#命令
CMD ["redis-server", "/usr/local/etc/redis.conf"]

编写shell文件entrypoint.sh

#!/bin/sh
#只作用于当前进程,不作用于其创建的子进程
set -e
#$0--Shell本身的文件名 $1--第一个参数 $@--所有参数列表
# allow the container to be started with `--user`
if [ "$1" = 'redis-server' -a "$(id -u)" = '0' ]; then
    sed -i 's/REDIS_PORT/'$REDIS_PORT'/g' /usr/local/etc/redis.conf
    chown -R redis .  #改变当前文件所有者
    exec gosu redis "$0" "$@"  #gosu是sudo轻量级”替代品”
fi

exec "$@"

编写redis.conf

#端口
port REDIS_PORT
#开启集群
cluster-enabled yes
#配置文件
cluster-config-file nodes.conf
cluster-node-timeout 5000
#更新操作后进行日志记录
appendonly yes
#设置主服务的连接密码
# masterauth
#设置从服务的连接密码
# requirepass

注意:

  • requirepass和masterauth不能启用,否则redis-trib创建集群失败。
  • protected-mode 保护模式是禁止公网访问,但是不能设置密码和bind ip。

以上编写了3个文件,其实可以创建镜像了,不过也可以在docker-compose中自动创建。
编写docker-compose.yml文件

version: '3'

services:
 redis1:
  build: ./
  restart: always
  volumes:
   - /data/redis/8001/data:/data
  environment:
   - REDIS_PORT=8001
  ports:
    - '8001:8001'       #服务端口
    - '18001:18001'   #集群端口

 redis2:
  build: ./
  restart: always
  volumes:
   - /data/redis/8002/data:/data
  environment:
   - REDIS_PORT=8002
  ports:
    - '8002:8002'
    - '18002:18002'

 redis3:
  build: ./
  restart: always
  volumes:
   - /data/redis/8003/data:/data
  environment:
   - REDIS_PORT=8003
  ports:
    - '8003:8003'
    - '18003:18003'

 redis4:
  build: ./
  restart: always
  volumes:
   - /data/redis/8004/data:/data
  environment:
   - REDIS_PORT=8004
  ports:
    - '8004:8004'
    - '18004:18004'

 redis5:
  build: ./
  restart: always
  volumes:
   - /data/redis/8005/data:/data
  environment:
   - REDIS_PORT=8005
  ports:
    - '8005:8005'
    - '18005:18005'

 redis6:
  build: ./
  restart: always
  volumes:
   - /data/redis/8006/data:/data
  environment:
   - REDIS_PORT=8006
  ports:
    - '8006:8006'
    - '18006:18006'

创建镜像

窗口模式
docker-compose up
后台进程
docker-compose up -d
重新创建
docker-compose up --build

若创建成功,可以使用命令登录并查看集群信息。

[root@localhost src]# ./redis-cli -c -p 8003
127.0.0.1:8003> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

从上面的日志里看到,当前集群状态失败。

与上面相同,重新执行上面的第二步,再次查看结果如下:

127.0.0.1:8003> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:39
cluster_stats_messages_pong_sent:37
cluster_stats_messages_meet_sent:4
cluster_stats_messages_sent:80
cluster_stats_messages_ping_received:34
cluster_stats_messages_pong_received:43
cluster_stats_messages_meet_received:3
cluster_stats_messages_received:80

以上信息显示集群状态ok。

设置集群密码
上面提供了publicisworldwide/redis-cluster的镜像不能设置永久密码,是因为它没有给redis.conf写入权限,在我们重创建的镜像里是有写入权限的,所以可以创建永久密码。
集群构建完毕后再通过config set + config rewrite命令逐个机器设置密码
在之前的镜像上设置密码会有权限问题,如下:

127.0.0.1:8001> config set masterauth 1111
OK
127.0.0.1:8001> config set requirepass 1111
OK
127.0.0.1:8001> auth
(error) ERR wrong number of arguments for 'auth' command
127.0.0.1:8001> auth 1111
OK
127.0.0.1:8001> config rewrite
(error) ERR Rewriting config file: Permission denied

在新的镜像上添加密码,如下:

[root@localhost src]# ./redis-cli -c -p 8001
127.0.0.1:8001> config set masterauth 1111
OK
127.0.0.1:8001> config set requirepass 1111
OK
127.0.0.1:8001> config rewrite
(error) NOAUTH Authentication required.
127.0.0.1:8001> auth 1111
OK
127.0.0.1:8001> config rewrite
OK

当重登录8001机器并get数据时就会提示需要密码了。其它机器也要一并设置密码才行。

查看密码文件
是否将密码写入到配置文件redis.conf,可以登录到容器查看即可。

docker exec -it 容器ID /bin/bash
cat /usr/local/etc/redis.conf

可以看到masterauth和requirepass被追加到文件的最后,即使重启密码也还生效。
在单机上部署意义不大,可以把slave结点放到不同的服务器上,再通过docker的redis-trib容器来连接,十分灵活和方便。

异常处理
在第三步执行可能会现以下异常,如下:

[root@localhost redis-cluster]# docker run --rm -it inem0o/redis-trib create --replicas 1 172.19.165.222:8001 172.19.165.222:8002 172.19.165.222:8003 172.19.165.222:8004 172.19.165.222:8005 172.19.165.222:8006
>>> Creating cluster
[ERR] Node 172.19.165.222:8001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

这个异常是因为在目录下/data/redis/800*已经存在redis的配置文件,要先清除掉。

[root@localhost redis-cluster]# rm /data/redis/800* -rf