×

Kubernetes集群和Docker私有库搭建(CentOS 7)

96
负二贷
2017.10.18 16:44* 字数 2658

环境规划

手里的环境是四台安装了CentOS 7的主机。环境规划如下:

Kubernetes Master 节点:192.168.20.145

Kubernetes Node 节点:192.168.20.146, 192.168.20.147

Docker私有库节点:192.168.20.148

每台主机上都运行了如下命令来关闭防火墙和启用ntp:

# systemctl stop firewalld

# systemctl disable firewalld

# yum -y install ntp

# systemctl start ntpd

# systemctl enable ntpd

Kubernetes Master节点的安装与配置

在Kubernetes Master节点上安装etcd, docker和Kubernetes

# yum -y install etcd docker kubernetes flannel

对etcd进行配置,编辑/etc/etcd/etcd.conf,内容如下:

ETCD_NAME=default

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

其中ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"表示etcd在2379端口上监听所有网络接口。

对Master节点上的Kubernetes进行配置,编辑配置文件/etc/kubernetes/config

KUBE_LOGTOSTDERR="--logtostderr=true"

KUBE_LOG_LEVEL="--v=0"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://192.168.20.145:8080"

KUBE_MASTER="--master=http://192.168.20.145:8080"是将Kubernetes的apiserver进程的服务地址告诉Kubernetes的controller-manager, scheduler和proxy进程。

编辑配置文件/etc/kubernetes/apiserver

KUBE_API_ADDRESS="--address=0.0.0.0"

KUBE_API_PORT="--port=8080"

KUBELET_PORT="--kubelet-port=10250"

KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

KUBE_API_ARGS=""

默认配置文件是没有去掉ServiceAccount,这里一定要去掉,不然创建pod会失败

这些配置让apiserver进程在8080端口上监听所有网络接口,并告诉apiserver进程etcd服务的地址。

对flannel进行配置,编辑/etc/sysconfig/flanneld,内容如下:

FLANNEL_ETCD="http://192.168.20.145:2379"

FLANNEL_ETCD_KEY="/atomic.io/network"

现在,启动Kubernetes Master节点上的etcd, docker, apiserver, controller-manager和scheduler进程并查看其状态:

# for SERVICES  in etcd docker kube-apiserver kube-controller-manager kube-scheduler flanneld;  do

systemctl restart $SERVICES

systemctl enable $SERVICES

systemctl status $SERVICES

done

在etcd里定义flannel网络配置:

# etcdctl mk /atomic.io/network/config '{"Network":"172.17.0.0/16"}'

在随后Kubernetes的Node节点搭建和配置时,我们可以看到,etcd里的/atomic.io/network/config节点会被Node节点上的flannel用来创建网络的iptables

现在我们可以使用kubectl get nodes命令来查看,当然,目前还没有Node节点加入到该Kubernetes集群,所以命令的执行结果是空的:

# kubectl get nodes

NAME              STATUS    AGE

Kubernetes Node节点的安装与配置

在Kubernetes Node节点上安装flannel, docker和Kubernetes

# yum -y install flannel docker kubernetes

对flannel进行配置,编辑/etc/sysconfig/flanneld,内容如下:

FLANNEL_ETCD="http://192.168.20.145:2379"

FLANNEL_ETCD_KEY="/atomic.io/network"

配置信息告诉flannel进程etcd服务的位置以及在etcd上网络配置信息的节点位置

对Node节点上的Kubernetes进行配置,两台Node节点(192.168.20.146和192.168.20.147)上的配置文件/etc/kubernetes/config内容和Master节点(192.168.20.145)相同,内容如下:

KUBE_LOGTOSTDERR="--logtostderr=true"

KUBE_LOG_LEVEL="--v=0"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://192.168.20.145:8080"

KUBE_MASTER="--master=http://192.168.20.145:8080"是将Kubernetes的apiserver进程的服务地址告诉Kubernetes的controller-manager, scheduler和proxy进程。

而两台Node节点上的/etc/kubernetes/kubelet配置文件内容略微有点不同。

192.168.20.146节点的/etc/kubernetes/kubelet:

KUBELET_ADDRESS="--address=0.0.0.0"

KUBELET_PORT="--port=10250"

KUBELET_HOSTNAME="--hostname-override=192.168.20.146"

KUBELET_API_SERVER="--api-servers=http://192.168.20.145:8080"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_ARGS=""

192.168.20.147节点的/etc/kubernetes/kubelet:

KUBELET_ADDRESS="--address=0.0.0.0"

KUBELET_PORT="--port=10250"

KUBELET_HOSTNAME="--hostname-override=192.168.20.147"

KUBELET_API_SERVER="--api-servers=http://192.168.20.145:8080"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_ARGS=""

分别在两个Kubernetes Node节点上启动kube-proxy kubelet docker和flanneld进程并查看其状态:

# for SERVICES in kube-proxy kubelet docker flanneld; do

systemctl restart $SERVICES

systemctl enable $SERVICES

systemctl status $SERVICES

done

现在我们在Master节点上使用kubectl get nodes命令来查看,可以看到加入的两个Node节点:

# kubectl get nodes

NAME              STATUS    AGE

192.168.20.146  Ready    2d

192.168.20.147  Ready    2d

至此,Kubernetes集群环境搭建完毕,但是我的故事还没有结束。

Docker私有库搭建

搭建完了Kubernetes集群环境,我满心欢喜地去创建Pods,失败了。用kubectl describe和kubectl logs命令定位原因,发现原因是我的集群环境无法从gcr.io(Google Container-Registry)拉取镜像,但是从Docker hub可以拉取镜像。所以我萌生了搭建一个Docker私有镜像的想法。查阅资料后,过程描述如下。

为了让Docker私有库更加安全,我生成了自签名的证书来配置TLS. 首先编辑/etc/pki/tls/openssl.cnf,在[ v3_ca ]下增加了一行:

[plain]view plaincopy

[ v3_ca ]

subjectAltName = IP:192.168.20.148

然后使用openssl命令在当前的certs目录下创建了一个自签名的证书:

# mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt

在证书的创建过程中,会询问国家、省分、城市、组织、部门和common name的信息,其中common name信息我填写的是主机的IP 192.168.20.148. 证书创建完毕后,在certs目录下出现了两个文件:证书文件domain.crt和私钥文件domain.key

在192.168.20.148上安装docker

# yum -y install docker

将前面生成的domain.crt文件复制到/etc/docker/certs.d/192.168.20.148:5000目录下,然后重启docker进程:

# mkdir -p /etc/docker/certs.d/192.168.20.148:5000

# cp certs/domain.crt /etc/docker/certs.d/192.168.20.148:5000/ca.crt

# systemctl restart docker

在Docker私有库节点192.168.20.148上运行registry容器,并暴露容器的5000端口:

# docker run -d -p 5000:5000 --restart=always --name registry  -v `pwd`/certs:/certs  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key  registry:2

最后,将domain.crt文件复制到Kubernetes集群里的所有节点的/etc/docker/certs.d/192.168.20.148:5000目录下,并重启各节点的docker进程,例如在192.168.169.121节点上运行:

# mkdir -p /etc/docker/certs.d/192.168.20.148:5000

# scp root@192.168.20.148:~/certs/domain.crt /etc/docker/certs.d/192.168.20.148:5000/ca.crt

# systemctl restart docker

至此,Docker私有库搭建完成。

Kubernetes Web UI搭建

这节我以搭建Kubernetes Web UI(kubernetes-dashboard)来简要演示如何使用Docker私有库。

由于gcr.io被墙,我的Kubernetes集群无法直接从gcr.io拉取kubernetes-dashboard的镜像,我事先从docker.io下载了下载地址docker.io/mritd/kubernetes-dashboard-amd64:

# docker pull mritd/kubernetes-dashboard-amd64

# docker images

REPOSITORY                                        TAG                IMAGE ID            CREATED            SIZE

registry                                          2                  c6c14b3960bd        3 days ago          33.28 MB

ubuntu                                            latest              42118e3df429        9 days ago          124.8 MB

hello-world                                      latest              c54a2cc56cbb        4 weeks ago        1.848 kB

docker.io/mritd/kubernetes-dashboard-amd64      v1.1.0              20b7531358be        5 weeks ago        58.52 MB

registry                                          2                  8ff6a4aae657        7 weeks ago        171.5 MB

我为加载的kubernetes-dashboard镜像打上私有库的标签并推送到私有库:

# docker tag 20b7531358be 192.168.20.148:5000/kubernetes-dashboard-amd64

# docker push 192.168.20.148:5000/kubernetes-dashboard-amd64

kubernetes-dashboard.yaml配置文件如下,因为最新版本是1.7变化很大,所以这里用1.5的配置文件代替,对其进行编辑如下:

# Copyright 2015 Google Inc. All Rights Reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Configuration to deploy release version of the Dashboard UI.

#

# Example usage: kubectl create -f

kind: Deployment

apiVersion: extensions/v1beta1

metadata:

labels:

app: kubernetes-dashboard

version: v1.1.0

name: kubernetes-dashboard

namespace: kube-system

spec:

replicas: 1

selector:

matchLabels:

app: kubernetes-dashboard

template:

metadata:

labels:

app: kubernetes-dashboard

spec:

containers:

- name: kubernetes-dashboard

image: 192.168.20.148:5000/kubernetes-dashboard-amd64

imagePullPolicy: Always

ports:

- containerPort: 9090

protocol: TCP

args:

# Uncomment the following line to manually specify Kubernetes API server Host

# If not specified, Dashboard will attempt to auto discover the API server and connect

# to it. Uncomment only if the default does not work.

- --apiserver-host=192.168.20.145:8080

livenessProbe:

httpGet:

path: /

port: 9090

initialDelaySeconds: 30

timeoutSeconds: 30

---

kind: Service

apiVersion: v1

metadata:

labels:

app: kubernetes-dashboard

name: kubernetes-dashboard

namespace: kube-system

spec:

type: NodePort

ports:

- port: 80

targetPort: 9090

selector:

app: kubernetes-dashboard

尤其要注意:1 创建的Pods所要拉取的镜像是Docker私有库的192.168.20.148:5000/kubernetes-dashboard-adm64; 2 apiserver-host参数是192.168.20.145:8080,即Kubernetes Master节点的aipserver服务地址。

修改完kubernetes-dashboard.yaml后保存到Kubernetes Master节点192.168.20.145节点上,在该节点上用kubectl create命令创建kubernetes-dashboard:

# kubectl create -f kubernetes-dashboard.yaml

创建完成后,查看Pods和Service的详细信息:

# kubectl get pods --all-namespaces

NAMESPACE    NAME                                    READY    STATUS    RESTARTS  AGE

default      nginx                                  1/1      Running  0          3h

kube-system  kubernetes-dashboard-4164430742-lqhcg  1/1      Running  0          2h

# kubectl describe pods/kubernetes-dashboard-4164430742-lqhcg --namespace="kube-system"

Name:        kubernetes-dashboard-4164430742-lqhcg

Namespace:    kube-system

Node:        192.168.20.146/192.168.20.146

Start Time:    Mon, 01 Aug 2016 16:12:02 +0800

Labels:        app=kubernetes-dashboard,pod-template-hash=4164430742

Status:        Running

IP:        172.17.17.3

Controllers:    ReplicaSet/kubernetes-dashboard-4164430742

Containers:

kubernetes-dashboard:

Container ID:    docker://40ab377c5b8a333487f251547e5de51af63570c31f9ba05fe3030a02cbb3660c

Image:        192.168.20.148:5000/kubernetes-dashboard-amd64

Image ID:        docker://sha256:20b7531358be693a34eafdedee2954f381a95db469457667afd4ceeb7146cd1f

Port:        9090/TCP

Args:

--apiserver-host=192.168.20.145:8080

QoS Tier:

cpu:        BestEffort

memory:        BestEffort

State:        Running

Started:        Mon, 01 Aug 2016 16:12:03 +0800

Ready:        True

Restart Count:    0

Liveness:        http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3

Environment Variables:

Conditions:

Type        Status

Ready    True

No volumes.

No events.

# kubectl describe service/kubernetes-dashboard --namespace="kube-system"

Name:            kubernetes-dashboard

Namespace:        kube-system

Labels:            app=kubernetes-dashboard

Selector:        app=kubernetes-dashboard

Type:            NodePort

IP:            10.254.213.209

Port:                80/TCP

NodePort:            31482/TCP

Endpoints:        172.17.17.3:9090

Session Affinity:    None

No events.

从kubernetes-dashboard的service详细信息可以看到,该service绑定到了Node节点的31482端口上。现在,通过浏览器访问该端口就能看到Kubernetes的Web UI,地址:192.168.20.145:8080/ui会自动进行跳转:


日记本
Web note ad 1