×

使用kubeadm安装Kubernetes 1.8版本

96
Jack0111
2017.10.25 00:10* 字数 3520

kubeadm是Kubernetes官方提供的快速安装和初始化Kubernetes集群的工具,目前的还处于孵化开发状态,伴随Kubernetes每个版本的发布都会同步更新。 当然,目前的kubeadm是不能用于生产环境的。 但伴随着Kubernetes每次版本升级,kubeadm都会对集群配置方面的一些实践做调整,通过实验kubeadm我们可以学习到Kubernetes官方在集群配置上一些新的最佳实践。

Kubernetes 1.8已经发布,为了跟上官方更新的脚本,接下来体验一下Kubernetes 1.8中的kubeadm。

1.准备

1.1系统配置

在安装之前,需要先做如下准备。两台CentOS 7.3主机如下:

cat/etc/hosts192.168.61.11node1192.168.61.12node2

如果各个主机启用了防火墙,需要开放Kubernetes各个组件所需要的端口,可以查看Installing kubeadm中的”Check required ports”一节。 这里简单起见在各节点禁用防火墙:

systemctl stop firewalld

systemctl disable firewalld

创建/etc/sysctl.d/k8s.conf文件,添加如下内容:

net.bridge.bridge-nf-call-ip6tables=1net.bridge.bridge-nf-call-iptables=1

执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

禁用SELINUX:

setenforce0

vi/etc/selinux/config

SELINUX=disabled

Kubernetes 1.8开始要求关闭系统的Swap,如果不关闭,默认配置下kubelet将无法启动。可以通过kubelet的启动参数–fail-swap-on=false更改这个限制。 我们这里关闭系统的Swap:

swapoff-a

修改 /etc/fstab 文件,注释掉 SWAP 的自动挂载,使用free -m确认swap已经关闭。

swappiness参数调整,修改/etc/sysctl.d/k8s.conf添加下面一行:

vm.swappiness=0

执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

1.2安装Docker

yum install-y yum-utils device-mapper-persistent-data lvm2

yum-config-manager \--add-repo \

https://download.docker.com/linux/centos/docker-ce.repo

查看当前的Docker版本:

yum list docker-ce.x86_64--showduplicates|sort-r

docker-ce.x86_6417.09.0.ce-1.el7.centosdocker-ce-stable

docker-ce.x86_6417.06.2.ce-1.el7.centosdocker-ce-stable

docker-ce.x86_6417.06.1.ce-1.el7.centosdocker-ce-stable

docker-ce.x86_6417.06.0.ce-1.el7.centosdocker-ce-stable

docker-ce.x86_6417.03.2.ce-1.el7.centosdocker-ce-stable

docker-ce.x86_6417.03.1.ce-1.el7.centosdocker-ce-stable

docker-ce.x86_6417.03.0.ce-1.el7.centosdocker-ce-stable

Kubernetes 1.8已经针对Docker的1.11.2, 1.12.6, 1.13.1和17.03.2等版本做了验证。 因为我们这里在各节点安装docker的17.03.2版本。

yum makecache fast

yum install-y--setopt=obsoletes=0\

docker-ce-17.03.2.ce-1.el7.centos\

docker-ce-selinux-17.03.2.ce-1.el7.centossystemctl start docker

systemctl enable docker

Docker从1.13版本开始调整了默认的防火墙规则,禁用了iptables filter表中FOWARD链,这样会引起Kubernetes集群中跨Node的Pod无法通信,在各个Docker节点执行下面的命令:

iptables-P FORWARD ACCEPT

可在docker的systemd unit文件中以ExecStartPost加入上面的命令:

ExecStartPost=/usr/sbin/iptables-P FORWARD ACCEPT

systemctl daemon-reload

systemctl restart docker

2.安装kubeadm和kubelet

下面在各节点安装kubeadm和kubelet:

cat</etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpghttps://packages.cloud.google.com/yum/doc/rpm-package-key.gpgEOF

测试地址https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64是否可用,如果不可用需要科学上网。

curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

yum makecache fast

yum install-y kubelet kubeadm kubectl...Installed:kubeadm.x86_640:1.8.0-0kubectl.x86_640:1.8.0-0kubelet.x86_640:1.8.0-0DependencyInstalled:kubernetes-cni.x86_640:0.5.1-0socat.x86_640:1.7.3.2-2.el7

从安装结果可以看出还安装了kubernetes-cni和socat两个依赖:

可以看出官方Kubernetes 1.8依赖的cni还是0.5.1版本

socat是kubelet的依赖

我们之前在Kubernetes 1.6 高可用集群部署中手动安装这两个依赖的

Kubernetes文档中kubelet的启动参数

--cgroup-driverstringDriverthat the kubelet uses to manipulate cgroups on the host.Possiblevalues:'cgroupfs','systemd'(default"cgroupfs")

默认值为cgroupfs,但是我们注意到yum安装kubelet,kubeadm时生成10-kubeadm.conf文件中将这个参数值改成了systemd。

查看kubelet的 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf文件,其中包含如下内容:

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"

使用docker info打印docker信息:

docker info......ServerVersion:17.03.2-ce......CgroupDriver:cgroupfs

可以看出docker 17.03使用的Cgroup Driver为cgroupfs。

于是修改各节点docker的cgroup driver使其和kubelet一致,即修改或创建/etc/docker/daemon.json,加入下面的内容:

{"exec-opts":["native.cgroupdriver=systemd"]}

重启docker:

systemctl restart docker

systemctl status docker

在各节点开机启动kubelet服务:

systemctl enable kubelet.service

3.使用kubeadm init初始化集群

接下来使用kubeadm初始化集群,选择node1作为Master Node,在node1上执行下面的命令:

kubeadm init \--kubernetes-version=v1.8.0\--pod-network-cidr=10.244.0.0/16\--apiserver-advertise-address=192.168.61.11

因为我们选择flannel作为Pod网络插件,所以上面的命令指定–pod-network-cidr=10.244.0.0/16。

kubeadm init \>--kubernetes-version=v1.8.0\>--pod-network-cidr=10.244.0.0/16\>--apiserver-advertise-address=192.168.61.11[kubeadm]WARNING:kubeadmisinbeta,pleasedonotuseitforproduction clusters.[init]UsingKubernetesversion:v1.8.0[init]UsingAuthorizationmodes:[NodeRBAC][preflight]Runningpre-flight checks[preflight]WARNING:firewalldisactive,pleaseensureports[644310250]are openoryour cluster maynotfunctioncorrectly[preflight]Startingthe kubelet service[kubeadm]WARNING:startingin1.8,tokens expire after24hoursbydefault(ifyourequirea non-expiring tokenuse--token-ttl0)[certificates]Generatedca certificateandkey.[certificates]Generatedapiserver certificateandkey.[certificates]apiserver serving certissignedforDNS names[node1 kubernetes kubernetes.defaultkubernetes.default.svc kubernetes.default.svc.cluster.local]andIPs[10.96.0.1192.168.61.11][certificates]Generatedapiserver-kubelet-client certificateandkey.[certificates]Generatedsa keyandpublickey.[certificates]Generatedfront-proxy-ca certificateandkey.[certificates]Generatedfront-proxy-client certificateandkey.[certificates]Validcertificatesandkeys now existin"/etc/kubernetes/pki"[kubeconfig]WroteKubeConfigfile to disk:"admin.conf"[kubeconfig]WroteKubeConfigfile to disk:"kubelet.conf"[kubeconfig]WroteKubeConfigfile to disk:"controller-manager.conf"[kubeconfig]WroteKubeConfigfile to disk:"scheduler.conf"[controlplane]WroteStaticPodmanifestforcomponent kube-apiserver to"/etc/kubernetes/manifests/kube-apiserver.yaml"[controlplane]WroteStaticPodmanifestforcomponent kube-controller-manager to"/etc/kubernetes/manifests/kube-controller-manager.yaml"[controlplane]WroteStaticPodmanifestforcomponent kube-scheduler to"/etc/kubernetes/manifests/kube-scheduler.yaml"[etcd]WroteStaticPodmanifestforalocaletcd instance to"/etc/kubernetes/manifests/etcd.yaml"[init]Waitingforthe kubelet to boot up the control planeasStaticPodsfromdirectory"/etc/kubernetes/manifests"[init]Thisoften takes around a minute;orlongerifthe control plane images have to be pulled.[apiclient]Allcontrol plane components are healthy after28.505733seconds[uploadconfig]Storingthe configuration usedinConfigMap"kubeadm-config"inthe"kube-system"Namespace[markmaster]Willmark node node1asmasterbyadding a labelanda taint[markmaster]Masternode1 taintedandlabelledwithkey/value:node-role.kubernetes.io/master=""[bootstraptoken]Usingtoken:9e68dd.7117f03c900f9234[bootstraptoken]ConfiguredRBAC rules to allowNodeBootstraptokens to postCSRsinorderfornodes togetlongterm certificate credentials[bootstraptoken]ConfiguredRBAC rules to allow the csrapprover controller automatically approveCSRsfromaNodeBootstrapToken[bootstraptoken]Creatingthe"cluster-info"ConfigMapinthe"kube-public"namespace[addons]Appliedessential addon:kube-dns[addons]Appliedessential addon:kube-proxyYourKubernetesmaster has initialized successfully!Tostartusingyour cluster,you need to run(asa regular user):mkdir-p $HOME/.kube

sudo cp-i/etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id-u):$(id-g)$HOME/.kube/configYoushould now deploy a pod network to the cluster.Run"kubectl apply -f [podnetwork].yaml"withone of the options listed at:http://kubernetes.io/docs/admin/addons/Youcan now join any number of machinesbyrunning the following on each nodeasroot:kubeadm join--token9e68dd.7117f03c900f9234192.168.61.11:6443--discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7

上面记录了完成的初始化输出的内容。

其中由以下关键内容:

kubeadm 1.8当前还处于beta状态,还不能用于生产环境。目前来看这东西安装的etcd和apiserver都是单节点,当然不能用于生产环境。

RBAC模式已经在Kubernetes 1.8中稳定可用。kubeadm 1.8也默认启用了RBAC

接下来是生成证书和相关的kubeconfig文件,这个目前我们在Kubernetes 1.6 高可用集群部署也是这么做的,目前没看出有什么新东西

生成token记录下来,后边使用kubeadm join往集群中添加节点时会用到

另外注意kubeadm还报了starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use –token-ttl 0)的警告

下面的命令是配置常规用户如何使用kubectl访问集群:

mkdir-p $HOME/.kube

sudo cp-i/etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id-u):$(id-g)$HOME/.kube/config

最后给出了将节点加入集群的命令kubeadm join –token 9e68dd.7117f03c900f9234 192.168.61.11:6443 –discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7

查看一下集群状态:

kubectlgetcs

NAME                STATUS    MESSAGE              ERROR

schedulerHealthyok

controller-managerHealthyok

etcd-0Healthy{"health":"true"}

确认个组件都处于healthy状态。

集群初始化如果遇到问题,可以使用下面的命令进行清理:

kubeadm reset

ifconfig cni0 down

ip linkdeletecni0

ifconfig flannel.1down

ip linkdeleteflannel.1rm-rf/var/lib/cni/

4.安装Pod Network

接下来安装flannel network add-on:

mkdir-p~/k8s/wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.ymlkubectl apply-f  kube-flannel.yml

clusterrole"flannel"created

clusterrolebinding"flannel"created

serviceaccount"flannel"created

configmap"kube-flannel-cfg"created

daemonset"kube-flannel-ds"created

这里注意kube-flannel.yml这个文件中已经包含了ServiceAccount, ClusterRole和ClusterRoleBinding,原来是在一个单独的kube-flannel-rbac.yml文件中。kube-flannel.yml这个文件里的flannel的镜像是0.9.0,quay.io/coreos/flannel:v0.9.0-amd64

如果Node有多个网卡的话,参考flannel issues 39701,目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称,否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地,flanneld启动参数加上–iface=

......apiVersion:extensions/v1beta1

kind:DaemonSetmetadata:name:kube-flannel-ds......containers:-name:kube-flannel

image:quay.io/coreos/flannel:v0.9.0-amd64

command:["/opt/bin/flanneld","--ip-masq","--kube-subnet-mgr","--iface=eth1"]......

使用kubectl get pod –all-namespaces -o wide确保所有的Pod都处于Running状态。

kubectlgetpod--all-namespaces-o wide

5.master node参与工作负载

使用kubeadm初始化的集群,出于安全考虑Pod不会被调度到Master Node上,也就是说Master Node不参与工作负载。

这里搭建的是测试环境可以使用下面的命令使Master Node参与工作负载:

kubectl taint nodes node1 node-role.kubernetes.io/master-node"node1"untainted

6.测试DNS

kubectl run curl--image=radial/busyboxplus:curl-i--ttyIfyou don't see a command prompt, try pressing enter.

[ root@curl-2716574283-xr8zd:/ ]$

进入后执行nslookup kubernetes.default确认解析正常:

nslookup kubernetes.defaultServer:10.96.0.10Address1:10.96.0.10kube-dns.kube-system.svc.cluster.localName:kubernetes.defaultAddress1:10.96.0.1kubernetes.default.svc.cluster.local

7.向Kubernetes集群添加Node

下面我们将k8s-node2这个主机添加到Kubernetes集群中,在k8s-node2上执行:

kubeadm join--token9e68dd.7117f03c900f9234192.168.61.11:6443--discovery-token-ca-cert-hash sha256:82a08ef9c830f240e588a26a8ff0a311e6fe3127c1ee4c5fc019f1369007c0b7[kubeadm]WARNING:kubeadmisinbeta,pleasedonotuseitforproduction clusters.[preflight]Runningpre-flight checks[discovery]Tryingto connect to APIServer"192.168.61.11:6443"[discovery]Createdcluster-info discovery client,requesting infofrom"https://192.168.61.11:6443"[discovery]Requestinginfofrom"https://192.168.61.11:6443"again to validate TLS against the pinnedpublickey[discovery]Clusterinfo signatureandcontents are validandTLS certificate validates against pinned roots,willuseAPIServer"192.168.61.11:6443"[discovery]Successfullyestablished connectionwithAPIServer"192.168.61.11:6443"[bootstrap]Detectedserver version:v1.8.0[bootstrap]Theserver supports theCertificatesAPI(certificates.k8s.io/v1beta1)Nodejoin complete:*Certificatesigning request sent to masterandresponse

received.*Kubeletinformed ofnewsecure connection details.Run'kubectl get nodes'on the master to seethismachine join.

这次很是顺利,下面在master节点上执行命令查看集群中的节点:

kubectlgetnodes

NAME      STATUS    ROLES    AGE      VERSION

node1Readymaster25mv1.8.0node2Ready10mv1.8.0

如何从集群中移除Node

如果需要从集群中移除node2这个Node执行下面的命令:

在master节点上执行:

kubectl drain node2--delete-local-data--force--ignore-daemonsets

kubectldeletenode node2

在node2上执行:

kubeadm reset

ifconfig cni0 down

ip linkdeletecni0

ifconfig flannel.1down

ip linkdeleteflannel.1rm-rf/var/lib/cni/

8.dashboard插件部署

注意当前dashboard的版本已经是1.7.1了。 而1.7.x版本的dashboard对安全做了增强,默认需要以https的方式访问,增加了登录的页面,同时增加了一个gcr.io/google_containers/kubernetes-dashboard-init-amd64的init容器。

另外需要注意dashboard调整了部署文件的源码目录结构:

mkdir-p~/k8s/wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yamlkubectl create-f kubernetes-dashboard.yaml

kubernetes-dashboard.yaml文件中的ServiceAccount kubernetes-dashboard只有相对较小的权限,因此我们创建一个kubernetes-dashboard-admin的ServiceAccount并授予集群admin的权限,创建kubernetes-dashboard-admin.rbac.yaml:

---apiVersion:v1

kind:ServiceAccountmetadata:labels:k8s-app:kubernetes-dashboard

name:kubernetes-dashboard-adminnamespace:kube-system---apiVersion:rbac.authorization.k8s.io/v1beta1

kind:ClusterRoleBindingmetadata:name:kubernetes-dashboard-admin

labels:k8s-app:kubernetes-dashboard

roleRef:apiGroup:rbac.authorization.k8s.io

kind:ClusterRolename:cluster-admin

subjects:-kind:ServiceAccountname:kubernetes-dashboard-adminnamespace:kube-system

kubectl create-f kubernetes-dashboard-admin.rbac.yaml

serviceaccount"kubernetes-dashboard-admin"created

clusterrolebinding"kubernetes-dashboard-admin"created

查看kubernete-dashboard-admin的token:

kubectl-n kube-systemgetsecret|grep kubernetes-dashboard-admin

kubernetes-dashboard-admin-token-pfss5  kubernetes.io/service-account-token314skubectl describe-n kube-system secret/kubernetes-dashboard-admin-token-pfss5Name:kubernetes-dashboard-admin-token-pfss5Namespace:kube-systemLabels:Annotations:kubernetes.io/service-account.name=kubernetes-dashboard-admin

kubernetes.io/service-account.uid=1029250a-ad76-11e7-9a1d-08002778b8a1Type:kubernetes.io/service-account-tokenData====ca.crt:1025bytesnamespace:11bytes

token:eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1wZnNzNSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjEwMjkyNTBhLWFkNzYtMTFlNy05YTFkLTA4MDAyNzc4YjhhMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.Bs6h65aFCFkEKBO_h4muoIK3XdTcfik-pNM351VogBJD_pk5grM1PEWdsCXpR45r8zUOTpGM-h8kDwgOXwy2i8a5RjbUTzD3OQbPJXqa1wBk0ABkmqTuw-3PWMRg_Du8zuFEPdKDFQyWxiYhUi_v638G-R5RdZD_xeJAXmKyPkB3VsqWVegoIVTaNboYkw6cgvMa-4b7IjoN9T1fFlWCTZI8BFXbM8ICOoYMsOIJr3tVFf7d6oVNGYqaCk42QL_2TfB6xMKLYER9XDh753-_FDVE5ENtY5YagD3T_s44o0Ewara4P9C3hYRKdJNLxv7qDbwPl3bVFH3HXbsSxxF3TQ

在dashboard的登录窗口使用上面的token登录。

9.heapster插件部署

下面安装Heapster为集群添加使用统计和监控功能,为Dashboard添加仪表盘。 使用InfluxDB做为Heapster的后端存储,开始部署:

mkdir-p~/k8s/heapster

cd~/k8s/heapster

wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yamlwget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yamlwget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yamlwget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yamlkubectl create-f./

最后确认所有的pod都处于running状态,打开Dashboard,集群的使用统计会以仪表盘的形式显示出来。

本次安装涉及到的Docker镜像:

gcr.io/google_containers/kube-apiserver-amd64:v1.8.0gcr.io/google_containers/kube-controller-manager-amd64:v1.8.0gcr.io/google_containers/kube-scheduler-amd64:v1.8.0gcr.io/google_containers/kube-proxy-amd64:v1.8.0gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5quay.io/coreos/flannel:v0.9.0-amd64

gcr.io/google_containers/etcd-amd64:3.0.17gcr.io/google_containers/pause-amd64:3.0quay.io/coreos/flannel:v0.9.0-amd64

gcr.io/google_containers/kubernetes-dashboard-amd64:v1.7.1gcr.io/google_containers/kubernetes-dashboard-init-amd64:v1.0.0gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3gcr.io/google_containers/heapster-grafana-amd64:v4.4.3gcr.io/google_containers/heapster-amd64:v1.4.0

日记本
Web note ad 1