Kubernetes搭建(ubutu)-master集群
环境设置
1.关闭防火墙
2.关闭selinux
3.关闭swap分区
4.配置本地地址解析
5.修改hostname文件
容器进行时
1.每个节点安装docker
2.配置开机自启、配置Docker守护进程
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
3.重新启动Docker
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker
安装 kubeadm、kubectl、kubelet
1.前提：开启某些端口，swap分区必须禁用
2.确保每个节点上 MAC 地址和 product_uuid 的唯一性
#检查mac地址和 product_uuid
ip link |awk '/link\/ether/ {print $2}'
cat /sys/class/dmi/id/product_uuid
3.修改内核参数
创建模块配置文件
#官方基础配置
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
#可尝试使用下列配置
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
EOF
重新加载配置模块
modprobe br_netfilter
检查模块是否加载成功
lsmod | grep br_netfilter
4.允许 iptables 检查桥接流量
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
5.检查所需端口
6.安装 kubeadm、kubelet 和 kubectl
-
kubeadm：用来初始化集群的指令。 -
kubelet：在集群中的每个节点上用来启动 Pod 和容器等。 -
kubectl：用来与集群通信的命令行工具
添加 Kubernetes apt 仓库：
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并锁定其版本：
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
7.设置kubelet服务自启动
sudo systemctl start kubelet
sudo systemctl enable kubelet
sudo systemctl start kubelet
8.执行脚本，下载镜像(仅在master节点下载)
vim /opt/kubeadm-image-pull.sh
aliyun_registry=registry.cn-hangzhou.aliyuncs.com/google_containers
images_arry=$(kubeadm config images list |grep k8s)
for k8s_image in ${images_arry[@]}
do
image=$(basename $k8s_image)
docker pull ${aliyun_registry}/${image}
if [ $? -eq 0 ];then
docker tag ${aliyun_registry}/${image} ${k8s_image}
docker rmi ${aliyun_registry}/${image}
echo "${k8s_image}" is success
else
echo "${k8s_image} is fail"
fi
done
9.确认镜像版本与kubeadm 版本是否为一致
kubeadm初始化集群
1.设置虚拟ip（vip），使用 keepalived
安装keepalived
sudo apt-get install keepalived
修改配置文件,如果没有请创建
sudo vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state MASTER # 主MASTER 从BACKUP
interface ens15f0 # 网卡名称
virtual_router_id 51 #路由器标识，MASTER和BACKUP必须是一致的
priority 100 #定义优先级，数字越大，优先级越高，在同一个vrrp_instance下，MASTER的优先级必须大于BACKUP的优先级。这样MASTER故障恢复后，就可以将VIP资源再次抢回来
advert_int 1
authentication {
auth_type PASS
auth_pass memoriae123456
}
virtual_ipaddress {
10.10.8.60 # 虚拟ip
}
}
mater从节点keepalived配置文件
vrrp_instance VI_1 {
state BACKUP # 主MASTER 从BACKUP
interface ens15f1 # 网卡名称
virtual_router_id 51 #路由器标识，MASTER和BACKUP必须是一致的
priority 90 #定义优先级，数字越大，优先级越高，在同一个vrrp_instance下，MASTER的优先级必须大于BACKUP的优先级。这样MASTER故障恢复后，就可以将VIP资源再次抢回来
advert_int 1
authentication {
auth_type PASS
auth_pass memoriae123456
}
virtual_ipaddress {
10.10.8.60 # 虚拟ip
}
}
启动keepalived，并设置开机自启
sudo systemctl start keepalived.service
sudo systemctl enable keepalived.service
2.kubuadm init
单节点
只需启动一台master节点，其余节点作为node节点参与；
kubeadm init --kubernetes-version=v1.23.6 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.10.8.60:80 --ignore-preflight-errors=Swap --control-plane-endpoint=memo.com --upload-certs
多节点
启动多个master节点；
kubeadm init --kubernetes-version=v1.23.6 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=Swap --control-plane-endpoint=memo.com --upload-certs
参数解释：
--node-name 增加该参数设置节点名称，默认名称为当前系统名，--node-name master1-memo
--kubernetes-version 参数与kubeadm版本一致，使用kubeadm查看当前版本
--pod-network-cidr=10.244.0.0/16 用来规定pod的网络使用范围
--apiserver-advertise-address，定义当前网络ip地址用于为控制平面节点的Apiserver设置广播地址，向集群成员发布apiserver的ip地址
--control-plane-endpoint 允许ip地址和DNS名称，用来添加master节点，此处应在/etc/hosts文件配置本地解析10.10.9.64 memo.com
3.查看节点状态
sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1-memo Ready control-plane,master 22m v1.23.6
4.网络插件-flannel
下载网络插件
git clone https://github.com/blackmed/kubernetes-kubeadm.git
进入下载目录，启动配置文件
sudo kubectl apply -f kubernetes-kubeadm/flannel.yaml
停止网络插件
sudo kubectl delete-f kubernetes-kubeadm/flannel.yaml
5.查看节点状态
查看组件pod状态
sudo kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-64897985d-hfg9p 1/1 Running 0 26m
kube-system coredns-64897985d-kjzvg 1/1 Running 0 26m
kube-system etcd-master1-memo 1/1 Running 0 26m
kube-system kube-apiserver-master1-memo 1/1 Running 0 26m
kube-system kube-controller-manager-master1-memo 1/1 Running 0 26m
kube-system kube-flannel-ds-nkrlt 1/1 Running 0 7m3s
kube-system kube-proxy-w4txh 1/1 Running 0 26m
kube-system kube-scheduler-master1-memo 1/1 Running 0 26m
检查pod容器日志
sudo kubectl logs kube-flannel-ds-ppgp5 -n kube-system
检查集群scheduler和controller-manager组件健康状态
sudo kubectl get cs
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
如果检查集群状态显示为：Unhealthy
将两个配置文件中–port注释掉
vim /etc/kubernetes/manifests/kube-scheduler.yaml
vim /etc/kubernetes/manifests/kube-controller-manager.yaml
重新启动kubelet服务,再次查看状态
sudo systemctl restart kubelet
6.生成Token及证书
注意Token时效，如果过期，请重新生成
# kubeadm token create --print-join-command
kubeadm join memo.com:6443 --token wdbmng.cq2l3sxu7gdvwc8v --discovery-token-ca-cert-hash sha256:72156db92a1e71d14eb644f5260ab095d68096fe316e3c982680d58d552ce35e
# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
40720d8b9e322b752e2d1cc9b24efd6d5129769245ff5b301210a7d7b0866172
7.加入节点
master从节点执行(加入master集群)
kubeadm join memo.com:6443 --token wdbmng.cq2l3sxu7gdvwc8v --discovery-token-ca-cert-hash sha256:72156db92a1e71d14eb644f5260ab095d68096fe316e3c982680d58d552ce35e --control-plane --certificate-key 40720d8b9e322b752e2d1cc9b24efd6d5129769245ff5b301210a7d7b0866172
node节点执行(加入工作节点)
kubeadm join memo.com:6443 --token wdbmng.cq2l3sxu7gdvwc8v --discovery-token-ca-cert-hash sha256:72156db92a1e71d14eb644f5260ab095d68096fe316e3c982680d58d552ce35e
8.查看当前master集群状态
sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1-memo Ready control-plane,master 15h v1.23.6
master2-memo Ready control-plane,master 15h v1.23.6
9.检查etcd状态
master集群部署3个，首先获取集群中的 etcd pod 列表
sudo kubectl get pods -n kube-system | grep etcd
etcd-master1-memo 1/1 Running 0 17h
etcd-master2-memo 1/1 Running 0 17h
etcd-master3-memo 1/1 Running 0 157m
进入任意一个 pod，通过 kubectl 工具进入 pod 内部：
sudo kubectl exec -it kube-flannel-ds-ppgp5 sh -n kube-system
进入后执行
#配置环境
export ETCDCTL_API=3
alias etcdctl='etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
查看 etcd 集群成员列表
etcdctl member list
3a82a2a54a23544d, started, master1-memo, https://10.10.9.60:2380, https://10.10.9.60:2379, false
40be7a5531c2ce6e, started, master2-memo, https://10.10.9.48:2380, https://10.10.9.48:2379, false
a0589eb911d55583, started, master3-memo, https://10.10.9.49:2380, https://10.10.9.49:2379, false
清除节点
node节点
sudo kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
sudo kubectl delete node <node name>
#执行完卸载和剔除节点之后，重置node节点，node节点执行
sudo kubeadm reset
master节点
sudo kubeadm reset
执行kubeadm reset之后，请删除执行kubeadm命令用户的 .kube集群文件，防止再次加入集群时报错
sudo rm -r $HOME/.kube
污点(Taints)
kubectl describe node <NODE_NAME> | grep Taint
kubectl describe node master | grep Taint
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
