环境:
- 操作系统: ubuntu 18.04
- k8s集群: v1.15.12 (kubeadm安装)
阿里云坑爹的文档,害我搞了3天才搞定
https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/getting-started.md
获取region_id.instance_id
echo `curl -s http://100.100.100.200/latest/meta-data/region-id`.`curl -s http://100.100.100.200/latest/meta-data/instance-id`
修改 kubelet启动参数
添加 --cloud-provider=external --provider-id=region_id.instance_id替换为上面获取到的id(provider-id是阿里云控制器用来识别ecs,添加路由表,添加负载均衡监听和虚拟服务器组)
不需要--hostname-override,会导致kubeadm join无法正常结束
(未验证)如果是已经加入集群的node,只修改kubectl并重启没用,估计可以直接修改node的spec,添加 providerID: cn-hongkong.i-xxxx
$ sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
...
Environment="KUBELET_CLOUD_PROVIDER_ARGS=--cloud-provider=external --provider-id=$region_id.$instance_id"
...
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CLOUD_PROVIDER_ARGS
创建access_key ConfigMap
获取阿里云账户access_key,需要负载均衡和专用网络VPC路由表权限
https://usercenter.console.aliyun.com/#/manage/ak
$ vi cloud-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-config
namespace: kube-system
data:
special.keyid: __access_key__
special.keysecret: __access_secret__
安装阿里云控制器
官方资源文档:https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/examples/cloud-controller-manager.yml
需要修改下面3项
- ${CLUSTER_CIDR}为集群pod子网地址
- 镜像版本(一定要修改!)
推荐镜像版本:cloud-controller-manager-amd64:v1.9.3.81-gca19cd4-aliyun - 配置文件的挂载方式,默认是挂载宿主机目录,建议改为ConfigMap
云控制器镜像版本目前是:cloud-controller-manager-amd64:v1.9.3.10-gfb99107-aliyun
有几个问题
- node name必须是region_id.instance_id,类似cn-hangzhou.i-xxxx,没有可读性
- 默认使用share规格的负载均衡,阿里云已经停用了,导致控制器无法创建新的负载均衡,需要通过service的annotation指定规格 service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
- 并且很多k8s service 的annotation不可用,比如指定使用已创建的负载均衡(推荐)service.beta.kubernetes.io/alicloud-loadbalancer-id:lb-xxxx
详情见注解文档
注解文档:https://help.aliyun.com/document_detail/86531.html#title-6ou-i9x-2lr
注意!!!老版注解前缀为alicloud,很多网上的其他文档用的是新版前缀alibaba-cloud(坑爹)
配置文件
将云控制器配置文件调整为ConfigMap挂载方式(推荐)
也可以按照官方文档创建etc/kubernetes/cloud-controller-manager.conf,改为下面data的内容
$ vi cloud-controller-manager.yml
...
apiVersion: extensions/v1beta1
kind: DaemonSet
...
volumes:
- configMap:
name: cloud-controller-manager
name: k8s
...
---
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-controller-manager
namespace: kube-system
data:
cloud-controller-manager.conf: |-
kind: Config
contexts:
- context:
cluster: kubernetes
user: system:cloud-controller-manager
name: system:cloud-controller-manager@kubernetes
current-context: system:cloud-controller-manager@kubernetes
users:
- name: system:cloud-controller-manager
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ${ca_data}
server: ${k8s_master_url}
name: kubernetes
修改${ca_data}替换为下面的证书内容
cat /etc/kubernetes/pki/ca.crt|base64 -w 0
修改${k8s_master_url} 为集群地址
安装
kubectl apply -f cloud-controller-manager.yml
测试
$ vi nginx.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:latest
name: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
service.beta.kubernetes.io/alicloud-loadbalancer-id: "lb-xxxx"
service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: "true"
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer
$ kubectl apply -f nginx.yaml