从零开始搭建Kubernetes集群(五、搭建K8S Ingress)

一、前言

上一文《从零开始搭建Kubernetes集群(四、搭建K8S Dashboard)》介绍了如何搭建Dashboard。本篇将介绍如何搭建Ingress来访问K8S集群的Service。

二、Ingress简介

Ingress是个什么鬼,网上资料很多(推荐官方),大家自行研究。简单来讲,就是一个负载均衡的玩意,其主要用来解决使用NodePort暴露Service的端口时Node IP会漂移的问题。同时,若大量使用NodePort暴露主机端口,管理会非常混乱。

好的解决方案就是让外界通过域名去访问Service,而无需关心其Node IP及Port。那为什么不直接使用Nginx?这是因为在K8S集群中,如果每加入一个服务,我们都在Nginx中添加一个配置,其实是一个重复性的体力活,只要是重复性的体力活,我们都应该通过技术将它干掉。

Ingress就可以解决上面的问题,其包含两个组件Ingress Controller和Ingress:

  • Ingress
    将Nginx的配置抽象成一个Ingress对象,每添加一个新的服务只需写一个新的Ingress的yaml文件即可
  • Ingress Controller
    将新加入的Ingress转化成Nginx的配置文件并使之生效

好了,废话不多,走你~

三、准备操作

官方文档

人生苦短,不造轮子,本文将以官方的标准脚本为基础进行搭建,参考请戳官方文档。官方文档中要求依次执行如下命令:

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
    | kubectl apply -f -

以上yaml文件创建Ingress用到的Namespace、ConfigMap,以及默认的后端default-backend。最关键的一点是,由于之前我们基于Kubeadm创建了K8S集群,则还必须执行:

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \
    | kubectl apply -f -

这是由于Kubeadm创建的集群默认开启了RABC,因此Ingress也必须创建相应的RABC权限控制。

导入镜像

但是,直接按照上述方式执行,我们的Ingress很可能会无法使用。所以,我们需要将上述Yaml文件全部wget下来,经过一些修改后才能执行kubectl apply -f创建。另外需要注意的是,这些yaml文件中提到的一些镜像,国内目前无法下载,如:

gcr.io/google_containers/defaultbackend:1.4
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0

本人已经提前下载好,大家请戳:

地址:https://pan.baidu.com/s/1N-bK9hI7JTZZB6AzmaT8PA
密码:1a8a

拿到镜像后,在每个节点上执行如下命令导入镜像:

docker load < quay.io#kubernetes-ingress-controller#nginx-ingress-controller_0.14.0.tar
docker tag 452a96d81c30 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
docker load < gcr.io#google_containers#defaultbackend.tar
docker tag 452a96d81c30 gcr.io/google_containers/defaultbackend

如上所示,导入镜像后,别忘记给打tag,否则镜像名称为<none>:


image.png

四、主要文件介绍

这里,我们先对一些重要的文件进行简单介绍。

default-backend.yaml

default-backend的作用是,如果外界访问的域名不存在的话,则默认转发到default-http-backend这个Service,其会直接返回404:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend

rbac.yaml

rbac.yaml负责Ingress的RBAC授权的控制,其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding。在上文《从零开始搭建Kubernetes集群(四、搭建K8S Dashboard)》中,我们已对这些概念进行了简单介绍。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

with-rbac.yaml

with-rbac.yaml是Ingress的核心,用于创建ingress-controller。前面提到过,ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置。

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          securityContext:
            runAsNonRoot: false

如上,可以看到nginx-ingress-controller启动时传入了参数,分别为前面创建的default-backend-service以及configmap。

五、创建Ingress

1.创建Ingress-controller

需要注意的是,官方提供的with-rbac.yaml文件不能直接使用,我们必须修改两处:

加入hostNetwork配置

如下,在serviceAccountName上方添加hostNetwork: true:

spec:
      hostNetwork: true
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io

配置hostNetwork: true是一种直接定义Pod网络的方式。定义后,Ingress-controller的IP就与宿主机k8s-node1一样(192.168.56.101),并且端口80也是宿主机上的端口。这样,我们通过该192.168.56.101:80,就可以直接访问到Ingress-controller(实际上就是nginx),然后Ingress-controller则会转发我们的请求到相应后端。

加入环境变量

在其env部分加入如下环境变量:

          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_MASTER 
              value: http://192.168.56.101:8080

否则,创建后会提示如下错误:

[root@k8s-node1 ingress]# kubectl describe pod nginx-ingress-controller-9fbd7596d-rt9sf  -n ingress-nginx
省略前面...
Events:
  Type     Reason                 Age                From                Message
  ----     ------                 ----               ----                -------
  Normal   Scheduled              30s                default-scheduler   Successfully assigned nginx-ingress-controller-9fbd7596d-rt9sf to k8s-node1
  Normal   SuccessfulMountVolume  30s                kubelet, k8s-node1  MountVolume.SetUp succeeded for volume "nginx-ingress-serviceaccount-token-lq2dt"
  Warning  BackOff                21s                kubelet, k8s-node1  Back-off restarting failed container
  Normal   Pulled                 11s (x3 over 29s)  kubelet, k8s-node1  Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0" already present on machine
  Normal   Created                11s (x3 over 29s)  kubelet, k8s-node1  Created container
  Warning  Failed                 10s (x3 over 28s)  kubelet, k8s-node1  Error: failed to start container "nginx-ingress-controller": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory": unknown

修改with-rbac.yaml后,使用kubectl -f create命令分别执行如下yaml文件,即可创建Ingress-controller:

image.png

创建成功后如下所示:

[root@k8s-node1 ingress]# kubectl get pod -n ingress-nginx -o wide
NAME                                        READY     STATUS    RESTARTS   AGE       IP              NODE
default-http-backend-5c6d95c48-pdjn9        1/1       Running   0          23s       192.168.36.81   k8s-node1
nginx-ingress-controller-547cd7d9cb-jmvpn   1/1       Running   0          8s        192.168.36.82   k8s-node1

2.创建自定义Ingress

有了ingress-controller,我们就可以创建自定义的Ingress了。这里已提前搭建好了Kibana服务,我们针对Kibana创建一个Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: default

spec:
  rules:
  - host: myk8s.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601

其中:

  • rules中的host必须为域名,不能为IP,表示Ingress-controller的Pod所在主机域名,也就是Ingress-controller的IP对应的域名。
  • paths中的path则表示映射的路径。如映射/表示若访问myk8s.com,则会将请求转发至Kibana的service,端口为5601。

创建成功后,查看:

[root@k8s-node1 ingress]# kubectl get ingress -o wide
NAME             HOSTS       ADDRESS   PORTS     AGE
kibana-ingress   myk8s.com             80        6s

我们再执行kubectl exec nginx-ingress-controller-5b79cbb5c6-2zr7f -it cat /etc/nginx/nginx.conf -n ingress-nginx,可以看到生成nginx配置,篇幅较长,各位自行筛选:

    ## start server myk8s.com
    server {
        server_name myk8s.com ;
        
        listen 80;
        
        listen [::]:80;
        
        set $proxy_upstream_name "-";
        
        location /kibana {
            
            log_by_lua_block {
                
            }
            
            port_in_redirect off;
            
            set $proxy_upstream_name "";
            
            set $namespace      "kube-system";
            set $ingress_name   "dashboard-ingress";
            set $service_name   "kibana";
            
            client_max_body_size                    "1m";
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Real-IP              $the_real_ip;
            
            proxy_set_header X-Forwarded-For        $the_real_ip;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Original-URI         $request_uri;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         "off";
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";
            
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
            proxy_next_upstream_tries               0;
            
            # No endpoints available for the request
            return 503;
            
        }
        
        location / {
            
            log_by_lua_block {
                
            }
            
            port_in_redirect off;
            
            set $proxy_upstream_name "";
            
            set $namespace      "default";
            set $ingress_name   "kibana-ingress";
            set $service_name   "kibana";
            
            client_max_body_size                    "1m";
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Real-IP              $the_real_ip;
            
            proxy_set_header X-Forwarded-For        $the_real_ip;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Original-URI         $request_uri;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         "off";
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";
            
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
            proxy_next_upstream_tries               0;
            
            # No endpoints available for the request
            return 503;
            
        }
        
    }
    ## end server myk8s.com

3.设置host

首先,我们需要在Ingress-controller的Pod所在主机上(这里为k8s-node1),将上面提到的域名myk8s.com追加入/etc/hosts文件:

192.168.56.101 myk8s.com

除此之外,如果想在自己的Windows物理机上使用浏览器访问kibana,也需要在C:\Windows\System32\drivers\etc\hosts文件内加入上述内容。设置后,分别在k8s-node1和物理机上测试无误即可:

image.png

image.png

六、测试

在Windows物理机上,使用Chrome访问myk8s.com,也就是相当于访问了192.168.56.101:80

image.png

随意访问一个错误的地址myk8s.com/abc,返回预期的404:

image.png

七、废话

至此,我们的Ingress已经搭建完毕,实现了在外部通过域名访问K8S集群Service的功能。如果大家有兴趣,可以尝试为Ingress配置TLS,这样就可以访问如Dashboard这种https服务了。下一章节《从零开始搭建Kubernetes集群(五、在K8S上部署Redis 集群)》,敬请期待。

本人水平有限,难免有错误或遗漏之处,望大家指正和谅解,欢迎评论留言。

欢迎关注本人微信公众号:


爱你之心.jpg

推荐阅读更多精彩内容