一、前言

上一文《从零开始搭建Kubernetes集群（四、搭建K8S Dashboard）》介绍了如何搭建Dashboard。本篇将介绍如何搭建Ingress来访问K8S集群的Service。

二、Ingress简介

Ingress是个什么鬼，网上资料很多（推荐官方），大家自行研究。简单来讲，就是一个负载均衡的玩意，其主要用来解决使用NodePort暴露Service的端口时Node IP会漂移的问题。同时，若大量使用NodePort暴露主机端口，管理会非常混乱。

好的解决方案就是让外界通过域名去访问Service，而无需关心其Node IP及Port。那为什么不直接使用Nginx？这是因为在K8S集群中，如果每加入一个服务，我们都在Nginx中添加一个配置，其实是一个重复性的体力活，只要是重复性的体力活，我们都应该通过技术将它干掉。

Ingress就可以解决上面的问题，其包含两个组件Ingress Controller和Ingress：

• Ingress
  将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可
• Ingress Controller
  将新加入的Ingress转化成Nginx的配置文件并使之生效

好了，废话不多，走你~

三、准备操作

官方文档

人生苦短，不造轮子，本文将以官方的标准脚本为基础进行搭建，参考请戳官方文档。官方文档中要求依次执行如下命令：

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
    | kubectl apply -f -

以上yaml文件创建Ingress用到的Namespace、ConfigMap，以及默认的后端default-backend。最关键的一点是，由于之前我们基于Kubeadm创建了K8S集群，则还必须执行：

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \
    | kubectl apply -f -

这是由于Kubeadm创建的集群默认开启了RABC，因此Ingress也必须创建相应的RABC权限控制。

导入镜像

但是，直接按照上述方式执行，我们的Ingress很可能会无法使用。所以，我们需要将上述Yaml文件全部wget下来，经过一些修改后才能执行kubectl apply -f创建。另外需要注意的是，这些yaml文件中提到的一些镜像，国内目前无法下载，如：

gcr.io/google_containers/defaultbackend:1.4
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0

本人已经提前下载好，大家请戳：

地址：https://pan.baidu.com/s/1N-bK9hI7JTZZB6AzmaT8PA
密码：1a8a

拿到镜像后，在每个节点上执行如下命令导入镜像：

docker load < quay.io#kubernetes-ingress-controller#nginx-ingress-controller_0.14.0.tar
docker tag 452a96d81c30 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
docker load < gcr.io#google_containers#defaultbackend.tar
docker tag 452a96d81c30 gcr.io/google_containers/defaultbackend

如上所示，导入镜像后，别忘记给打tag，否则镜像名称为<none>：

image.png

四、主要文件介绍

这里，我们先对一些重要的文件进行简单介绍。

default-backend.yaml

default-backend的作用是，如果外界访问的域名不存在的话，则默认转发到default-http-backend这个Service，其会直接返回404：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend

rbac.yaml

rbac.yaml负责Ingress的RBAC授权的控制，其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding。在上文《从零开始搭建Kubernetes集群（四、搭建K8S Dashboard）》中，我们已对这些概念进行了简单介绍。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

with-rbac.yaml

with-rbac.yaml是Ingress的核心，用于创建ingress-controller。前面提到过，ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置。

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          securityContext:
            runAsNonRoot: false

如上，可以看到nginx-ingress-controller启动时传入了参数，分别为前面创建的default-backend-service以及configmap。

五、创建Ingress

1.创建Ingress-controller

需要注意的是，官方提供的with-rbac.yaml文件不能直接使用，我们必须修改两处：

加入hostNetwork配置

如下，在serviceAccountName上方添加hostNetwork: true:

spec:
      hostNetwork: true
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io

配置hostNetwork: true是一种直接定义Pod网络的方式。定义后，Ingress-controller的IP就与宿主机k8s-node1一样(192.168.56.101)，并且端口80也是宿主机上的端口。这样，我们通过该192.168.56.101:80，就可以直接访问到Ingress-controller（实际上就是nginx），然后Ingress-controller则会转发我们的请求到相应后端。

加入环境变量

在其env部分加入如下环境变量：

          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_MASTER 
              value: http://192.168.56.101:8080

否则，创建后会提示如下错误：

[root@k8s-node1 ingress]# kubectl describe pod nginx-ingress-controller-9fbd7596d-rt9sf  -n ingress-nginx
省略前面...
Events:
  Type     Reason                 Age                From                Message
  ----     ------                 ----               ----                -------
  Normal   Scheduled              30s                default-scheduler   Successfully assigned nginx-ingress-controller-9fbd7596d-rt9sf to k8s-node1
  Normal   SuccessfulMountVolume  30s                kubelet, k8s-node1  MountVolume.SetUp succeeded for volume "nginx-ingress-serviceaccount-token-lq2dt"
  Warning  BackOff                21s                kubelet, k8s-node1  Back-off restarting failed container
  Normal   Pulled                 11s (x3 over 29s)  kubelet, k8s-node1  Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0" already present on machine
  Normal   Created                11s (x3 over 29s)  kubelet, k8s-node1  Created container
  Warning  Failed                 10s (x3 over 28s)  kubelet, k8s-node1  Error: failed to start container "nginx-ingress-controller": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory": unknown

修改with-rbac.yaml后，使用kubectl -f create命令分别执行如下yaml文件，即可创建Ingress-controller：

image.png

创建成功后如下所示：

[root@k8s-node1 ingress]# kubectl get pod -n ingress-nginx -o wide
NAME                                        READY     STATUS    RESTARTS   AGE       IP              NODE
default-http-backend-5c6d95c48-pdjn9        1/1       Running   0          23s       192.168.36.81   k8s-node1
nginx-ingress-controller-547cd7d9cb-jmvpn   1/1       Running   0          8s        192.168.36.82   k8s-node1

2.创建自定义Ingress

有了ingress-controller，我们就可以创建自定义的Ingress了。这里已提前搭建好了Kibana服务，我们针对Kibana创建一个Ingress：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: default

spec:
  rules:
  - host: myk8s.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601

其中：

• rules中的host必须为域名，不能为IP，表示Ingress-controller的Pod所在主机域名，也就是Ingress-controller的IP对应的域名。
• paths中的path则表示映射的路径。如映射/表示若访问myk8s.com，则会将请求转发至Kibana的service，端口为5601。

创建成功后，查看：

[root@k8s-node1 ingress]# kubectl get ingress -o wide
NAME             HOSTS       ADDRESS   PORTS     AGE
kibana-ingress   myk8s.com             80        6s

我们再执行kubectl exec nginx-ingress-controller-5b79cbb5c6-2zr7f -it cat /etc/nginx/nginx.conf -n ingress-nginx，可以看到生成nginx配置，篇幅较长，各位自行筛选：

    ## start server myk8s.com
    server {
        server_name myk8s.com ;
        
        listen 80;
        
        listen [::]:80;
        
        set $proxy_upstream_name "-";
        
        location /kibana {
            
            log_by_lua_block {
                
            }
            
            port_in_redirect off;
            
            set $proxy_upstream_name "";
            
            set $namespace      "kube-system";
            set $ingress_name   "dashboard-ingress";
            set $service_name   "kibana";
            
            client_max_body_size                    "1m";
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Real-IP              $the_real_ip;
            
            proxy_set_header X-Forwarded-For        $the_real_ip;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Original-URI         $request_uri;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         "off";
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";
            
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
            proxy_next_upstream_tries               0;
            
            # No endpoints available for the request
            return 503;
            
        }
        
        location / {
            
            log_by_lua_block {
                
            }
            
            port_in_redirect off;
            
            set $proxy_upstream_name "";
            
            set $namespace      "default";
            set $ingress_name   "kibana-ingress";
            set $service_name   "kibana";
            
            client_max_body_size                    "1m";
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Real-IP              $the_real_ip;
            
            proxy_set_header X-Forwarded-For        $the_real_ip;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Original-URI         $request_uri;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         "off";
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";
            
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
            proxy_next_upstream_tries               0;
            
            # No endpoints available for the request
            return 503;
            
        }
        
    }
    ## end server myk8s.com

3.设置host

首先，我们需要在Ingress-controller的Pod所在主机上（这里为k8s-node1），将上面提到的域名myk8s.com追加入/etc/hosts文件：

192.168.56.101 myk8s.com

除此之外，如果想在自己的Windows物理机上使用浏览器访问kibana，也需要在C:\Windows\System32\drivers\etc\hosts文件内加入上述内容。设置后，分别在k8s-node1和物理机上测试无误即可：

image.png

image.png

六、测试

在Windows物理机上，使用Chrome访问myk8s.com，也就是相当于访问了192.168.56.101:80：

image.png

随意访问一个错误的地址myk8s.com/abc，返回预期的404：

image.png

七、废话

至此，我们的Ingress已经搭建完毕，实现了在外部通过域名访问K8S集群Service的功能。如果大家有兴趣，可以尝试为Ingress配置TLS，这样就可以访问如Dashboard这种https服务了。下一章节《从零开始搭建Kubernetes集群（五、在K8S上部署Redis 集群）》，敬请期待。

本人水平有限，难免有错误或遗漏之处，望大家指正和谅解，欢迎评论留言。