Kubernetes上部署Heketi Gluster

前言

Heketi

Heketi提供了一个RESTful管理界面，可以用来管理GlusterFS卷的生命周期。通过Heketi，就可以像使用OpenStack Manila，Kubernetes和OpenShift一样申请可以动态配置GlusterFS卷。Heketi会动态在集群内选择bricks构建所需的volumes，这样以确保数据的副本会分散到集群不同的故障域内。同时Heketi还支持任意数量的ClusterFS集群，以保证接入的云服务器不局限于单个GlusterFS集群。

Gluster-Kubernetes

Gluster-Kubernetes是一个可以将GluserFS和Hekiti轻松部署到Kubernetes集群的开源项目。另外也提供在Kubernetes中可以采用StorageClass来动态管理GlusterFS卷。

部署

Heketi

服务

组件	版本
Kubernetes	1.9.1
Gluster-Kubernetes	v1.2.0
GlusterFS	3.12.1

服务器配置

服务器	存储IP	hostname	硬盘
node1	10.17.64.14	10.17.64.14	/dev/sdc
node2	10.17.64.15	10.17.64.15	/dev/sdc
node3	10.17.64.16	10.17.64.16	/dev/sdc
node4	10.17.64.17	10.17.64.17	/dev/sdc
node5	10.17.64.18	10.17.64.18	/dev/sdc
node6	10.17.64.19	10.17.64.19	/dev/sdc

1. 下载cluster-kubernetes源码

$ wget https://github.com/gluster/gluster-kubernetes/archive/v1.2.0.zip

$ unzip v1.2.0.zip

$ ls gluster-kubernetes-1.2.0

LICENSE   Makefile  README.md deploy    docs      tests     vagrant

2. 同步各节点的hosts文件

3. 修改topology.json文件。

$ cat deploy/topology.json (转成yaml格式，部署时请转成json格式.)

---
clusters:
- nodes:
  - node:
      hostnames:
        manage:
        - 10.17.64.14
        storage:
        - 10.17.64.14
      zone: 1
    devices:
    - "/dev/sdc"
  - node:
      hostnames:
        manage:
        - 10.17.64.15
        storage:
        - 10.17.64.15
      zone: 1
    devices:
    - "/dev/sdc"
  - node:
      hostnames:
        manage:
        - 10.17.64.16
        storage:
        - 10.17.64.16
      zone: 1
    devices:
    - "/dev/sdc"
  - node:
      hostnames:
        manage:
        - 10.17.64.17
        storage:
        - 10.17.64.17
      zone: 1
    devices:
    - "/dev/sdc"
  - node:
      hostnames:
        manage:
        - 10.17.64.18
        storage:
        - 10.17.64.18
      zone: 1
    devices:
    - "/dev/sdc"
  - node:
      hostnames:
        manage:
        - 10.17.64.19
        storage:
        - 10.17.64.19
      zone: 1
    devices:
    - "/dev/sdc"

4. 部署Heketi

$ kubectl create ns heketi
$ ./gk-deploy -g -n heketi topology.json

5. 部署完成后可以看下具体的部署内容

labels: nodelabels

$ kubectl get node --show-labels
NAME          STATUS                     ROLES     AGE       VERSION   LABELS
10.17.64.14   Ready                      <none>    5d        v1.9.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.17.64.14,storagenode=glusterfs
10.17.64.15   Ready,SchedulingDisabled   <none>    5d        v1.9.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.17.64.15,storagenode=glusterfs
10.17.64.16   Ready,SchedulingDisabled   <none>    5d        v1.9.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.17.64.16,storagenode=glusterfs
10.17.64.17   Ready,SchedulingDisabled   <none>    5d        v1.9.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.17.64.17,storagenode=glusterfs
10.17.64.18   Ready,SchedulingDisabled   <none>    5d        v1.9.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.17.64.18,storagenode=glusterfs
10.17.64.19   Ready,SchedulingDisabled   <none>    5d        v1.9.1    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.17.64.19,storagenode=glusterfs

secrets: heketi-service-account-token-bpt9h

apiVersion: v1
data:
  ca.crt: aaaa
  token: bbbb
  namespace: cccc
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: heketi-service-account
  name: heketi-service-account-token-bpt9h
  namespace: heketi
type: kubernetes.io/service-account-token

serviceaccounts: heketi-service-account

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    glusterfs: heketi-sa
    heketi: sa
  name: heketi-service-account
  namespace: heketi
secrets:
- name: heketi-service-account-token-bpt9h

clusterrolebindings: heketi-sa-view

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    glusterfs: heketi-sa-view
    heketi: sa-view
  name: heketi-sa-view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- kind: ServiceAccount
  name: heketi-service-account
  namespace: heketi

daemonset: glusterfs

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  annotations:
    description: GlusterFS DaemonSet
    tags: glusterfs
  labels:
    glusterfs: daemonset
  name: glusterfs
  namespace: heketi
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      glusterfs: pod
      glusterfs-node: pod
  template:
    metadata:
      labels:
        glusterfs: pod
        glusterfs-node: pod
      name: glusterfs
    spec:
      containers:
      - env:
        - name: GB_GLFS_LRU_COUNT
          value: "15"
        - name: TCMU_LOGDIR
          value: /var/log/glusterfs/gluster-block
        image: gluster/gluster-centos:latest
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl status glusterd.service
          failureThreshold: 15
          initialDelaySeconds: 40
          periodSeconds: 25
          successThreshold: 1
          timeoutSeconds: 3
        name: glusterfs
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - systemctl status glusterd.service
          failureThreshold: 15
          initialDelaySeconds: 40
          periodSeconds: 25
          successThreshold: 1
          timeoutSeconds: 3
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          capabilities: {}
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/heketi
          name: glusterfs-heketi
        - mountPath: /run
          name: glusterfs-run
        - mountPath: /run/lvm
          name: glusterfs-lvm
        - mountPath: /etc/glusterfs
          name: glusterfs-etc
        - mountPath: /var/log/glusterfs
          name: glusterfs-logs
        - mountPath: /var/lib/glusterd
          name: glusterfs-config
        - mountPath: /dev
          name: glusterfs-dev
        - mountPath: /var/lib/misc/glusterfsd
          name: glusterfs-misc
        - mountPath: /sys/fs/cgroup
          name: glusterfs-cgroup
          readOnly: true
        - mountPath: /etc/ssl
          name: glusterfs-ssl
          readOnly: true
      hostNetwork: true
      nodeSelector:
        storagenode: glusterfs
      restartPolicy: Always
      volumes:
      - hostPath:
          path: /var/lib/heketi
          type: ""
        name: glusterfs-heketi
      - emptyDir: {}
        name: glusterfs-run
      - hostPath:
          path: /run/lvm
          type: ""
        name: glusterfs-lvm
      - hostPath:
          path: /etc/glusterfs
          type: ""
        name: glusterfs-etc
      - hostPath:
          path: /var/log/glusterfs
          type: ""
        name: glusterfs-logs
      - hostPath:
          path: /var/lib/glusterd
          type: ""
        name: glusterfs-config
      - hostPath:
          path: /dev
          type: ""
        name: glusterfs-dev
      - hostPath:
          path: /var/lib/misc/glusterfsd
          type: ""
        name: glusterfs-misc
      - hostPath:
          path: /sys/fs/cgroup
          type: ""
        name: glusterfs-cgroup
      - hostPath:
          path: /etc/ssl
          type: ""
        name: glusterfs-ssl
  updateStrategy:
    type: OnDelete

deployment: heketi

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    description: Defines how to deploy Heketi
  labels:
    glusterfs: heketi-deployment
    heketi: deployment
  name: heketi
  namespace: heketi
spec:
  replicas: 1
  selector:
    matchLabels:
      glusterfs: heketi-pod
      heketi: pod
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        glusterfs: heketi-pod
        heketi: pod
      name: heketi
    spec:
      containers:
      - env:
        - name: HEKETI_USER_KEY
        - name: HEKETI_ADMIN_KEY
        - name: HEKETI_EXECUTOR
          value: kubernetes
        - name: HEKETI_FSTAB
          value: /var/lib/heketi/fstab
        - name: HEKETI_SNAPSHOT_LIMIT
          value: "14"
        - name: HEKETI_KUBE_GLUSTER_DAEMONSET
          value: "y"
        image: heketi/heketi:dev
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /hello
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        name: heketi
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /hello
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        volumeMounts:
        - mountPath: /var/lib/heketi
          name: db
        - mountPath: /etc/heketi
          name: config
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      serviceAccount: heketi-service-account
      serviceAccountName: heketi-service-account
      volumes:
      - glusterfs:
          endpoints: heketi-storage-endpoints
          path: heketidbstorage
        name: db
      - name: config
        secret:
          defaultMode: 420
          secretName: heketi-config-secret

service: heketi

apiVersion: v1
kind: Service
metadata:
  annotations:
    description: Exposes Heketi Service
  labels:
    glusterfs: heketi-service
    heketi: service
  name: heketi
  namespace: heketi
  spec:
  ports:
  - name: heketi
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    glusterfs: heketi-pod
  sessionAffinity: None
  type: ClusterIP

Heketi-Cli

1. 下载部署Heketi-Cli二进制代码

$ wget https://github.com/heketi/heketi/releases/download/v5.0.1/heketi-client-v5.0.1.linux.amd64.tar.gz

$ tar xf heketi-client-v5.0.1.linux.amd64.tar.gz

$ mv heketi-client/bin/heketi-cli /usr/bin/heketi-cli

2. Cli的使用

参看集群信息

$ heketi-cli -s http://10.17.74.130:8080 cluster list

Clusters:
Id:d1932cc37442e2a62311fdd38bfe7ad4

$ heketi-cli -s http://10.17.74.130:8080 cluster info d1932cc37442e2a62311fdd38bfe7ad4

Nodes:
0a5dacb89307c6427e2bcd2df58a9731
2a8d50de2aea3edd69950fa31286e20a
4216c8c67263047851f0e76263546fba
537ff5b706551997f0cbd771465d3faa
67c1017f74ebb0a0fdfc7519f5899cbc
ea7b16b923f0f1526eb597cb94c4014d
Volumes:
9228fbbba5c943a3a84c31135bd25e8c

查看node信息

$ heketi-cli -s http://10.17.74.130:8080 node list

Id:0a5dacb89307c6427e2bcd2df58a9731 Cluster:d1932cc37442e2a62311fdd38bfe7ad4
Id:2a8d50de2aea3edd69950fa31286e20a Cluster:d1932cc37442e2a62311fdd38bfe7ad4
Id:4216c8c67263047851f0e76263546fba Cluster:d1932cc37442e2a62311fdd38bfe7ad4
Id:537ff5b706551997f0cbd771465d3faa Cluster:d1932cc37442e2a62311fdd38bfe7ad4
Id:67c1017f74ebb0a0fdfc7519f5899cbc Cluster:d1932cc37442e2a62311fdd38bfe7ad4
Id:ea7b16b923f0f1526eb597cb94c4014d Cluster:d1932cc37442e2a62311fdd38bfe7ad4

$ heketi-cli -s http://10.17.74.130:8080 node info 0a5dacb89307c6427e2bcd2df58a9731

Node Id: 0a5dacb89307c6427e2bcd2df58a9731
State: online
Cluster Id: d1932cc37442e2a62311fdd38bfe7ad4
Zone: 1
Management Hostname: 10.17.64.15
Storage Hostname: 10.17.64.15
Devices:
Id:81ed9952dba486619cc7685928c78857   Name:/dev/sdc            State:online    Size (GiB):1862    Used (GiB):2       Free (GiB):1860

查看volume信息

$ heketi-cli -s http://10.17.74.130:8080 volume list

Id:9228fbbba5c943a3a84c31135bd25e8c    Cluster:d1932cc37442e2a62311fdd38bfe7ad4    Name:heketidbstorage

$ heketi-cli -s http://10.17.74.130:8080 volume info 9228fbbba5c943a3a84c31135bd25e8c

Name: heketidbstorage
Size: 2
Volume Id: 9228fbbba5c943a3a84c31135bd25e8c
Cluster Id: d1932cc37442e2a62311fdd38bfe7ad4
Mount: 10.17.64.15:heketidbstorage
Mount Options: backup-volfile-servers=10.17.64.17,10.17.64.18,10.17.64.14,10.17.64.16,10.17.64.19
Durability Type: replicate
Distributed+Replica: 3

3. 接入StorageClass

heketi-kubernetes.storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: heketi-kubernetes
parameters:
  clusterid: d1932cc37442e2a62311fdd38bfe7ad4
  gidMax: "50000"
  gidMin: "40000"
  restauthenabled: "false"            //关闭auth认证
  resturl: http://10.17.74.130:8080
  restuser: admin
  secretName: heketi-config-secret
  secretNamespace: heketi
  volumetype: replicate:3
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete

4. 创建pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: magine1989
  annotations:
    volume.beta.kubernetes.io/storage-class: heketi-glusterfs
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

遇到的坑

1. ubuntu在部署Heketi时，宿主机内核需要加载dm_thin_pool模块。

$ lsmod |grep thin

dm_thin_pool           65536  2
dm_persistent_data     69632  1 dm_thin_pool
dm_bio_prison          20480  1 dm_thin_pool

2. topology.json如果device填写有误，需要把heketi的所有的Resource删掉重新创建。

3. heketi需要调用kubernetes api获取glusterfs node信息，如果API地址不通则会出现创建volume失败.