Deploying Heketi and Glusterfs in Kubernetes

Deploying Heketi and Glusterfs in Kubernetes

[TOC]

1. Preface

In Kubernetes, using the Glusterfs file system, the procedure is usually:
Create brick--> Create volume--> Create pv--> create pvc-->pod mount PVC
If you are creating more than one PV, you need to manually repeat these interlocking steps, Heketi can resolve these duplication problems.
Heketi is used to manage the life cycle of the glusterfs volume and provides a RESTful API interface for kubernetes calls because Glusterfs does not provide the means for API calls, so we use Heketi, through Heketi, Kubernetes can dynamically configure Glusterfs volumes, Heketi dynamically selects bricks within the cluster to create the volumes required to ensure that copies of the data are scattered across different fault domains in the cluster, while Heketi also supports Glusterfs multi-cluster management. Easy for administrators to operate on Glusterfs.
Heketi requires a bare disk on each glusterfs node, because Heketi is used to create PV and VG, and if there is a Heketi, the PV can be created by Storageclass, with only the following steps:
Create storageclass--> Create pvc-->pod mount PVC
This approach is called dynamic resource provisioning based on Storageclass, although there are only two simple steps, but it does not work much less than the above steps, but most of the work is done by Heketi behind us.

2. Environmental statement

# k8s 192.168.105.92 lab1  # master1192.168.105.93 lab2  # master2192.168.105.94 lab3  # master3192.168.105.95 lab4  # node4192.168.105.96 lab5  # node5192.168.105.97 lab6  # node6192.168.105.98 lab7  # node7

3. Gluster-kubernetes Deployment

Tag node that needs to deploy the Glusterfs node

[[email protected] glusterfs]# kubectl label node lab4 storagenode=glusterfs node/lab4 labeled[[email protected] glusterfs]# kubectl label node lab5 storagenode=glusterfs node/lab5 labeled[[email protected] glusterfs]# kubectl label node lab7 storagenode=glusterfsnode/lab7 labeled

Preparing the Deployment file

git clone https://github.com/gluster/gluster-kubernetes.gitcd gluster-kubernetes/deploymv topology.json.sample topology.json

Modify Configurationtopology.json

{"clusters": [{"Nodes": [{"Node": {"hostnames": { "Manage": ["LAB4"], "storage": ["192.168.105.95" ]}, "zone": 1}, "Devices": ["/dev/sdb"]} {"Node": {"hostnames": {"Manage": ["LAB5"], "Storage": ["192.168.105.96"]}, "zone": 1}, "Devices": ["/dev/sdb"]}, {"node": {"hostnames": { "Manage": ["Lab7"], "storage": ["192.168.105.98" ]}, "zone": 1}, "Devices": ["/dev/sdb"]} ] } ]}

topology-sample.jsonThe file, called the topology file, provides the Kubernetes node IP that runs the Gluster pod, the corresponding disk block device on each node, modifies the hostnames/manage, and sets the value to the Name field that is displayed with Kubectl get nodes, Typically node IP, modify the IP under Hostnames/storage, the IP address of the storage network, or node IP.

After the cluster deployment succeeds, the configuration is modified and needs to be loaded again using the following command:
/usr/bin/kubectl -n default exec -i $(kubectl get pod|grep heketi|awk ‘{print $1}‘) -- heketi-cli -s http://localhost:8080 --user admin --secret ‘‘ topology load --json=/etc/heketi/topology.json

After the execution heketi-cli topology load , what exactly did Heketi do on the server?
into any glusterfs pod, execution gluster peer status Discovery has added the peer to the trusted storage pool (TSP).
On the node running the Gluster pod, a VG is created automatically, which is topology.json created by the disk bare devices in the file.
A disk device creates a VG, a PVC that is created later, the LV that is divided from this VG.
HEKETI-CLI topology Info view topology, showing the ID of each disk device, corresponding to the VG ID, total space, used space, free space and other information.
Can be seen through the Heketi Pod log.

Perform deployment

./gk-deploy -g# bash -x ./gk-deploy -g  # 调试运行过程

Attention:

• The disk in JSON above is not new VG, PV.
• Deployment failure using the ./gk-deploy -g --abort delete pod, then emptying the directory of the node /var/lib/glusterd , deleting the disk's VG and PV
• After gk-deploy I have modified create -f into apply -f , to avoid secret not update, appropriate in the confirmation but also affect the script to run the local comments exit , such as Error: Volume heketidbstorage alreay exists

Problem
https://github.com/gluster/gluster-kubernetes/issues/507:

  Mountvolume.setup failed for volume "Heketi-storage": Mount Failed:mount failed:exit Status Mounting comm And:systemd-run mounting arguments:--description=kubernetes transient mount for/var/lib/kubelet/pods/ E2531fbe-a133-11e8-b55d-000c2931d938/volumes/kubernetes.io~glusterfs/heketi-storage--scope--Mount-t Glusterfs-o log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/ heketi-storage-copy-job-ptnx4-glusterfs.log,backup-volfile-servers= 192.168.105.95:192.168.105.96:192.168.105.98,log-level=error 192.168.105.95:heketidbstorage/var/lib/kubelet/ Pods/e2531fbe-a133-11e8-b55d-000c2931d938/volumes/kubernetes.io~glusterfs/heketi-storage output:running Scope as Unit Run-53418.scope. Mount:unknown filesystem type ' Glusterfs ' The following error information is pulled from the Glusterfs log to help Diagn OSE this issue:could not open log file for pod heketi-storage-copy-job-ptnx4  

The above indicates that the mount failed, in fact, needs to be installed on the running deploy-heketi node yum -y install glusterfs-fuse .

After all deployments are successful:

[[email protected] deploy]# kubectl get podNAME                                READY     STATUS    RESTARTS   AGEcurl-87b54756-wzm66                 1/1       Running   0          17hglusterfs-9xj2r                     1/1       Running   0          15hglusterfs-kbqpc                     1/1       Running   1          15hglusterfs-wwg5w                     1/1       Running   0          15hheketi-86f98754c-dvqpk              1/1       Running   0          38snginx-deployment-7f46fc97b9-hn8g7   1/1       Running   0          14hnginx-deployment-7f46fc97b9-t82fv   1/1       Running   0          17h[[email protected] deploy]# kubectl  exec -it heketi-86f98754c-dvqpk -- df |grep heketidb192.168.105.95:heketidbstorage   2086912   54280   2032632   3% /var/lib/heketi

export HEKETI_CLI_SERVER=$(kubectl get svc/deploy-heketi --template ‘http://{{.spec.clusterIP}}:{{(index .spec.ports 0).port}}‘)curl $HEKETI_CLI_SERVER/hello

Return to the following normal:

Hello from Heketi

Resources:
[1] http://blog.51cto.com/newfly/2134514
[2] http://blog.51cto.com/newfly/2139393
[3] Https://github.com/gluster/gluster-kubernetes
[4] Https://github.com/gluster/gluster-kubernetes/blob/master/docs/examples/hello_world/README.md

Deploying Heketi and Glusterfs in Kubernetes