Deploying Heketi and Glusterfs in Kubernetes (ii)

Deploying Heketi and Glusterfs in Kubernetes (ii)
In the previous section, Heketi was not deployed in a production environment because the data for the Heketi pod was not persisted, causing Heketi data loss and Heketi data saved in /var/lib/heketi/heketi.dbfile, you need to mount this directory to Glusterfs distributed storage.

Follow the steps in the previous section to perform HEKETI-CLI topology load--json=topology-sample.json

$ echo $HEKETI_CLI_SERVERhttp://10.254.49.43:8080$ heketi-cli topology load --json=topology-sample.jsonCreating cluster ... ID: 5b930ef6081fd22e895c25a3dfb0c516    Allowing file volumes on cluster.    Allowing block volumes on cluster.    Creating node 10.30.1.15 ... ID: b120572be40db6c1d979c3903876430b        Adding device /dev/sdb ... OK    Creating node 10.30.1.16 ... ID: 7ce13ffc5eabe64a3791e93233fd3c1a        Adding device /dev/sdb ... OK    Creating node 10.30.1.17 ... ID: f9abdc2e5d4cfa17c035a97f984a1a3b        Adding device /dev/sdb ... OK

Immediately following execution: The Heketi-storage.json file will be generated.

$ heketi-cli setup-openshift-heketi-storageSaving heketi-storage.json

When you're done, look at the pod Deploy-heketi log information to see what's done:

#只截取了部分日志, the basic operation is to enter into each glusterfs pod create brick directory and create a copy of the replicate volume, volume named Heketidbstorage[kubeexec] DEBUG 2018 /07/09 07:07:23/src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246:host:10.30.1.17 Pod: Glusterfs-8qrpt command:mkdir-p/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_ 6f0ce82692e70ce5ae2ec55a60f237c6result: [kubeexec] DEBUG 2018/07/09 07:07:23/src/github.com/heketi/heketi/ executors/kubeexec/kubeexec.go:246:host:10.30.1.15 pod:glusterfs-c4859 command:mkdir-p/var/lib/heketi/mounts/vg_ 19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4result: [kubeexec] DEBUG 2018/07/09 07:07:23/src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246:host:10.30.1.16 POD:GLUSTERFS-25CM8 Command:mkdir-p/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ Ba4091b858d94a088b21a582d8d4abaa[kubeexec] DEBUG 2018/07/09 07:07:26/src/github.com/heketi/heketi/executors/ kubeexec/kubeexec.go:246:host:10.30.1.17 PoD:glusterfs-8qrpt Command:mkdir/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_ 6f0ce82692e70ce5ae2ec55a60f237c6/brick[kubeexec] DEBUG 2018/07/09 07:07:26/src/github.com/heketi/heketi/executors /kubeexec/kubeexec.go:246:host:10.30.1.17 pod:glusterfs-8qrpt Command:mkdir/var/lib/heketi/mounts/vg_ A146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246:host:10.30.1.15 pod:glusterfs-c4859 Command: mkdir/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick[ Kubeexec] DEBUG 2018/07/09 07:07:26/src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246:host: 10.30.1.16 POD:GLUSTERFS-25CM8 Command:mkdir/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ Ba4091b858d94a088b21a582d8d4abaa/brickresult: [CmdExec] INFO 2018/07/09 07:07:26 Creating Volume heketidbstorage Replica 3[kUbeexec] DEBUG 2018/07/09 07:07:27/src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246:host: 10.30.1.16 POD:GLUSTERFS-25CM8 command:gluster--mode=script Volume create heketidbstorage replica 3 10.30.1.16:/var/li B/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick 10.30.1.17:/ Var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/ Brickresult:volume create:heketidbstorage:success:please Start the volume to access data

Go to any glusterfs pod to view volume information:

Analysis under Heketi-storage.json:

#将创建如下资源信息: endpoints:name:heketi-storage-endpoints 10.30.1.16:1 10.30.1.15:1 10.30.1.17:1service:name:          Heketi-storage-endpoints Job:name:heketi-storage-copy-job Images:heketi/heketi:dev declared the Volume:heketi-storage                                    "Volumes": [{"Name": "Heketi-storage",                                         "Glusterfs": {"Endpoints": "Heketi-storage-endpoints",                                "Path": "Heketidbstorage"}                                                }, Mount to/heketi:volumemounts ": [{ "Name": "Heketi-storage", "Mountpath": "/hek ETI "}, execute command at startup: Cp/db/heketi.db/heketi #由此可知, this job is to copy data files from Heketi to/h Eketi, while the/heketi directory is mounted on the volume HekEti-storage, and Heketi-storage volume is created when you execute "heketi-cli setup-openshift-heketi-storage" in front of the 

Create it:

$ kubectl create -f heketi-storage.jsonsecret "heketi-storage-secret" createdendpoints "heketi-storage-endpoints" createdservice "heketi-storage-endpoints" createdjob "heketi-storage-copy-job" created

When the job is finished, you can delete it:

$ kubectl get job NAME                      DESIRED   SUCCESSFUL   AGEheketi-storage-copy-job   1         1            1m

When the job is complete, delete the components associated with the bootstrap Heketi instance:

#把之前由heketi-bootstrap.json创建的资源删除$ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"deployment "deploy-heketi" deletedjob "heketi-storage-copy-job" deletedpod "deploy-heketi-69bfbd4bbd-q8tsk" deletedservice "deploy-heketi" deletedsecret "heketi-storage-secret" deleted

The previously created Pod,service named Deploy-heketi has been deleted:

$ kubectl get podNAME              READY     STATUS              RESTARTS   AGEglusterfs-25cm8   1/1       Running             1          1hglusterfs-8qrpt   1/1       Running             1          1hglusterfs-c4859   1/1       Running             1          1h$ kubectl get svcNAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGEheketi-storage-endpoints      ClusterIP   10.254.191.233   <none>        1/TCP      4m

Finally, redeploy the Heketi using the Heketi-deployment.json file

$ cat Heketi-deployment.json {"kind": "List", "apiversion": "V1", "items": [{"Kind": "Secret", "apivers          Ion ":" V1 "," metadata ": {" name ":" Heketi-db-backup "," labels ": {" Glusterfs ":" Heketi-db ",      "Heketi": "DB"}, "Data": {}, "type": "Opaque"}, {"Kind": "Service", "Apiversion": "V1", "metadata": {"name": "Heketi", "labels": {"Glusterfs": "Heketi-service        "," Deploy-heketi ":" Support "}," annotations ": {" description ":" Exposes Heketi Service "            }}, "Spec": {"selector": {"name": "Heketi"}, "ports": [{      "Name": "Heketi", "Port": 8080, "Targetport": 8080}]}}, { "Kind": "Deployment", "apiversion": "Extensions/v1beta1", "metadata": {"name": "Heketi", "labels ": {" Glusterfs":" Heketi-deployment "}," annotations ": {" description ":" Defines how to deploy Heketi "}            }, "spec": {"Replicas": 1, "template": {"metadata": {"name": "Heketi",          "Labels": {"name": "Heketi", "Glusterfs": "Heketi-pod"}},                "Spec": {"Serviceaccountname": "Heketi-service-account", "containers": [{                "Image": "Heketi/heketi:dev", "Imagepullpolicy": "Always", "name": "Heketi",                  "env": [{"Name": "Heketi_executor", "Value": "Kubernetes" }, {"Name": "Heketi_db_path", "Value": "/var/lib/hek Eti/heketi.db "}, {" Name ":" Heketi_fstab "," value ":"/var/lib/heketi/fstaB "}, {" Name ":" Heketi_snapshot_limit "," value ":                    "+"}, {"Name": "Heketi_kube_gluster_daemonset", "Value": "Y"}], "ports": [{"Contai                    Nerport ": 8080}]," volumemounts ": [{                    "Mountpath": "/backupdb", "name": "Heketi-db-secret"}, {                    "Name": "DB", "Mountpath": "/var/lib/heketi"}, {                "Name": "config", "Mountpath": "/etc/heketi"}], "Readinessprobe": {"timeoutseconds": 3, "Initialdelayseconds": 3, "             HttpGet ": {       "Path": "/hello", "Port": 8080}}, "Livenessprob                    E ": {" timeoutseconds ": 3," initialdelayseconds ": +," HttpGet ": {            "Path": "/hello", "Port": 8080}}                  ], "volumes": [{"Name": "DB", "Glusterfs": {              "Endpoints": "Heketi-storage-endpoints", "path": "Heketidbstorage"}}, {"Name": "Heketi-db-secret", "secret": {"Secretname": "Heketi-db                  -backup "}}, {" name ":" Config "," secret ": {    "Secretname": "Heketi-config-secret"}}]}} }  ]}

The Heketi-deployment.json file creates the following resources:

Service:     name: heketi     port: 8080Deployment：    name: heketi    replicas: 1    image: heketi/heketi:dev    volumeMounts:            name: db   mountPath: /var/lib/heketi    volumes:       endpoints: heketi-storage-endpoints   #由heketi-storage.json文件创建       path：heketidbstorage  #这是gluster volume名，此volume是由"heketi-cli setup-openshift-heketi-storage"自动创建。    # 将heketi容器内的/var/lib/heketi 目录挂载到了GlusterFS volume “heketidbstorage”中。

Deploy it:

$ kubectl create -f heketi-deployment.json secret "heketi-db-backup" createdservice "heketi" createddeployment "heketi" created$ kubectl get deployment NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGEheketi    1         1         1            1           45s$ kubectl get svc NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGEheketi                        ClusterIP   10.254.239.189   <none>        8080/TCP   51sheketi-storage-endpoints      ClusterIP   10.254.191.233   <none>        1/TCP      31m

Verify that Heketi is being used with Gluster volume:

$ kubectl get podNAME                      READY     STATUS              RESTARTS   AGEglusterfs-25cm8           1/1       Running             1          1hglusterfs-8qrpt           1/1       Running             1          1hglusterfs-c4859           1/1       Running             1          1hheketi-7898db85dd-nb6kn   1/1       Running             0          1m$ kubectl exec heketi-7898db85dd-nb6kn -it bash[[email protected] /]# mount |grep heketi10.30.1.15:heketidbstorage on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

At this point, Heketi DB has been properly configured with the Glusterfs volume.

Deploying Heketi and Glusterfs in Kubernetes (ii)