Managing Ceph RBD Images with Go-ceph

This is a creation in Article, where the information may have evolved or changed.

In the article "using Ceph RBD to provide storage volumes for kubernetes clusters," we learned that one step in the integration process for kubernetes and Ceph is to manually create the RBD image below the Ceph OSD pool. We need to find a way to remove this manual step. The first thing that comes to mind about the scenario is whether you can invoke the rest API provided by Ceph to manage the RBD pool and image?

Ceph provides two sets of REST API scenarios: CEPH-REST-API and Calamari. However, from the available data, neither of these rest APIs seem to provide a service interface to manipulate the image under the pool. Calamari plans to implement the service interface for image, but it has not been implemented at this time.

In the case where the Ceph REST API covers the RBD, we can only do it ourselves: we need to use Ceph to provide the library API for managing pool and image, and to provide custom service APIs externally. If you are a gopher, then go-ceph this Golang ceph library API binding will bring you a little help. Go-ceph is essentially the golang binding of a Ceph C library through CGO, covering more comprehensive: Rados, RBD, and CEPHFS support.

I. Installation of GO-CEPH and dependencies

First, because of the use of CGO, the program using the Go-ceph package is necessary to link Ceph's C library in compiling the situation, so we need to install some dependencies on the Go-ceph package first in the development environment (on Ubuntu 14.04):

# apt-get install librados-dev# apt-get install librbd-dev# ls /usr/include/radosbuffer_fwd.h  buffer.h  crc32c.h  librados.h  librados.hpp  memory.h  page.h  rados_types.h  rados_types.hpp# ls /usr/include/rbdfeatures.h  librbd.h  librbd.hpp

The next step is to install Go-ceph itself, which we can download to the Go-ceph package with the most popular go get commands.

# go get github.com/ceph/go-ceph

Second, Go-ceph: Connect ceph Cluster

Go-ceph is not a lot of documents, but Go-ceph is not difficult to use, about the usage of each package in Go-ceph, you can refer to the *_test.go file in the corresponding package.

One of the ways to connect a ceph cluster is as follows:

//github.com/bigwhite/experiments/blob/master/go-ceph/conn.gopackage mainimport (    "fmt"    "github.com/ceph/go-ceph/rados")func main() {    conn, err := rados.NewConn()    if err != nil {        fmt.Println("error when invoke a new connection:", err)        return    }    err = conn.ReadDefaultConfigFile()    if err != nil {        fmt.Println("error when read default config file:", err)        return    }    err = conn.Connect()    if err != nil {        fmt.Println("error when connect:", err)        return    }    fmt.Println("connect ceph cluster ok!")    conn.Shutdown()}

Here the Conn object takes the Mon node information in a way that reads the default configuration file (/etc/ceph/ceph.conf), which is also known in the Go-ceph document as command-line arguments and environment variables. But the way the command line arguments, I personally tried several times have not been able to connect. Even the parameters of the document that follow the Librados C API are not passed.

Third, Go-ceph: Manage Pool

Pool is a logical concept for ceph clusters, where a ceph cluster can have multiple pool, each of which is a logically isolated unit. The different pool can have a completely dissimilar data processing method, such as replica Size (number of copies), Placement Groups, CRUSH Rules, snapshot, owner, etc. Go-ceph supports the creation, viewing, and deletion of pool management operations:

Github.com/bigwhite/experiments/blob/master/go-ceph/pool.go. Func newconn () (*rados. Conn, error) {Conn, err: = Rados. Newconn () if err! = Nil {return nil, err} ERR = conn. Readdefaultconfigfile () if err! = Nil {return nil, err} ERR = conn. Connect () if err! = Nil {return nil, err} return conn, Nil}func Listpools (Conn *rados. Conn, prefix string) {pools, err: = Conn. Listpools () if err! = Nil {fmt. PRINTLN ("Error when list Pool", err) OS. Exit (1)} FMT. Println (prefix, ":", pools)}func main () {conn, err: = Newconn () if err! = Nil {fmt. PRINTLN ("Error when invoke a new connection:", err) return} defer conn. Shutdown () fmt.    Println ("Connect ceph cluster ok!") Listpools (conn, "Before make New pool") Err = conn. Makepool ("New_pool") if err! = Nil {fmt. PRINTLN ("Error when do New_pool", err) return} listpools (conn, "After make new pool") Err = CoNn. Deletepool ("New_pool") if err! = Nil {fmt. PRINTLN ("Error when delete pool", err) return} listpools (conn, "after delete New_pool")}

Execution Pool.go:

# go run pool.goconnect ceph cluster ok!before make new pool : [rbd rbd1]after make new pool : [rbd rbd1 new_pool]after delete new_pool : [rbd rbd1]

Iv. Go-ceph: Managing Image

Image is the object we really want to manage (the pool can take the default "RBD"), and image management relies on the RBD package under Go-ceph:

Github.com/bigwhite/experiments/blob/master/go-ceph/image.go. Func listimages (Ioctx *rados. Iocontext, prefix string) {imagenames, err: = RBD. Getimagenames (IOCTX) if err! = Nil {fmt. PRINTLN ("Error when getimagesnames", err) OS. Exit (1)} FMT. Println (prefix, ":", Imagenames)}func main () {conn, err: = Newconn () if err! = Nil {fmt. PRINTLN ("Error when invoke a new connection:", err) return} defer conn. Shutdown () fmt.    Println ("Connect ceph cluster ok!") Ioctx, ERR: = conn. Openiocontext ("RBD") if err! = Nil {fmt. PRINTLN ("Error when Openiocontext", err) return} defer ioctx. Destroy () listimages (Ioctx, "before create New image") Name: = "Go-ceph-image" img, err: = RBD. Create (ioctx, Name, 1<<20,) if err! = Nil {fmt. PRINTLN ("error when create RBD image", err) return} listimages (Ioctx, "after Create new image") Err = img . Remove () if err! = Nil {fmt. PrINTLN ("Error when remove image", err) return} listimages (Ioctx, "after remove new image")} 

The note here is RBD. Create this method, if the third parameter (image size) is passed too small, then RBD. Create will error, for example, if we change that code to:

img, err := rbd.Create(ioctx, name, 1<<10, 10)

Then when you execute Image.go, you get an error:

error when create rbd image rbd: ret=-33

33 is the Linux errno, which means:

#define EDOM        33  /* Math argument out of domain of func */

Guessing that the units of this parameter are bytes, the legal scope of the specific parameters, the document and the code are not given explicit instructions.

V. Summary

Go-ceph implements the basic management functions of RBD Pool/images and lays the foundation for providing RBD restful APIs. After writing three long articles, to a short, nutrition is not much, for the memo is OK.

, Bigwhite. All rights reserved.