Extended development _php Tutorial for Ceph management platform Calamari

Extended development of the Ceph management platform Calamari

Close to the big six months did not write the log, perhaps it is more and more lazy. But sometimes writing and writing can make a deposit, or come back and record it. into adult college half a year, familiar with some related work, currently mainly engaged in the research and development of distributed systems, the current development is mainly to stay in the management level of development, has not yet reached the revision code. This half-year time is familiar with two very good distributed systems Glusterfs and Ceph. Each of the two distributed storage products has advantages, where Glusterfs provides file services that the CEPH system cannot provide. The unified architecture of Ceph's block devices, object storage, and file systems is glusterfs. So each has an advantage.

From the code level, the Glusterfs code is simple, the level is obvious, the stack-type processing process is very clear. It is very easy to extend the functionality of the file system (add processing modules on the client and server side), although the server side, client code is a piece of code, but overall the code is clear, the code is small.

While Ceph uses C + + development, and the system itself has a number of processes, multiple processes constitute a large cluster, and there are small clusters within the cluster, relative to Glusterfs, the code is much more complex, and ceph itself to achieve self-tuning and self-repair. Support the customization of the software system, find the storage location of the object through the crush algorithm.

Ceph is a fire in terms of current heat, but Glusterfs is a good choice for file systems.

Recently engaged in the development of CEPH related management platform, familiar with the official Calamari platform, the platform currently mainly provides the management of Ceph distributed storage System, the overall main is to provide the means of page management ceph. From the current implementation point of view, the platform has some limitations, can not complete the powerful function, or the current version of the offer can only provide some basic functions. But Calamari's framework is really good. Ceph is open source software, Calamari is open source software, and Calamari is a series of open source software combinations, these open source software has only completed its specific functions. Although it is patchwork, the overall framework of the management platform is worthy of reference.
The following sections refer to http://www.openstack.cn/?p=2708.
Architecture diagram for Calamari

Where the red box is part of the Calamari Code implementation, the Non-red box part is the non-calamari implementation of the open source framework.

The components installed in Cephserver node are diamond and salt-minion. Diamond collects monitoring data, it supports very many data types and metrics, and each type of data is a collector that collects critical resource usage and performance data in addition to the status information of Ceph itself, including CPU, memory, network, I/O load and disk metrics. Collector uses the local command line to collect data and then reports it to graphite.

Graphite is not only an enterprise-class monitoring tool, it can also be plotted in real time. Carbon-cache is the back-end process of the highly extensible event-driven I/O architecture implemented by Python, which effectively communicates with a large number of clients and handles a large amount of traffic at a lower cost.

Whisper, like RRDtool, provides a database development library to the application to manipulate and retrieve file data stored in special formats (time data point data), whisper the most basic operation is to create a new whisper file, update the new data point into a file, And get the retrieved data points

Graphite_web is a user interface that is used to generate images that users can access directly through URLs.

Calamari uses Saltstack to let Calamari server and CEPH server node communicate. Saltstack is an open source, automated operations management tool that is similar to the chef and puppet features. Salt-master sends instructions to the specified salt-minion to complete the management of Cpeh cluster; salt-minion in the Ceph server When node installs, it synchronizes and installs a ceph.py file from master that contains the Ceph API, which invokes Librados or the command line to eventually communicate with Ceph cluster.

Calamari_rest provides calamari REST API, detailed interface please refer to the official documentation. Ceph's REST API is a low-level interface in which each URL directly maps to the equivalent Ceph Cli;calamari REST API provides a higher interface, and API users can get used to using the Get/post/patch method to manipulate objects, Without having to know the underlying CEPH commands, the main difference is that Ceph's Rest API's users need to know more about ceph itself, and Calamari's Rest API is closer to the Ceph resource description, so it's more appropriate to call the upper-level application.

Cthulhu is understood to be the service layer of Calamari server, which provides an interface to the API for lowering the salt-master.

Calamari_clients is a set of user interfaces, Calamari server will first create a Opt/calamari/webapp directory during installation and Webapp/calamari manager.py (Django Configuration) file, Calamari_web all the content into the Opt/calamari/webapp below to provide UI access to the page.

Calamari-web Package The following files provide all web-related configurations, both Calamari_rest and calamari_clients.

The framework uses a large number of open source software, but it is worth learning from an extended perspective, where saltstack implements the communication link between the management node and the server node, and supports multi-node management, so that there is no need to consider the communication problem between the management node and the server. On the server side only need to implement specific business logic, that is, the implementation of specific management tasks. At the same time Saltstack is developed using Python, so as to facilitate the rapid development of the system, very convenient for managers in the field to debug, positioning problems. Ceph itself also provides a Python API interface that enables cluster control directly through the Ceph API. The use of Saltstack allows the cluster to reach a certain scale. The master side of the Saltstack is actually the control interface of the management side, and saltstack acts as the agent side of the server. Send heartbeat messages through saltstack in calamari, check server information, cluster information, control command distribution. It can be said that understanding the basic pattern of saltstack can understand the development and expansion of calamari.

Another set of very important open source software in the framework is Diamond+graphite, in which Diamond completes the collection of server-side information, and graphite implements the provision of chart information. Diamond currently provides the vast majority of open source system Information collection, provides the server basic information collection (CPU, memory, disk and so on information), also uses the Python implementation, very easy to expand and Debug. Ceph's information collection is now available in diamond. Graphite mainly provides time-series data for the front desk, simplifying the rewriting of specific business logic.

To learn and understand calamari, you must understand some basic components and master the roles and purposes of these components. Here's how to extend calamari from a code level.

1 Extension of the Calamari

On the basis of calamari new function development, mainly divided into the following modules, this part includes Rest-api part, Cthulhu, salt Client extension. The basic steps for extending a new feature are as follows:

>> extend the URL module, determine the corresponding response interface parameters, corresponding to the response interface in the Viewset.

>> completes the implementation of some interfaces in Viewset, this part mainly involves the interaction with Cthulhu, how to obtain the data information, and in some cases, it is necessary to get the serialized operation of the object in serializer.

>> completes the extension of the corresponding type in the background rpc.py, which is mainly for part of the post operation.

>> complete the cluster_monitor.py extension, the corresponding requestfactory must be provided to support operations such as CREATE, update, delete, and so on, for some functions that provide the operation. In cluster_monitor.py, you need to add the corresponding requestfactory in your code.

>> completion of the corresponding Requestfactory class, which is mainly to complete the command operation of the package. and build the corresponding request operation.

>> salt-minion extension, this part is mainly for the extension of ceph.py files, of course, can also provide new xxx.py files.

Next, take the control and operation of PG as an example to illustrate.

1.1URL Module Expansion

Currently Calmamari is in REST-API form and is supported by the Django Rest-framework Framework, which is part of the REST-API code directory. Django uses the implementation of URL and code separation, so URLs can be extended separately.

Add the following URL for PG in rest-api/calamari-rest/urls/v2.py:

URL (r ' ^cluster/(? P [a-za-z0-9-]+]/pool/(? P \d+)/pg$ ', Calamari_rest.views.v2.PgViewSet.as_view ({' Get ': ' List '}), Name= ' cluster-pool-pg-list '),

URL (r ' ^cluster/(? P [a-za-z0-9-]+]/pool/(? P \d+)/pg/(? P [0-9a-fa-f]+\.[ 0-9a-fa-f]+)/command/(? P [a-za-z_]+) $ ',

Calamari_rest.views.v2.PgViewSet.as_view ({' Post ': ' Apply '}),

Name= ' Cluster-pool-pg-control '),

The above defines two URLs, respectively:

Api/v2/cluster/xxxx/pool/x/pg

Api/v2/cluster/xxxx/pool/x/pg/xx/command/xxx

The above two URLs specify the interface in Pgviewset, and the Get method of the URL corresponds to the list interface. The post interface corresponds to the Apply interface. These are the two interfaces that must be implemented in Pgviewset.

Expansion of the 1.2ViewSet

After extending the URL, the next step is to extend the corresponding response interface, which is mainly implemented for the interface class specified in the URL. In the previous PG specified two different interfaces, namely get and manipulate the command, the corresponding code path is/rest-api/calamari-rest/view/v2.py, the specific code is as follows:

Class Pgviewset (Rpcviewset):

serializer_class= Pgserializer

Deflist (self, request, Fsid, pool_id):

poolname = Self.client.get (Fsid, POOL, int (pool_id)) [' Pool_name ']

Pg_summary = Self.client.get_sync_object (Fsid, PGSUMMARY.STR)

Pg_pools = pg_summary[' Pg_pools ' [' By_pool '][int (pool_id)]

FORPG in Pg_pools:

pg[' pool ' = poolname

Return Response (Pgserializer (Pg_pools, many=true). Data)

Defapply (self, request, Fsid, pool_id, pg_id, command):

Return Response (self.client.apply (Fsid, PG, pg_id, command), status=202)

From the implementation of the above, the code implements two interfaces, namely the list and the Apply interface, which corresponds to the previous get, post operation. All two of these actions interact with the background Cthulhu. Get the parameters and submit the request, respectively. There are some differences in what is returned.

The serialization is also set in the list interface, which is Pgserializer, which is implemented in rest-api/calamari-rest/serializer/v2.py.

1.2.1 Serialization operations

The serialization of data is usually done in Rest-api, which is not necessarily done, and is usually necessary in the operation that needs to be changed. The following is the serialization of the PG operation:

Class Pgserializer (serializers. Serializer):

Classmeta:

Fields = (' IDs ', ' pool ', ' state ', ' up ', ' acting ', ' up_primary ', ' acting_primary ')

ID =serializers. Charfield (source= ' Pgid ')

Pool =serializers. Charfield (help_text= ' pool name ')

State =serializers. Charfield (source= ' state ', help_text= ' pg State ')

Up =serializers. Field (help_text= ' pg up set ')

Acting =serializers. Field (help_text= ' PG Acting set ')

Up_primary = serializers. Integerfield (help_text= ' pg up primary ')

Acting_primary =serializers. Integerfield (help_text= ' PG Acting primary ')

This is not a necessary part. Some modules may not exist for this part of the operation. In the previous three steps basically implemented the REST-API part of the extension, where the main viewset extension. The viewset actually implements the interaction method between Cthulhu and Rest-api.

In the viewset extension, RPC is actually used to interact with the background, so the implementation part of the Cthulhu is mainly to handle the corresponding RPC request.

1.3RPC expansion

All requested operations are implemented in rpc.py, but the new extended operation is also required to support extensions, and in the case of PG continue to explain:

Defapply (self, fs_id, object_type, object_id, command):

"""

Apply commands that does not modify an object in a cluster.

"""

Cluster = Self._fs_resolve (fs_id)

Ifobject_type = = OSD:

# Run A resolve to throw exception if it ' s unknown

Self._osd_resolve (cluster, object_id)

Return cluster.request_apply (OSD, object_id, command)

Elifobject_type = = PG:

Return cluster.request_apply (pg,object_id, command)

Else

Raise Notimplementederror (Object_type)

The list of PG is obtained by Pgsummary. This part already exists in the previous implementation, and the previous code is implemented as follows:

Defget_sync_object (self, fs_id, object_type, Path=none):

"""

GetOne of the objects that Clustermonitor keeps a copy of the Mon, such

Asthe cluster maps.

:p Aram Fs_id:the Fsid of a cluster

:p Aram Object_type:string, one of the Sync_object_types

:p Aram Path:list, optional, a path within the object to return insteadof the whole thing

: Return:the requested data, or None if it is not found (including ifany element of ' path '

was not found)

"""

Ifpath:

Obj =self._fs_resolve (fs_id). Get_sync_object (Sync_object_str_type[object_type])

Try

For part in Path:

If Isinstance (obj, dict):

obj = Obj[part]

Else

obj = GetAttr (obj, part)

Except (Attributeerror, Keyerror) as E:

Log.exception ("Exception%s traversing%s:obj=%s"% (e, path,obj))

Raise NotFound (object_type, Path)

return obj

Else

Returnself._fs_resolve (fs_id). Get_sync_object_data (Sync_object_str_type[object_type])

1.4cluster_monitor.py extension

The operation of the request will be controlled by the cluster, which can be implemented by Cluster_monitor, which is illustrated by the example of PG.

Def__init__ (self, fsid, cluster_name, Notifier, Persister, Servers, eventer,requests):

Super (Clustermonitor, self). __init__ ()

Self.fsid = Fsid

Self.name = Cluster_Name

Self.update_time = Datetime.datetime.utcnow (). Replace (TZINFO=UTC)

Self._notifier = notifier

Self._persister= Persister

Self._servers = Servers

Self._eventer = eventer

Self._requests = Requests

#Which Mon We is currently using for running requests,

#identified by Minion ID

Self._favorite_mon = None

Self._last_heartbeat = {}

Self._complete = Gevent.event.Event ()

Self.done = Gevent.event.Event ()

Self._sync_objects = SyncObjects (self.name)

Self._request_factories = {

Crush_map:crushrequestfactory,

Crush_node:crushnoderequestfactory,

Osd:osdrequestfactory,

Pool:poolrequestfactory,

Cachetier:cachetierrequestfactory,

Pg:pgrequestfactory,

Erasure_profile:erasureprofilerequestfactory,

Async_command:asynccomrequestfactory

}

Self._plugin_monitor = Pluginmonitor (servers)

Self._ready = Gevent.event.Event ()

This part mainly binds the corresponding request to the corresponding request factory class in order to produce the appropriate request.

1.5 Factory class Writing

The factory class is mainly for different needs, implementation of specific interface classes, different objects have different request classes, with PG as an example:

From Cthulhu.manager.request_factory importrequestfactory

From Cthulhu.manager.user_request importradosrequest

From Calamari_common.types Importpg_implemented_commands, pgsummary

Class Pgrequestfactory (Requestfactory):

def scrub (self,pg_id):

Return Radosrequest (

"Initiating scrub On{cluster_name}-pg{id}". Format (cluster_name=self._cluster_monitor.name,id=pg_id),

Self._cluster_monitor.fsid,

Self._cluster_monitor.name,

[(' PG scrub ', {' Pgid ': pg_id})])

Defdeep_scrub (self, pg_id):

Return Radosrequest (

"Initiating Deep-scrub on{cluster_name}-osd. {ID} '. Format (cluster_name=self._cluster_monitor.name,id=pg_id),

Self._cluster_monitor.fsid,

Self._cluster_monitor.name,

[(' pg Deep-scrub ', {' Pgid ': pg_id})])

Defrepair (self, pg_id):

Return Radosrequest (

"Initiating repair ON{CLUSTER_NAME}-OSD. {ID} '. Format (cluster_name=self._cluster_monitor.name,id=pg_id),

Self._cluster_monitor.fsid,

Self._cluster_monitor.name,

[(' PG repair ', {' Pgid ': pg_id})])

Defget_valid_commands (self, pg_id):

Ret_val = {}

File ('/tmp/pgsummary.txt ', ' A + '). Write (pgsummary.str + ' \ n ')

Pg_summary = Self._cluster_monitor.get_sync_object (pgsummary)

Pg_pools = pg_summary[' Pg_pools ' [' By_pool ']

pool_id = Int (Pg_id.split ('. ') [0])

Pool= pg_pools[pool_id]

FORPG in Pool:

If pg[' pgid '] = = pg_id:

RET_VAL[PG_ID] = {' Valid_commands ': pg_implemented_commands}

Else

RET_VAL[PG_ID] = {' Valid_commands ': []}

Return Ret_val

This class implements the implementation of three different commands, the command is mainly for the corresponding encapsulation, this part of the keyword needs to be selected according to the parameters in the Ceph source code, so the code needs to refer to the corresponding command in the Ceph source code name of the JSON parameter.

Extension of the 1.6salt-minion

This part is the Salt extension module, mainly used to obtain the corresponding data information, execute the corresponding operation command and so on. Executes the corresponding action command in Cthulhu with a salt. The ceph.py has an interface such as Rados.commands, which can be used to perform ceph commands. The commands encapsulated in the factory class will eventually be executed through the interface.

Summarize
Overall, calamari code structure is clear, and the open source framework is also worth learning, in the subsequent distributed management system can also refer to the Saltstack+diamond+graphite architecture, the former implementation of the control logic, The following two data acquisition and storage display.

http://www.bkjia.com/PHPjc/1092984.html www.bkjia.com true http://www.bkjia.com/PHPjc/1092984.html techarticle Ceph Management Platform Calamari extension development Nearly half a year did not write a log, perhaps they are more and more lazy it. But sometimes write things can let themselves precipitate, or come back to record a bit ...

