Common time Series Database Overview __ Database

For the storage of time series, a special time series database is used instead of MySQL or MONGO (but Zabbix is the MySQL used, so it encounters a bottleneck on Io). Now there are a lot of databases of time series, such as Graphite, OPENTSDB and the new influxdb. These three databases have been studied recently, and the Institute is now recorded. Graphite

Graphite is a veteran time series storage solution, graphite consists of three parts, respectively, carbon, whisper and graphite web, stolen map:

Carbon: is actually a series of daemons that use the twisted event-driven network engine to listen for time series data. The twisted framework allows the carbon daemon to handle a large number of clients and traffic at a low cost.
Whisper: is a database for storing time series data, which can then be obtained and manipulated by the application using Create,update and fetch operations.
Graphite Web: Use a web of Django development to provide some common aggregate functions that allow you to display graphics in a friendly interface. Re-theft Map:

Whisper supports RRD, it's easy to define retention, and define storage scheme, and you don't need to do it manually, graphite will automatically help you implement aggregation according to different scheme. So the size of each metric is fixed, so the data can theoretically be stored permanently. There are two main graphite cluster schemes, namely, the use of graphite relay or the use of third-party tools.

When using the relay, it is only necessary to configure which machines are to be relay in the configuration file so that when the data is written, the node being written will relay to the machine. At the time of reading, the list of hosts is configured in the settings of the graphite web, so that the data will be read from these hosts in turn in the Django Web.

Another is the need for Third-party relay tools, booking company Open source out of their use in C written Carbon-c-relay, as well as go to write Carbon-relay-ng. The basic idea is to use a consistent hash, different metric can be sent to different machines, to achieve the purpose of the cluster, according to booking said their graphite cluster reached the scale of hundreds of machines, and mertic also reached million levels.

The above two ways will encounter a common problem, is the cluster expansion of the problem, I do not know how booking is to solve this problem, but my side is now thinking of several ways, this in my previous article has elaborated: Graphite cluster expansion plan exploration, Here is not to go into the details.

Finally, in addition to the cluster problem, graphite also has a performance problem is poor reading performance, which depends on the way it is stored, in fact, read the Whisper file (although the Django layer did cache, but the caching function is relatively weak), Obtain the location of the data by means of seek, in order to remove the data. Opentsdb

Opentsdb is a relatively heavy time series solution, why do you say he is heavy. Because it's made up of this:

You can see that the storage that OPENTSDB relies on is the HBase cluster. TSD in which the responsibility is IO part, TSD is actually a backstage daemon, generally I will use a group of TSD to reach a TSD cluster (in fact, not as a cluster), no master/slave points, there is no shared state, and then used lb equipment to do load balancing, The official comparison recommendation is to use varnish.

While the back-end storage can also be used for other examples such as Hadoop, it is recommended that the hbase be used because OPENTSDB is the product that the HBase community hatches. So the problem is, HBase's operation will be a difficult task, this relies on the zookeeper set of the cluster pit is still a lot of, I looked at the official document there are thousands of pages. The optimization of this maintenance needs to have professional hbase experts to complete.

In addition, opentsdb do not graphite that automatic downsample, that is, do not RRD to store data, need to do a layer of manual work on the outside, and then write the data after the aggregation of HBase, alas, is really a time-consuming and laborious work.

In addition Opentsdb is very good, read and write performance is very high, and support tag, support TTL, support a variety of aggregate functions. Now a lot of monitoring of the metric storage is used OPENTSDB, well, yes, as long as the ability to play is a good choice. Influxdb

Influxdb is the latest in a time series database, the latest a year or two, but already has a very high popularity. Influxdb is written with Go, now v0.9 is under development, the most stable version of the previous open source is 0.88, but 0.8X is not a cluster solution, but in 0.9 will join in.

The 0.9 version of INFLUXDB will be a big change before, back-end storage has leveldb replaced with Boltdb, read and write API also has a great change, will support clustering, continuous query, support retention policy, Read and write performance is also Wah, can be said to be the perfect time series storage scheme, but because it is still very young, there may be many problems, as is now being developed 0.9, the release of a drag and drop, is due to some technical barriers have not fallen.

For influxdb I do not want to say anything more, and then intend to open a special topic, specifically to say this thing, because I see the domestic almost no detailed articles for INFLUXDB. Summary

It is simple to say that the current comparison of the three time series database, you can see the pros and cons, in our selection, according to their own situation to choose the most suitable for their own solutions, in fact, we are now in distress, seemingly influxdb is the most perfect solution, But v0.9 is not a long overdue. Opentsdb, although mature, but lack of experts in this area, graphite, poor performance, and the cluster is too dead. In distress ...