Influxdb Technology of Time-series database for Internet-level monitoring system

The time series database, or the timing database, is a whole new field, and the most important feature is that each piece of data is provided with a date column.

What business scenarios can be used in the time series database, the answer is: the monitoring system.

Baidu, Internet Monitoring System, we will find millet, hungry and other internet giants are using time series database to achieve enterprise-level Internet monitoring system.

A lot of people will say, with Zabbix not to take care of, in fact, not so, simple host resource monitoring, network monitoring, small-scale deployment environment, Zabbix can fix.

If in IDC thousands of server environment, distributed Application architecture, a variety of middleware, in this case we want to monitor the host resources of thousands of services, network, according to different latitude monitoring service performance, TPS, monitoring all kinds of middleware, program monitoring buried point. Zabbix will not be able

Meet the needs. At this point, we have to build up their own monitoring system. Speaking of which, what is behind every monitoring chart?

1. Timeline

2. Data values (different metric latitude)

For example, CPU usage over time

At this time, all kinds of Google, Baidu, you will certainly search for Influxdb, Opentsdb and other time series database.

Influxdb we studied for a long time, ready to use 2 articles, recommended to you, in this article, we share the key features of INFLUXDB, query syntax and usage scenarios.

First, Influxdb Key features

1. Support SQL-like query syntax

2. Provides direct access to the HTTP API

3. Store more than 1 billion levels of time-series data

4. Flexible data retention policies that can be defined to the database level (keeping only the hottest data)

5. Built-in management interface and CMD

6. Fly General Speed Aggregation query

7. Aggregate queries at different time periods

8. Built-in continuous query function, timing calculation of the specified time period of data, inserted into the specified table, can be understood as the timing of data collection

9. Horizontal expansion, support cluster mode

Second, Influxdb version and. NET support

1. According to our experience, the V0.10 version is very stable, V0.9.6 we used, there is a memory leak problem

2. There are very many. Net Libraby on GitHub that allow us to write and read data

Third, data written to write (Points)

Http API:
Curl-i-xpost ' http://localhost:8086/write?db=mydb '--data-binary ' cpu_load,host=server01,region=us-west value=0.64 1434055562000000000 '
Db:mydb, the database to write to
Measurement:cpu_load, table
Tag Keys:host region tag Value:server01 us-west
Tag tags can be understood as dimensions, optional parameters, to identify different data sources, and to make queries simpler and more efficient based on tags
Tags is indexed so queries on tag keys or tag values is more performant than queries in fields.
Key Field:value Value field:0.64
timestamp:1434055562000000000 Optional parameters, UTC

Support for bulk Write
Supports writing different data to the same timestamp
INFLUXDB supports flexible storage structure changes, can add measure, tags, fields, but each tag, field data type must be fixed.

Iv. querying query

Http API:
Curl-g ' http://localhost:8086/query?pretty=true '--data-urlencode "Db=mydb"--data-urlencode "Q=select value from Cpu_ Load_short WHERE region= ' us-west '
Return JSON format data
Supports simultaneous multiple query SQL
The query returns up to 10,000 points of data, and if the threshold is exceeded, you can set a chunk_size
Measurement, Tag, Field, data, and other case sensitive , SQL keywords are case-insensitive

Supports arithmetic calculations:
SELECT (Water_level * 2) + 4 from H2o_feet
Support for filtering query tags, conditions must use single quotation marks
SELECT water_level from h2o_feet WHERE location = ' Santa_monica '
Tag value is empty, NOT null filtered
SELECT * from H2o_feet WHERE location!~/.*/
SELECT * from H2o_feet WHERE location =~/.*/
Time range filtering
SELECT * from H2o_feet WHERE time > Now ()-7d
Field Value Filter
SELECT * from h2o_feet WHERE location = ' Coyote_creek ' and Water_level > 8

Aggregation functions, selection functions, conversion functions

V. Continuous enquiry (continuous Queries)

Continuous query is influxdb Automatic, periodic run of queries, results are automatically stored

The purpose of a continuous query design is to sample data by day, by month , for example

CREATE continuous QUERY <cq_name> on <database_name> [resample [every <interval>] [for <interval ] "BEGIN SELECT <function> (<stuff>) [,<function> (<stuff>)] into <different_ Measurement> from <current_measurement> [WHERE <stuff>] GROUP by Time (<interval>) [,<stuff> ] END

Vi. Monitoring and application scenarios

Through the introduction of the above sections, the basic grammar of influxdb can be mastered. What is the effect of:

1. Real-time acquisition of monitoring data, written by Influxdb

2. Aggregate query monitoring data at different latitude for monitoring display

3. Continuous query, time collection of data for a specified period, for greater time range monitoring data display

Summing up, the scene combined with practice, through the actual monitoring system application, and everyone to share the use of influxdb and skills. Our own monitoring system is built up a little bit by this routine.

At present, our monitoring platform, 2,500 monitoring items, 500 servers real-time monitoring, daily processing on T data, hundreds of monitoring charts, influxdb meet our daily ultra-large-scale monitoring needs.

At the same time, influxdb in the Big data show area, also has the good performance, the Druid integration also is great.

Zhou

2017/7/11

Influxdb Technology of Time-series database for Internet-level monitoring system