Operation and maintenance architecture service monitoring Open-falcon

This is a creation in Article, where the information may have evolved or changed.

absrtact: monitoring System is the entire operation and maintenance link, and even the entire product life cycle of the most important one, in advance timely early warning found fault, and afterwards provide accurate data for tracking location problems. Monitoring system as a mature operation and maintenance products, the industry has a lot of open source implementation to choose from. When the company just started, small business, operations and maintenance team has just been established early, choose an open source monitoring system, is a time-saving, the most efficient solution.

First, Introduction

Monitoring system is the entire operation and maintenance links, and even the entire product life cycle of the most important one, in advance timely early warning found fault, and afterwards provide accurate data for tracking location problems. Monitoring system as a mature operation and maintenance products, the industry has a lot of open source implementation to choose from. When the company just started, small business, operations and maintenance team has just been established early, choose an open source monitoring system, is a time-saving, the most efficient solution. Then, as the business scale continues to grow rapidly, the object of monitoring is becoming more and more complex, and the object of the monitoring system is expanded from the first few SRE to more devs,sre. At this time, the capacity of the monitoring system and the user's "use efficiency" become the most prominent problem.

The monitoring system industry has many outstanding open source monitoring system. We have been using Zabbix in the early days, but with the rapid development of our business and the specific needs of internet companies, the existing open source monitoring system has been unable to support performance, scalability, and user efficiency.

Therefore, we in the past year, from some of the internet company's needs, from the use of SRE, SA, devs experience and feedback, combined with some of the industry's major internet companies to do monitoring, with some of the monitoring of the start, design and development of millet monitoring system: Open-falcon.

Second, the characteristics

1. Powerful and flexible data acquisition: Auto discovery, Support falcon-agent, SNMP, support user active push, user custom plugin support, OPENTSDB data model like (timestamp, endpoint, metric, Key-value tags)

2, horizontal expansion ability: Support each period of billions of data collection, alarm determination, historical data storage and query

3. Efficient alarm Policy Management: Efficient portal, support policy templates, template inheritance and overlay, multiple alarm modes, support callback calls

4. User-friendly Alarm settings: Maximum number of alarms, alarm level, alarm recovery notification, alarm pause, different thresholds for different periods, support maintenance cycle

5, high-efficiency graph components: single-stand support 2 million metric of the escalation, archiving, storage (1-minute period)

6. Efficient historical Data Query component: Using RRDtool data archiving strategy, second level return hundreds of metric year of historical data

7, Dashboard: Multi-dimensional data display, user-defined screen

8, High availability: The whole system without core single point, Isianvi, easy to deploy, can be horizontally expanded

9. Development language: The backend of the entire system, all Golang written, the portal and the dashboard are written using Python.

Third, the structure

Each server is installed Falcon-agent,falcon-agent is a Golang developed daemon program for self-discovery of various data and indicators of the acquisition of single machine, these indicators include not limited to the following aspects, a total of more than 200 indicators.

CPU-related

Disk-related

Io

Load

Memory-related

Network-related

Port survival, process survival

NTP offset (plug-in)

A process resource consumption (plug-in)

Netstat, SS and other related statistical items collection

Machine Core Configuration parameters

As long as the installation of Falcon-agent machine, will automatically start to collect the indicators, the initiative to report, do not need the user in the server to do any configuration (this and Zabbix is very different), the benefit is that the user maintenance is convenient, high coverage. Of course, this will also cause a large server side pressure, but Open-falcon's service-side components stand-alone performance is high enough, at the same time can be horizontally extended, so automatically collect enough data, but it is a good thing, for SRE and Dev, the post-mortem problem is no longer a problem.

In addition, Falcon-agent provides a proxy-gateway, the user can easily through the HTTP interface, push data to the native Gateway,gateway will help to efficiently forward to the server side.

Iv. Data Model

The Data model is powerful and flexible, and is critical to the "efficiency" of monitoring system users. For example, Zabbix as an example, the reported data is hostname (or IP), metric, then the user to add alarm policy, management alarm policy, only in these two dimensions. One of the most common scenarios:

Hosta disk space, less than 5%, on the alarm. On the general server, there will be two major partitions, the root partition and the home partition, in the Zabbix inside, you have to add two rules; if it is a hadoop machine, there will generally be more than 10 pieces of data disk, but also to add more than 10 rules, so it will be painful, not happy, is not conducive to automation (of course Zabbix can be configured with some automatic discovery strategy to solve this, but more troublesome).

V. Data collection

Transfer, receive the data sent by the client, do some data normalization, check, forward to a number of back-end systems to deal with. When forwarding to each backend business system, transfer will perform data sharding according to the consistent hash algorithm to achieve the level expansion of the backend business system.

Transfer provides JSONRPC interface and Telnet interface two ways, transfer itself is stateless, hanging off one or more will not have any effect, while transfer performance is high, can forward more than 5 million data per minute.

Transfer currently supports the business backend, there are three kinds, judge, graph, Opentsdb. Judge is a high-performance alarm determination component we developed, graph is a high-performance data storage, archiving, query component, and Opentsdb is an open source time series data storage service. Can be opened via the transfer configuration file.

There are three types of transfer data sources, typically:

1, the basic monitoring data collected by falcon-agent

2. Falcon-agent executes the data returned by the user-defined plug-in

3, the client library: Online business systems, are embedded using a unified Perfcounter.jar, for each RPC interface in the business system QPS, latency will actively collect and report

Note: The above three kinds of data, will be sent to the local proxy-gateway, and then forwarded to transfer by the gateway.

Basic monitoring means that as long as a machine (or container) can add monitoring, such as CPU mem NET IO disk, these monitoring methods fixed, do not need to configure, and do not require the user to provide additional parameters specified, as long as the agent can be directly collected to run the report up; non-basic monitoring is the opposite, such as port monitoring, you do not give me the port number, or I report all 65,535 ports of the monitoring state you can not use, such monitoring requires user configuration before beginning to collect escalated monitoring (including similar to port monitoring configuration trigger class monitoring, and similar to the MySQL plug-in script class monitoring), Generally it is not the scope of basic monitoring.

Vi.. Alarm

Alarm determination is done by the judge component. The user configures the associated alarm policy in Web portal, which is stored in MySQL. Heartbeat server periodically loads the contents of MySQL. Judge also maintains regular communication with heartbeat server to obtain relevant alarm policies.

Heartbeat Sever is more than simply loading MySQL content, based on template inheritance, template item overrides, alarm action overrides, templates, and HostGroup bindings, to calculate the final alarm policy associated to each endpoint, provided to the judge component to use.

Transfer forwarding to the judge of each data, will trigger the decision of the relevant policy, to determine whether to meet the alarm conditions, if the conditions are met, it will be sent to Alarm,alarm e-mail, SMS, rice Chat and other forms of notification to the relevant users, You can also perform user-preconfigured callback addresses.

Users can be very flexible to configure the alarm decision strategy, such as the continuous n times to meet the conditions, the maximum number of consecutive n to meet the conditions, different time periods of different thresholds, if in the maintenance period is ignored and so on.

In addition, it also supports the determination and alarm of the sudden descending class.

Vii. API

Here, the data has been successfully stored in graph. How to read it quickly, read the past 1 hours, the past 1 days, the past January, the past year, have to return within 1 seconds.

These are implemented by graph and API components, and transfer forwards the data to the graph component, and graph receives the data and stores it as a RRDtool data archive, while providing a query RPC interface.

API for the end-user, after receiving the query request, will go to multiple graph inside, query different metric data, summarized and returned to the user.

Eight, panel

Nine, storage

For the monitoring system, historical data storage and efficient query, is always a difficult problem!

Large data Volume: At present our monitoring system, each cycle, about 20 million times the data escalation (escalation cycle of 1 minutes and 5 minutes two, each accounted for 50%), 24 hours a day, never have a business low peak, whether it is day and night, each cycle, there will always be so much data to update.

Write operations: general business systems, usually read and write less, you can easily use a variety of cache technology, in addition to various types of databases, the processing efficiency of the query operation is much higher than the write operation. While the monitoring system is the opposite, the write operation is much higher than reading. Update operations of tens of millions of times per cycle are not possible for common databases (MySQL, PostgreSQL, MongoDB).

Efficient check: we say that the monitoring system read less, is to say that the relative write. The monitoring system itself is very demanding for reading, and users often have data on hundreds of meitric, in the past day, week, month, and year. How to return to the user in 1 seconds and draw, this is a very small challenge.

Open-falcon in this piece, put a greater energy. We divide the data into two categories, one for drawing, and the other for data mining by users.

For the data of the drawing, it is the key to query fast, and the information can not be lost. For the user to query 100 metric, in the past year of data, the amount of data itself is there, it is difficult to return in 1 seconds, in addition, even if returned, the front end can not render so much data, but also to sample, causing a lot of unnecessary consumption and waste. We refer to the concept of RRDtool, which is automatically sampled and archived every time the data is deposited. Our archive strategy is as follows, with historical data kept for 5 years. At the same time, in order to not lose information, data archiving, according to the average sampling, the maximum sampling, the minimum value of three copies.

Original link