Some summary of application level monitoring

1. The need for diversification, popular is to put the hardware and software indicators together to compare.

Sometimes we focus on the operational state of the application, not just the metrics of the application, but sometimes the key metrics of the application's operating environment (containers, virtual machines, hardware) at the same time. However, the application layer is heterogeneous with its running environment, so the acquisition tool is not the same. For example, we use OPENTSDB to monitor one of my web programs, and use ganglia to monitor its server, in fact, many times we pay more attention to the performance of hardware and software indicators at the same time, cut to cut too not intuitive. This is the problem, two different tools are displayed on different pages, can the Web program's indicators and the server's CPU, memory, file handles and other indicators together to see it?

In fact, it is easy to think of separate collection, unified storage can be. Now the main code calls the measurement of the SDK (Code embedded monitoring package), JMX Acquisition (large open source distributed system is basically provided), Text protocol acquisition (REDIS, memcached monitoring port, ZK stat, etc.), as well as hardware metrics acquisition. These monitoring data unify data structures and are stored in ES or hbase, which makes it easy to render the time curves of various heterogeneous system indicators on the UI. In fact, this requirement is fundamentally rooted in the relationship of calls between systems and the dependencies that the application runs in the environment.

In a unified storage structure, it is easy to define the invocation relationships and dependencies of various applications or systems, so that it is easy to find the relevant indicators when querying, and it is easier to identify problems. So the conclusion is that the diversified collection and centralized storage is the necessity of the demand.

2. For monitoring, in addition to the metric dimension, Time dimension, statistical model dimension, there should also be a call dependent dimension

What do you say?

Previously, we will focus on the current server CPU, MEM, disk indicator values, we use a number of system commands to view, the resulting data is one-dimensional, is a value of the indicator, can change the indicator.

Further, we have some tools, zenoss this, can be bandwidth consumption, CPU consumption in the time dimension projection, then came to a curve, this is two-dimensional: the indicator changes, time changes.

More advanced, for a method call we can look at it from various angles: call time-consuming statistics, call frequency, call count, call result statistics, and then package measures, any method is a universal measurement model. This is the indicator, the time, the model. The model represents a set of pervasive statistical methods. such as the now well-known metrics model: Codahale Implementation of the Java metrics.

What is the finer dimension of observation? For the indicators, the call between heterogeneous systems is often related, so the monitoring of a set of heterogeneous system indicators is necessary. But here to note, we say in the "1" in the way only to compare the statistical indicators of the correlation, but this size is too coarse, really do the correlation indicator observation, need to focus on the level of call to observe, in a call, a called B, such a relationship can not be accurately reflected in the time dimension, Must be one by one to match the line. So in essence, this is a new observational dimension. This is the trace system.

That is, the various monitored systems have a call ID after each invocation, the only one that records the call. So this call is associated with multiple metrics for both systems. In this model, in addition to defining the invocation relationship, statistics are something that can be done naturally. But in terms of implementation, it is necessary to integrate him within the framework of communication, because each invocation needs to be precisely marked.

In fact, this idea is similar to our previous high school biology textbook isotope labeling method, call ID is an isotope, each invocation process is a chemical change process.

Some summary of application level monitoring