DevOps: Monitor The System to Make Informed Business Decisions

DevOps monitoring is the process of collecting, analyzing, and using information to track applications and infrastructure to guide business decisions.Monitoring is a key function because it can give you a deep understanding of your system and work. Properly implemented monitoring can also provide you with rapid feedback so that you can quickly identify and resolve problems early in the software development life cycle.

Monitoring can also help you communicate information about your system to people in other areas of the software development and delivery pipeline and other business departments. Knowledge acquired downstream of operations may be communicated to upstream teams such as development and product management teams. For example, relevant personnel can gain some knowledge by operating a highly scalable application that uses the NoSQL database as a data storage space, and this knowledge may be valuable information for developers building similar applications.

Through this knowledge transfer, the team can quickly acquire knowledge, whether it comes from production problems, deployment errors, or from customer usage patterns. Later, you can share this knowledge throughout the organization to help people improve their level and improve the system.

How to implement monitoring
The following points are the key to effective monitoring:

Collect data from key aspects of the entire value chain, including application performance and infrastructure.
Use the collected data to make business decisions.
Data collection
To collect data more effectively, you should implement a monitoring solution (as a local service or hosted service) to understand development work, testing, quality inspection, and IT operations. Be sure to choose the appropriate indicators for the function and business. For a discussion of metrics and metrics in DevOps, see the white paper (PDF) from the 2015 DevOps Enterprise Forum.

Use data to make business decisions
When you transform and present the collected data, different audiences can access the data to make decisions. For example, you may want to share operational data upstream. You can also properly integrate this data into reports and briefings, and use this data in meetings to make informed business decisions. "Appropriate" here means "relevant, timely, accurate, and easy to understand."

In such meetings, be sure to provide relevant information to help people who may not be familiar with the data understand the relationship between the data and the discussion topic, and how the data can help people make informed decisions. For example, you may want to know how to answer the following questions:

Are these values relatively high or low?
Are they expected?
Do you anticipate changes?
How are these data different from historical reports?
Has your technology or infrastructure affected the relevant numbers in interesting or unobvious ways?
Common misunderstandings in monitoring
The following is a list of common misunderstandings when monitoring systems:

Reactivity monitoring. For example, relevant personnel only receive reminder notifications when the system fails, without using monitoring data in advance to proactively send reminder notifications when the system is approaching a critical threshold.

The monitoring range is too small. For example, the relevant personnel only monitor one or two aspects, but not the entire software development and delivery pipeline. This misunderstanding focuses on indicators and only on the measured aspects, which may not be the best aspect to monitor.

Focus on local optimization. For example, relevant personnel focus on reducing the response time of a service’s storage requirements without evaluating whether a broader infrastructure can also benefit from the same improvement measures.

Monitor everything. By collecting all the data on the system and reporting all the information on the system, you may face the risk of excessive system reminders or excessive data complexity. Adopting a sensible monitoring method helps to focus on key aspects.

Improve monitoring methods
In order to improve the monitoring effect, we recommend that you focus on two main aspects:

Collect data from key aspects of the entire value chain.

By analyzing the collected data and conducting a gap analysis, you can ensure that the appropriate data is collected for the organization.

Use the collected data to make business decisions.

The data you collect should bring value to the entire organization, and the metrics you choose must be meaningful to your organization. Meaningful data can be used by many teams (from DevOps teams to finance teams).

It is also important to find the right media to display monitoring information. You need to choose different representations for different uses of information. For DevOps teams, a real-time information center may be the most useful; for metrics measured over a long period of time, regularly generated business reports may be useful.

Most importantly, it is important to ensure that data is available, shared, and helps guide decision-making. If sharing a spreadsheet is the best way to help you get started early, use it and upgrade to an advanced information center later. Don't let perfectionism hinder your pace, just be good enough.

Method of measuring monitoring effect
Effective monitoring helps improve performance during software development and delivery. However, measuring the effectiveness of monitoring may be difficult to achieve in the system. Although you may be able to automatically measure how much data is collected from the system and the type of data, it is difficult to know whether or where the system is using the data.

To help you measure the effectiveness of monitoring in your organization, consider the extent to which people agree or disagree with the following statements:

People use data from application performance monitoring tools to make business decisions.
People use data from infrastructure monitoring tools to make business decisions.