Distributed Nagios configuration experience

The distributed Nagios has been used for more than half a year. I have some experiences in writing this article to share with you. I am not an expert and can only write SOME SUPERFICIAL OPINIONS. What you see is wrong, or a better way. You are welcome to shoot bricks.

In distributed Nagios, many people are talking about it, and many people are using it or are planning to use it. I did not want to write a tutorial on this post. The configuration of distributed Nagios is very simple, and there is no need to write any tutorials. I just want to share some ideas, 2. Nagios's distributed architecture has many understandings and implementation methods. I just want to write the ones I'm interested in.

1. When do I need to use distributed Nagios? 

For various reasons, a Nagios instance cannot meet the monitoring requirements. Maybe the cluster you want to monitor is located in two data centers, which are far away from each other. Maybe you have too many machines to monitor, I think a Nagios instance cannot be supported. It may be another reason. In short, you must have two or more Nagios instances to meet your monitoring needs. At this time, if you choose Nagios, then we need to configure distributed Nagios.

2. What is distributed Nagios? 

Simply configure several Nagios instances without any other work. This configuration is not very good. If you have two or three instances, you can open three web pages, you can view the three Nagios instances and maintain a list. When the monitoring configuration changes, you can go to the three Nagios instances. However, once the monitoring machines are multiple, for example, ten or eight Nagios instances, when you need to locate an error, you need to open a lot of web pages to see it, it is not very good.
Therefore, my personal understanding of distributed Nagios mainly includes the following aspects:

1. Distributed Detection. This is the most basic. The distributed configuration mainly requires each invigilator to monitor the objects in a range managed by itself.
2. Centralized presentation. I think this is necessary. When there are multiple machines, a centralized display page is required to avoid opening three web pages for three Nagios servers, 10 Nagios web pages are required.
3. centralized control. When we make some monitoring changes to a single machine, such as switching alarms, submitting detection results, and resending alarms, it is best to do so in a unified manner, instead of running many places to find them.
4. Centralized configuration. Configuration changes should also be made together and then distributed to various distributed Nagios monitoring machines.
5. Distributed alarms. Alarm tasks need to be distributed to various distributed monitoring machines to avoid unexpected situations, such as sudden disconnection of the internal network and failure to attach the machine, or when the machine's check results cannot be returned, the system can issue a normal alarm.
6. The most important thing is to avoid distribution. If you can avoid configuring distributed Nagios, try to avoid it! Nagios is a very efficient software. I have been using it for more than half a year and I feel that it cannot be supported at a time. However, every result shows that it is not a problem of Nagios, but a problem of my own configuration. Regardless of the distributed implementation method, Nagios is a performance loss. What's more, why should we use two machines to solve the problem?

All in all, the purpose of our configuration is that many Nagios servers are running, but we need to feel that we only have one Nagios working in total.

3. How to Implement the distributed architecture mentioned in the previous question? 

There are many specific implementation methods. I will list what I know. You are welcome to add:

Method 1: Use Nagios passive detection. 

This is the easiest way. The distributed Nagios uses the OCSP option to submit the inspection results to a general monitoring machine through NSCA. The general monitoring machine also configures Nagios, however, it does not perform host or service detection. All detection results are obtained from distributed Nagios through NSCA. The implementation is simple, but the effect is the worst.

A. the passive detection result is through Nagios. the CMD file is submitted, while the data transmission in the pipeline is very inefficient, and the default cmd cache is only 4096 rows. Of course, this value can be configured very large, however, this will inevitably affect the speed at which the check results will be recycled. If the setting is too small, there will be a lot of check results that cannot be recycled. In addition, the commands sent back from all web pages are also called Nagios. cmd. We will find that some operations will take effect for a long time, or even not.

B. Centralized control needs to be implemented separately. If the alarm function is implemented on the distributed Nagios, it must be implemented separately. However, if you want to trigger an alarm on the centralized machine, it is not impossible, but the problem above will be very prominent. In case of overflow of recovery results, it will be discarded, the alarm will never go out.

C. Centralized configuration also needs to be implemented.

Method 2: Use Ndo to unify the database and then display it in a centralized manner. 

A better solution is centreon, which is written in PHP and Perl and used in Ndo. Specifically, centreon configures all the monitored nodes and stores them in its database. Then, it exports these configurations to the Nagios configuration file and starts Nagios, use Ndo to import the check results to the database, and then read centreon to display the results.

This is a perfect solution. Centralized Control, centralized display, centralized configuration, distributed detection, and alarm are all implemented, and the RRD drawing and report functions are also provided. However, note the following:

A. configuration is troublesome. I can only say that the configuration is very troublesome and there are many dependent items, such as PHP, MySQL, Ndo, and many Perl Lib. Some scripts may not run after they are installed, A large number of errors are thrown. You need to troubleshoot them one by one and rewrite the script when necessary. If you are familiar with Nagios and have some knowledge about Perl, PHP, rrdtool, and MySQL, we recommend that you use centreon. If not, consider more, in many cases, they are stuck in the same place when they are configured.

B. Ndo warehouse receiving problems. Nagios is a very efficient software. If you have read its source code, I believe you will feel this way. He avoided any possible bottlenecks from the beginning to the end, so he deliberately bypassed the database and made a suitable balance between hard disk read/write and waiting for the check results, it tries its best to make its performance fully dependent on the processing capability of the processor, rather than low-speed hard disks, networks, and unpredictable plug-in. This makes me feel that both Ndo and passive detection will have a great impact on its performance. I once encountered this kind of thing. My Nagios was dragged down by a low-speed database, and the last_check time was evenly distributed in an hour ......

Method 3: by yourself. 

It is completely implemented by yourself. in Linux, this is completely feasible, and all the performance is under your control. This is a wonderful feeling, it is not as difficult as you think.

A. Centralized display 

Centralized display is implemented by CGI of Nagios. Several files need to be read during the CGI display of Nagios. The main configuration file Nagios. cfg; objects. cache file stores the Nagios object information; The status. dat file stores the status information. As long as we provide the three files to it, it will show the results we need.

The main configuration file is not mentioned, objects. cache file, you can use Nagios-pvnagios. CFG to generate objects. precache, which reads the configuration files of all objects and generates a file containing all object configurations. Then you can directly MV the video to objects. cache.

Status. the DAT file can be generated by various means, and the distributed Nagios status can be periodically configured using NFS, SCP, or other methods. dat is collected to the main Nagios, and then the State information contained in these files is integrated into a status using a script. dat. Originally, Nagios passed status. the presentation of DAT is an asynchronous presentation. The presentation time may not be very timely, but Nagios focuses on Alarms after all. Generally, when you receive an alarm, the web page should be displayed early.

In this way, we completed a deception. We cheated the CGI of Nagios and made it think that there was indeed a Nagios running in the background on this machine, but in fact this was just an illusion. Moreover, asynchronous operations have the least impact on Nagios performance.

B. Centralized Configuration 

Centralized configuration is actually not difficult. Since this machine needs to generate the total configuration files of all the objects you need, all the object configurations are required on this machine, you can use methods like SCP and SVN to distribute these configuration files to various distributed Nagios instances, and then restart the configuration files over SSH, which can be solidified into a script, the object configuration file can be placed in different folders according to the different distributed Nagios, and you only need to copy this folder.

C. Centralized Control 

Centralized control is the most troublesome aspect in this solution. The Nagios control is implemented through cmd. CGI write Nagios. if you have multiple Nagios instances, you have many Nagios instances. CMD: first, we need to know which CMD file to write, and then we need to find a way to write the control information to this CMD file. What's even more troublesome is that these files are distributed on different machines, it may be far apart. But there are always some solutions. Fortunately, I am in Linux. My solution is to use the built-in CGI Implementation of Nagios, that is, cmd. CGI: first, I will judge on which Nagios the host or service I operate, and then I will access its CMD. CGI is enough.

First, we need a list. At least we need to understand the host and service running on which Nagios, so that when we make control, we can read this list first, know which machine we need to control. This list can be implemented in the configuration distribution script. Every configuration change will be generated again. Of course, this requires some defined specifications, with different specifications and different scripts.

Then, let the CGI on the main Nagios read this file and call cmd. CGI is extinfo. CGI, we need to make some changes to the source code when it calls cmd. when Using CGI, you first read the list file and know the Nagios host name or IP address of the object to be operated. In short, it is an identifier. Then, this identifier is displayed in the called URL.

The next step is Apache's job. We use different identifiers to redirect cmd. cgi to the host corresponding to it, so that the control is implemented. For Nagios without an Internet address, you can use Apache reverse proxy or other. This is beyond the scope of our discussion.

This method is highly flexible. We can choose the method we like and deliberately bypass the database, which basically has no impact on the performance of Nagios, if the Nagios log is very important to you, you can choose to let each Nagios roll the log once a day, and then collect the log every night and combine it into a Nagios log, in this way, the Nagios log analysis tool can also be used. Of course, there are some real-time methods. You are welcome to provide them.