Use ngxtop to monitor webserver access in real time

With regard to the real-time monitoring of nginx web server access, I wanted to implement it a long time ago. Although there are some status extensions provided by nginx, they are global and cannot be subdivided into vhost, in addition, metric is rarely provided. In addition, at present, the nginx status is obtained every 5 minutes through cacti, and the real-time performance is not very good. I remember a little excited for maptail in the past and thought it could solve my problem, however, he only beautifully displays the real-time user's ip location information on the map, without other functions, but his idea is very important-"tail-f log ", today's ngxtop is actually such an idea. It enriches the functions. I encountered some problems during the test today. Now I have sorted out the process and provided my memo for your reference.

Although ngxtop still has some problems, such as the end of this article, it is only 15 days open-source, and it is worth looking forward!
Real-time metrics for nginx server: https://github.com/lebinh/ngxtop 1. Install python
Tar xzvf Python-2.7.3
Cd Python-2.7.3
./Configure -- prefix =/usr/local/python2.7
Make
Make install

Path of python2.7 after installation
/Usr/local/python2.7/bin/python2.7

To keep python2.4 unchanged, set up a soft connection of python2.7 to the path of python2.7.
Mv/usr/bin/python. bak
Ln-s/usr/local/python2.7/bin/python2.7/usr/bin/python2.7
Ln-s/usr/bin/python2.7/usr/bin/python

2. Install setuptools
Wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg -- no-check-certificate
Chmod + x setuptools-0.6c11-py2.7.egg
Sh setuptools-0.6c11-py2.7.egg

3.1 install ngxtop using pip, because the source is abroad, the download and installation process may be slow
Install pip
Https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2904f92d46aaa333267fb1c922bb
Tar xzvf pip-1.5.4.tar.gz
Cd pip-1.5.4
Python setup. py install

Install ngxtop
/Usr/local/bin/pip install ngxtop

3.2 install the SDK directly through ngxtop source code
Wget https://github.com/lebinh/ngxtop/archive/1c200d51fbae7824a30159714669146d6b214210.zip
Unzip ngxtop-1c200d51fbae7824a30159714669146d6b214210.zip
Cd ngxtop-1c200d51fbae7824a30159714669146d6b214210
Python setup. py install

If you are prompted that no pyparsing exists, download and install it.
Wget https://pypi.python.org/packages/source/p/pyparsing/pyparsing-2.0.1.zip
Unzip pyparsing-2.0.1.zip
Cd pyparsing-2.0.1
Python setup. py instal

4. Use and effect
If not, debug:
Ngxtop-t 1-l/var/log/nginx/access. log-d

Default output:
Ngxtop-t 1-l/var/log/nginx/access. log


Displays the most requested ip addresses and requests
Ngxtop-t 1-l/var/log/nginx/access. log top remote_addr request


Remote Real-Time Analysis (this is very practical, you do not need to install ngxtop on each server)
Sshpass-f pass.txt ssh 192.168.0.100 tail-f/var/log/nginx. log | ngxtop

Other usage

Ngxtop -- help

5. problems encountered during the test
(1) When ngxtop is executed, the following error will be reported: ImportError: No module named _ sqlite3
#/Usr/local/bin/ngxtop -- help
Traceback (most recent call last ):
File "/usr/local/bin/ngxtop", line 7, in <module>
From ngxtop. ngxtop import main
File "/usr/local/lib/python2.7/site-packages/ngxtop. py", line 60, in <module>
Import sqlite3
File "/usr/local/lib/python2.7/sqlite3/_ init _. py", line 24, in <module>
From dbapi2 import *
File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
From _ sqlite3 import *
ImportError: No module named _ sqlite3


Solution: Edit Python-2.7.3/Modules/_ sqlite/connection. c and add the following lines 44-57. Then re-compile python2.7.
[Root @ admin200 Python-2.7.3] # vim Modules/_ sqlite/connection. c
38 # if SQLITE_VERSION_NUMBER >=3003008
39 # ifndef SQLITE_OMIT_LOAD_EXTENSION
40 # define HAVE_LOAD_EXTENSION
41 # endif
42 # endif
43
44/*** add by lai **/
45 # ifdef SQLITE_INT64_TYPE
46 typedef SQLITE_INT64_TYPE sqlite_int64;
47 typedef unsigned SQLITE_INT64_TYPE sqlite_uint64;
48 # elif defined (_ MSC_VER) | defined (_ BORLANDC __)
49 typedef _ int64 sqlite_int64;
50 typedef unsigned _ int64 sqlite_uint64;
51 # else
52 typedef long int sqlite_int64;
53 typedef unsigned long int sqlite_uint64;
54 # endif
55 typedef sqlite_int64 sqlite3_int64;
56 typedef sqlite_uint64 sqlite3_uint64;
57/*** end ***/
58
59
60 static int pysqlite_connection_set_isolation_level (pysqlite_Connection * self, PyObject * isolation_level );
61 static void _ pysqlite_drop_unused_cursor_references (pysqlite_Connection * self );
62

(2) unable to start ngxtop, error: OSError: [Errno 2] No such file or directory
[Root @ admin200 nginx] #/usr/local/bin/ngxtop
Traceback (most recent call last ):
File "/usr/local/bin/ngxtop", line 11, in <module>
Sys. exit (main ())
File "/usr/local/python2.7/lib/python2.7/site-packages/ngxtop. py", line 446, in main
Process (args)
File "/usr/local/python2.7/lib/python2.7/site-packages/ngxtop. py", line 408, in process
Config = get_nginx_conf_path ()
File "/usr/local/python2.7/lib/python2.7/site-packages/ngxtop. py", line 120, in get_nginx_conf_path
Proc = subprocess. Popen (['nginx', '-V'], stderr = subprocess. PIPE)
File "/usr/local/python2.7/lib/python2.7/subprocess. py", line 679, in _ init __
Errread, errwrite)
File "/usr/local/python2.7/lib/python2.7/subprocess. py", line 1249, in _ execute_child
Raise child_exception
OSError: [Errno 2] No such file or directory
[Root @ admin200 nginx] #


Solution: Because nginx is a custom installation directory, ngxtop needs to find the nginx pid. Therefore, you need to add the nginx path to the environment variable or make a symbolic link.
Cp/usr/local/nginx/sbin/nginx/usr/bin/
Or
Ln-s/usr/local/nginx/sbin/nginx/usr/bin/nginx

(3) Remote Real-Time Analysis of logs of other servers
Sshpass-f pass.txt ssh 192.168.0.100 tail-f/var/log/nginx/bod-access.log | ngxtop

An error is reported after several seconds of running. The problem has been reported by developers.
Traceback (most recent call last ):
File "/usr/local/bin/ngxtop", line 11, in <module>
Sys. exit (main ())
File "build/bdist. linux-x86_64/egg/ngxtop. py", line 495, in main
File "build/bdist. linux-x86_64/egg/ngxtop. py", line 463, in process
File "build/bdist. linux-x86_64/egg/ngxtop. py", line 357, in process_log
File "build/bdist. linux-x86_64/egg/ngxtop. py", line 305, in process

Sqlite3.ProgrammingError: library routine called out of sequence