Examples of openresty (nginx lua) statistics on domain name status codes, average response time, and traffic

Background
 
Previously, we used to calculate the domain name status code, average response time, and traffic by adding a scheduled script on each machine to obtain access logs from the last minute of each domain name to a temporary file. Then zabbix makes statistics on the temporary log file for one minute. It has been running well and recently found that the load on a server suddenly increased. Using iotop to view logs of the last minute is found to occupy a very high IO usage. After the scheduled task is stopped, the task returns to normal. So I plan to use nginx lua to replace the current method. The new method features low resource usage and real-time statistics.

Why is it OpenResty?

OpenResty allows developers to use lua programming language to build an existing Nginx C module and support high-traffic applications.

Install OpenResty

Dependent software package:

■ Perl 5.6.1 +
■ Libreadline
■ Libpcre
■ Libssl

Debian and Ubuntu systems:

Apt-get install libreadline-dev libncurses5-dev libpcre3-dev libssl-dev perl make

Fedora, RedHat, and centos systems:

Yum install readline-devel pcre-devel openssl-devel

Download OpenResty

Wget http://openresty.org/download/ngx_openresty-1.5.8.1.tar.gz.

Decompress, compile, and install:

Tar xzvf ngx_openresty-1.5.8.1.tar.gz
Cd ngx_openresty-1.5.8.1/
./Configure -- with-luajit
Make
Make install

 

Other configuration options:

./Configure -- prefix =/opt/openresty
-- With-luajit
Without-http2_module
With-http_iconv_module
With-http_postgres_module
-J2
-- Help to see more options

 
Method introduction
 
The following describes how to use nginx lua to collect website-related data (for example, to collect the status code www.111cn.net 404 ):
Record process:

1. A shared dictionary access is defined to obtain the current timestamp and the current domain name, such as www.111cn.net;
2. We define the dictionary key used to store the status code as www.111cn.net-404-current timestamp;
3. Auto-increment the value of 1 key (www.111cn.net-404-current timestamp;
4. Cycle step 3.
 
Query process:

An interface is provided to accumulate the value of the key www.111cn.net-404-(previous 60 seconds timestamp-current timestamp) and return the result.
 
Method implementation
 
Nginx. conf settings
Http {
[...]
Lua_shared_dict access 10 m;
Log_by_lua_file conf/log_acesss.lua;
Server {
[...]
Location/domain_status {
Default_type text/plain;
Content_by_lua_file "conf/domain_status.lua ";
    }
[...]
}
[...]
}
 
Log_access.lua
Local access = ngx. shared. access
Local host = ngx. var. host
Local status = ngx. var. status
Local body_bytes_sent = ngx. var. body_bytes_sent
Local request_time = ngx. var. request_time
Local timestamp = OS. date ("% s ")
Local expire_time = 70
 
Local status_key = table. concat ({host, "-", status, "-", timestamp })
Local flow_key = table. concat ({host, "-flow-", timestamp })
Local req_time_key = table. concat ({host, "-reqt-", timestamp })
Local total_req_key = table. concat ({host, "-total_req-", timestamp })
 
-- Count total req
Local total_req_sum = access: get (total_req_key) or 0
Total_req_sum = total_req_sum + 1
Access: set (total_req_key, total_req_sum, expire_time)
 
-- Count status
Local status_sum = access: get (status_key) or 0
Status_sum = status_sum + 1
Access: set (status_key, status_sum, expire_time)
 
-- Count flow
Local flow_sum = access: get (flow_key) or 0
Flow_sum = flow_sum + body_bytes_sent
Access: set (flow_key, flow_sum, expire_time)
 
-- Count request time
Local req_sum = access: get (req_time_key) or 0
Req_sum = req_sum + request_time
Access: set (req_time_key, req_sum, expire_time)
Domain_status.lua
 
Local access = ngx. shared. access
Local args = ngx. req. get_uri_args ()
Local count = args ["count"]
Local host = args ["host"]
Local status = args ["status"]
Local one_minute_ago = tonumber (OS. date ("% s")-60
Local now = tonumber (OS. date ("% s "))
 
Local status_total = 0
Local flow_total = 0
Local reqt_total = 0
Local req_total = 0
 
If not host then
Ngx. print ("host arg not found .")
Ngx. exit (ngx. HTTP_ OK)
End
 
If count = "status" and not status then
Ngx. print ("status arg not found .")
Ngx. exit (ngx. HTTP_ OK)
End
 
If not (count = "status" or count = "flow" or count = "reqt") then
Ngx. print ("count arg invalid .")
Ngx. exit (ngx. HTTP_ OK)
End
 
For second_num = one_minute_ago, now do
Local flow_key = table. concat ({host, "-flow-", second_num })
Local req_time_key = table. concat ({host, "-reqt-", second_num })
Local total_req_key = table. concat ({host, "-total_req-", second_num })
 
If count = "status" then
Local status_key = table. concat ({host, "-", status, "-", second_num })
Local status_sum = access: get (status_key) or 0
Status_total = status_total + status_sum
Elseif count = "flow" then
Local flow_sum = access: get (flow_key) or 0
Flow_total = flow_total + flow_sum
Elseif count = "reqt" then
Local req_sum = access: get (total_req_key) or 0
Local req_time_sum = access: get (req_time_key) or 0
Reqt_total = reqt_total + req_time_sum
Req_total = req_total + req_sum
End
End
 
If count = "status" then
Ngx. print (status_total)
Elseif count = "flow" then
Ngx. print (flow_total)
Elseif count = "reqt" then
If req_total = 0 then
Reqt_avg = 0
Else
Reqt_avg = reqt_total/req_total
End
Ngx. print (reqt_avg)
End

Instructions for Use

1. Get the domain name status code

For example, the request for www.111cn.net contains 404 status codes per minute.
Request interface http: // $ host/domain_status? Count = status & host = www.111cn.net & status = 404.

2. Retrieve domain name traffic

Request interface http: // $ host/domain_status? Count = flow & host = www.111cn.net

3. Obtain the average response time of a domain name within one minute

Request interface http: // $ host/domain_status? Count = reqt & host = www.111cn.net