Compile a custom module for real-time statistics for nginx

Before the official start, let's talk about the notes for writing the nginx custom module:

In the function with the r-connection.log to print the log core, this is found that the ngx header file and lua header file reference sequence problem, put the ngx header file in the front to solve

Ngx_str_t, a string type of nginx, has two parameters: len and data. These two parameters must be used together. Because the end of data is not necessarily the length of len, pay attention to this.

To compile with the cpp file, add-with-ld-opt = "-lstdc ++" to the ngx compilation parameters"

OK. I am not talking about this nonsense. I will officially start to talk about the statistical module I wrote this time.

The requirement background is that many servers have been mounted behind nginx, and nginx needs to be used to calculate the success rate, response time, and other parameters. After half a day on the Internet, access_log is used in most cases, then we use the program to scan $ request_time for implementation. This is intolerable for a server with thousands of accesses per second, so there is no way to end up, so we can write it by ourselves ~

I read the development document of the nginx custom module again. The calling process is as follows:

 

 

However, the callback function at the end of the request is not found. The closest thing is to use the filter module (that is, the filter module). Of course, the request time calculated in this way is, it may be shorter than the actual time.

OK. After we have decided to write the module, let's take a look at the specific implementation.

To maximize performance, the UDP protocol is reported and the retrieval package is not received. The socket is not released after it is created.

Since network reporting is involved, you need to set parameters such as ip address, port, and source for reporting.

The filter module has two functions: ngx_http_output_header_filter_pt and ngx_http_output_body_filter_pt.

The reported fields should include method, uri, request_time, http status code, target IP address, and so on.

UDP is reported to the client here, because it uses the company's library and is very simple. I will not elaborate on it here. You will also see the code, I used static variables to ensure no structure analysis:

Static COpenApiMonitorClient client;

 

Here, the configuration code is as follows:

Typedef struct {
Ngx_str_t host;
Ngx_int_t port;
Ngx_str_t collect_point;
} Ngx_http_stat_report_conf_t;

Static ngx_command_t ngx_http_stat_report_filter_commands [] = {
{Ngx_string ("stat_report_host "),
NGX_HTTP_MAIN_CONF | NGX_HTTP_SRV_CONF | NGX_HTTP_LOC_CONF | NGX_CONF_TAKE1,
Ngx_conf_set_str_slot,
NGX_HTTP_LOC_CONF_OFFSET,
Offsetof (ngx_http_stat_report_conf_t, host ),
NULL },

{Ngx_string ("stat_report_port "),
NGX_HTTP_MAIN_CONF | NGX_HTTP_SRV_CONF | NGX_HTTP_LOC_CONF | NGX_CONF_TAKE1,
Ngx_conf_set_num_slot,
NGX_HTTP_LOC_CONF_OFFSET,
Offsetof (ngx_http_stat_report_conf_t, port ),
NULL },

{Ngx_string ("stat_report_collect_point "),
NGX_HTTP_MAIN_CONF | NGX_HTTP_SRV_CONF | NGX_HTTP_LOC_CONF | NGX_CONF_TAKE1,
Ngx_conf_set_str_slot,
NGX_HTTP_LOC_CONF_OFFSET,
Offsetof (ngx_http_stat_report_conf_t, collect_point ),
NULL },

Ngx_null_command
};

So I chose ngx_http_output_header_filter_pt or else here. I finally chose ngx_http_output_header_filter_pt. Although there are differences in computing request time, it is more complicated to write because the token will enter multiple times, so it's easy ~

The code is as follows:

Static ngx_int_t
Ngx_http_stat_report_header_filter (ngx_http_request_t * r)
{
Ngx_http_stat_report_conf_t * conf;

Conf = (ngx_http_stat_report_conf_t *) ngx_http_get_module_loc_conf (r, ngx_http_stat_report_filter_module );

SendStatReport (r, conf );

Return ngx_http_next_header_filter (r );
}

For the reported field, this is mainly about the meaning of each ngx_http_request_t field for a long time. I will not explain it here. I will paste the code directly and you should be able to understand it directly.

Ngx_http_request_body_t * rb = r-> request_body;

Char * body = NULL;
Int body_size = 0;

If (rb & rb-> buf)
{  
Body = (char *) rb-> buf-> pos;
Body_size = rb-> buf-> last-rb-> buf-> pos;
}  

String str_uri = r-> uri. data? String (char *) r-> uri. data, r-> uri. len ):"";

String protocol = r-> http_protocol.data? String (char *) r-> http_protocol.data, r-> http_protocol.len ):"";

String str_data;

Map <string, string> params;

If (r-> method = 2) // get
{  
Pkg. method = "GET ";
Str_data = r-> args. data? String (char *) r-> args. data, r-> args. len ):"";
}  
Else if (r-> method = 8)
{  
Pkg. method = "POST ";
Str_data = (body & body_size> 0 )? String (body, body_size ):"";
}  
Else
{  
Return-1;
}  
// Ngx_log_error (NGX_LOG_ERR, r-> connection-> log, 0, "args: % s uri: % s protocol: % s end", str_data.c_str (), str_uri.c_str (), protocol. c_str ());

Trans2MapParams (str_data, params );

Ngx_msec_int_t MS = get_pass_time_ms (r );
Double time_sec = (double) MS)/1000;

Pkg. appid = strtoul (params ["appid"]. c_str (), NULL, 10 );
Pkg. rc = 0;
If (r-> headers_out.status> = 400) // it indicates a problem occurs.
{  
Pkg. rc = r-> headers_out.status;
}  
Pkg. timestamp = time (NULL );
Pkg. time = time_sec;
Pkg. pf = params ["pf"];
// Forwarded IP address and port
Pkg. svr_name = "";
If (r-> upstream & r-> upstream-> peer. name & r-> upstream-> peer. name-> data)
{  
Pkg. svr_name = string (char *) r-> upstream-> peer. name-> data, r-> upstream-> peer. name-> len );
}
Pkg. interface = str_uri;
Pkg. protocol = ParseProtocol (protocol );
Pkg. collect_point = conf-> collect_point.data? String (char *) conf-> collect_point.data, conf-> collect_point.len ):"";

OK. This is basically the case for the entire code.

By convention, the new problems found below have plagued me for a long time and have not been solved yet.

When ngx_log_error is printed to % u, the process Exits. So far, you do not know why. The solution is to print it to % d.

Ngx_log_error is printed as int when % f is printed. Even if %. 2f is specified, the printed result is incorrect. Somehow, the solution is to convert the value * 1000 to int.

Finally, the uploaded googlecode

Https://vimercode.googlecode.com/svn/trunk/nginx_stat_report