Detailed analysis of Nginx logs growing too fast

Detailed analysis of Nginx logs growing too fast

Preface:

In the Nginx log, Mobileweb_access.log has a very large growth, with hundreds of megabytes per day and nearly 500 million access records. According to our current scale, there are only about popular users. Even if everyone accesses the Nginx log using a mobile app, how can there be such a large volume of url access? In the past, nginx was only installed and used, and no time has been taken for careful research. This time, nginx logs need to be thoroughly analyzed.

1. Log Classification
There are two main types: error logs and access logs. These configurations are in/usr/local/nginx/conf/nginx. conf, Which is enabled by default. You can also disable it.

1.1. Access logs
Access logs mainly record every request to access nginx. The format can be customized. in the conf file, you can view the detailed information of each request through access logs. The format of access logs is mainly restricted by the log_format in the configuration file.

1.1.1 log_format log format

$ Request_time: total time of the request.

$ Time_iso8601: Access time and time zone, such as 18/Jul/2012: 17: 00: 01 + 0800, the "+ 0800" at the end of the time information indicates that the time zone of the server is 8 hours after UTC.

$ Upstream_response_time: the response time of upstream during the request process.

$ Request_method: the action requested by the client, usually GET or POST.

$ Request_uri: The value sent by the browser. This value is the value after rewrite. For example, after internal redirects is implemented.
 

$ Args: this variable is equal to the parameters in the request line (GET request), for example, foo = 123 & bar = blahblah;

$ Query_string: Same as $ args.

$ Proxy_add_x_forwarded_for: The variable contains "X-Forwarded-For" in the client request header, which is separated from $ remote_addr by commas. If no "X-Forwarded-For" Request Header exists, then $ proxy_add_x_forwarded_for is equal to $ remote_addr.

$ Upstream_addr: the address of upstream, that is, the host address that actually provides services.
 

$ Status: record the http status code returned by the request. For example, if the request is successful, it is 200.

$ Http_user_agent: client browser Information

$ Http_range

$ Sent_http_content_length: length of the sent content

$ Body_bytes_sent: size of the file body content sent to the client, such as 899. This value in each log record can be accumulated to roughly estimate the server throughput.
 

$ Http_referer: record the Page Link from which the access is made.

$ Host: The Request host header field. Otherwise, it is the server name.

$ Http_x_forwarded_for: The real ip address of the client. Generally, the web server is placed behind the reverse proxy, so that the client's ip address cannot be obtained. The ip address obtained through $ remote_add is the ip address of the reverse proxy server. The reverse proxy server can add x_forwarded_for information in the http header of the forwarding request to record the IP address of the original client and the server address of the original client request.

$ Http_user_agent: client browser Information

$ Body_bytes_sent: size of the file body content sent to the client, such as 899. This value in each log record can be accumulated to roughly estimate the server throughput.
 

$ Ssl_protocol: SSL protocol version, such as tlsv1.

$ Ssl_cipher: Algorithms in data exchange, such as RC4-SHA.
 

Examples in the production environment:

Log_format main '$ proxy_add_x_forwarded_for $ remote_user [$ time_local] "$ request "'

'$ Status $ body_bytes_sent "$ http_referer "'

'"$ Http_user_agent" "$ http_x_forwarded_for "'

'Upsteam: $ upstream_addr ';

Access_log logs/access. log main;

Log_not_found off;
 

1.1.2. Access Log Path

Access_log logs/access. log main;

Nginx supports specifying powerful log records for each location. The same connection can be output to more than one log at a time. To disable logging, you can:

Access_log off;

The fields that can use the access_log command include http, server, and location.

PS: users and groups set by the Nginx process must have the permission to create files on the Log Path. Otherwise, an error is reported.

 
1.2, Error Log
Error logs mainly record the logs when an error occurs when the client accesses Nginx. The format cannot be customized. Through the error log, you can get the performance bottleneck of a system service or server. Therefore, you can get a lot of valuable information by making full use of logs. The error log is specified by the command error_log. The specific format is as follows:

Error_log path (storage path) level (Log level)

Path means the same as access_log, and level indicates the log level, as shown below:

[Debug | info | notice | warn | error | crit]

From left to right, the log details are gradually decreased, that is, debug is the most detailed and crit is the least. The example is as follows:

Error_log logs/mobileweb_error.log error;

Note that error_log off does not close the error log, but records the error log to a file named "off. The correct method to disable the error logging function is as follows:

Error_log/dev/null;

It indicates that the log storage path is set to "garbage bin ".

-------------------------------------- Split line --------------------------------------

Deployment of Nginx + MySQL + PHP in CentOS 6.2

Build a WEB server using Nginx

Build a Web server based on Linux6.3 + Nginx1.2 + PHP5 + MySQL5.5

Performance Tuning for Nginx in CentOS 6.3

Configure Nginx to load the ngx_pagespeed module in CentOS 6.3

Install and configure Nginx + Pcre + php-fpm in CentOS 6.4

Nginx installation and configuration instructions

Nginx log filtering using ngx_log_if does not record specific logs

-------------------------------------- Split line --------------------------------------

2. Define specific logs for each project
Location ~ * ^/MobileWeb/. * $ {

Client_max_body_size 5 m;

Include deny. conf;

Proxy_pass http: // mobilewebbackend;

Include proxy. conf;

Error_log logs/mobileweb_error.log error;

Access_log logs/mobileweb_access.log main;

Include gzip. conf;

}

In this way, the mobileWeb project's dedicated logs mobileweb_error.log and mobileweb_access.log will be generated under the Log Path/usr/local/nginx/logs/. If you want to query the access records of the mobileWeb project, you can view the two logs separately.

For more details, please continue to read the highlights on the next page:

1 2 Next Page