Apache log by Date Format <Good> --

In the Apache configuration file, find
Errorlog logs/error_log
Customlog logs/access_log common

Linux system configuration method:

Change it
Errorlog "|/usr/local/Apache/bin/rotatelogs/home/logs/www/% Y _ % m _ % d_error_log 86400 480 ″
Customlog "|/usr/local/Apache/bin/rotatelogs/home/logs/www/% Y _ % m _ % d_access_log 86400 480" common

Configuration method in Windows:

# Errorlog "| bin/rotatelogs.exe logs/vicp_net_error-% Y % m % d. Log 86400 480 ″
# Customlog "| bin/rotatelogs.exe logs/vicp_net_access-% Y % m % d. Log 86400 480" common

If you do not know how to set the 480 parameter for the first time, the log record time is 8 hours different from the server time. It turns out that rotatelogs has an offset parameter, which indicates the time difference (in minutes) relative to UTC and China is the eighth time zone, the difference is 480 minutes. 86400 indicates a day.

Appendix rotatelogs description

Rotatelogs logfile [rotationtime [offset] | [filesizem]

Option
Logfile
The reference name is the log file name. If logfile contains '%', it is considered as a string in strftime (3) format; otherwise, it is automatically added with the. nnnnnnnnnn Suffix in seconds. Both formats indicate the start time of the new log.
Rotationtime
Interval of Log File rollback in seconds
Offset
The number of minutes relative to the UTC time difference. If it is omitted, it is assumed that it is 0 and UTC time is used. For example, to specify the local time of the area with the UTC time difference of-5 hours, this parameter should be-300.
Filesizem
Specify the size of the file with the suffix m in MB when you roll back, instead of specifying the rollback time or time difference.

Ii. Set Apache logging format

The format of the custom log file involves two commands, namely the logformat command and the customlog command. By default, the httpd. conf file provides several examples of these two commands.

The logformat command defines the format and specifies a name for the format. Then we can directly reference this name. The customlog command sets the log file and specifies the format used by the log file (usually by the format name ).

The function of the logformat command is to define the log format and specify a name for it. For example, in the default httpd. conf file, we can find the following line of code:

Logformat "% H % L % u % t \" % R \ "%> S % B" common

This command creates a log format named "common". The log format is specified in the content enclosed by double quotation marks. Each variable in the format string represents a specific information, which is written to the log file in the order specified by the format string.

The Apache document has provided all the variables that can be used for format strings and their meanings:
%... A: remote IP Address
%... A: local IP Address
%... B: Number of sent bytes, excluding the HTTP Header
%... B: Number of sent bytes in CLF format, excluding the HTTP header. For example, if no data is sent, write '-' instead of 0.
%... {Foobar} e: content of the Environment Variable foobar
%... F: File Name
%... H: Remote Host
%... H request protocol
%... {Foobar} I: content of foobar, the header row of the request sent to the server.
%... L: Remote Login Name (from identd, if provided)
%... M Request Method
%... {Foobar} n: content of the annotation "foobar" from another module
%... {Foobar} O: content of foobar, response header line
%... P: port used by the server to respond to the request
%... P: the ID of the subprocess that responds to the request.
%... Q query string (if a query string exists, it contains "?" Otherwise, it is an empty string .)

%... R: the first line of the request.
%... S: status. For internal redirection requests, this refers to the status of * original * requests. If %…> S, that is, later requests.
%... T: Time in the format of Public log time (or standard English format)
%... {Format} t: Time in the specified format
%... T: The time it takes to respond to the request, in seconds
%... U: remote user (from auth; if the returned status (% s) is 401, it may be forged)
%... U: URL path requested by the user
%... V: servername of the server responding to the request
%... V: The server name obtained based on UseCanonicalName

Among all the variables listed above, "…" Indicates an optional condition. If no condition is specified, the value of the variable is replaced. The default httpd is used before the analysis. the example of the logformat command in the conf file shows that it creates a log format named "common", including: remote host, remote login name, remote user, request time, the first line of the Request Code, the Request status, and the number of bytes sent.

Sometimes we only want to record some specific and defined information in the log, then we need to use "…". If one or more HTTP status codes are put between "%" and the variable, only when the returned status code belongs to the specified status code, the content represented by the variable is recorded. For example, if you want to record all invalid links of a website, you can use:
Logformat @ 4 {Referer} I brokenlinks

If we want to record requests whose status code is not equal to the specified value, we only need to add a "!" Symbol:

Logformat %! 200u somethingwrong

3. Specially record a spider record

Setenvifnocase User-Agent baiduspbaidu_robot
Logformat "% H % t \" % R \ "%> S % B" Robot
Linux
Customlog "|/usr/local/apache2.2.0/bin/rotatelogs/usr/local/apache2.2.0/logs/baidu_policy?m=d.txt 86400 480" Robot Env = baidu_robot
Windows
Customlog "| bin/rotatelogs.exe logs/baidu_policy?m=d.txt 86400 480" Robot Env = baidu_robot
In this example, the log of "baidu_ .txt" is generated every day in the logs directory. Each record is similar to the following:
61.135.168.14 [22/OCT/2008: 22: 21: 26 + 0800] "Get/HTTP/1.1" 200 8427

4. Remove images from logs,JS,CSS, SWF File

<Filesmatch "\. (ICO | GIF | JPG | PNG | BMP | SWF | CSS | JS)">
Setenv imag 1
</Filesmatch>
Customlog "| bin/cronolog.exe logs/cpseadmin/Access _ % Y % m % d. log" combined Env =! Imag

 

How to clear error. log and access. log and limit Apache Log File Size
Access. log and error. the log file has not been moved since the server was installed. Today, the MySQL database connection error of discuz is suddenly displayed, and the error 2003 is displayed. log, access. the log is full, and the number of files reaches 30 GB. Grandma's, immediately search for the files. We need to kill the two boys.

The following method is found on the Internet. It immediately takes effect!

An example of setting in Windows is as follows:
Step 1: Delete the error. log and access. log files in The apache2/logs/directory.

Step 2: Open the httpd. conf configuration file of Apache and find the following two configurations:

Errorlog logs/error. Log
Customlog logs/access. Log common

Directly comment out and replace it with the following configuration file.

# Restrict the error log file to 1 MB
Errorlog "| bin/rotatelogs.exe-l logs/error-% Y-% m-% d. Log 1m"

# Generate an error log file every day
# Errorlog "| bin/rotatelogs.exe-l logs/error-% Y-% m-% d. Log 86400 ″

# Restrict access log files to 1 MB
Customlog "| bin/rotatelogs.exe-l logs/access-% Y-% m-% d. Log 1m 1m" common

# Generate an access log file every day
# Customlog "| bin/rotatelogs.exe-l logs/access-% Y-% m-% d. Log 86400" common

Refer:

How to process access. log and error. log files in Apache

In the past few days, Members and I have said that the website access speed is getting slower and slower. I checked and found that two log files under apache2 are very large, with a total of over 800 mb.
They are access. log and error. log.

So I found a method on the Internet to lose weight for access. log and error. log. This method allows the two files to be generated on a daily basis. In this way, you can choose
Delete the old file.

Find the following two sentences in the httpd. conf configuration file under Apache:

Errorlog logs/error. Log

Customlog logs/access. Log common

Then, replace the two sentences as follows:

Customlog "| D:/apache2/bin/rotatelogs.exe D:/apache2/logs/Access _ % Y _ % m _ % d. Log 86400 480" common

Errorlog "| D:/apache2/bin/rotatelogs.exe D:/apache2/logs/error _ % Y _ % m _ % d. Log 86400 480 ″

Everything is so simple, so that these two log files will start a new file every day, so that a single file will not be too large and cannot be opened, and the log information cannot be seen.
You can also delete the preceding log file.

Access. logs, the pieces will reach dozens or even hundreds of megabytes after running on the Web server for a period of time. If there is an error in Apache running, error. logs will also increase to dozens of megabytes. We know that the system is very memory-consuming to read and write a large text file. Therefore, it is necessary to limit the log file size.
By referring to the http://httpd.apache.org/docs/2.0/programs/rotatelogs.html, you can use Apache's own program rotatelogs.exe (located under the {$ Apache}/bin/directory) to limit the size of the log file.
Usage: rotatelogs [-L] [offset minutes from UTC] or
Add this:
Transferlog "| rotatelogs/Some/where 86400 ″
Or
Transferlog "| rotatelogs/Some/where 5m"
To httpd. conf. the generated name will be/Some/where. NNNN where NNNN is the system time at which the log nominally starts (N. b. if using a rotation time, the time will always be a multiple of the rotation time, so you can synchronizecron scripts with it ). at the end of each rotation time or when the file size is reached a new log is started.
An example of setting in Windows is as follows:
# Restrict the error log file to 1 MB
Errorlog "| bin/rotatelogs.exe-l logs/error-% Y-% m-% d. Log 1m"
# Generate an error log file every day
# Errorlog "| bin/rotatelogs.exe-l logs/error-% Y-% m-% d. Log 86400 ″
# Restrict access log files to 1 MB
Customlog "| bin/rotatelogs.exe-l logs/access-% Y-% m-% d. Log 1m 1m" common
# Generate an access log file every day
# Customlog "| bin/rotatelogs.exe-l logs/access-% Y-% m-% d. Log 86400" common
It should be similar in Linux/Unix.

How to clear Apache access. Log

A client server was built using Apache. Recently, the website is always very slow, the server is very slow, and sometimes the website cannot be opened. After investigation and analysis, it turns out to be the access in it. log and error. log files should be read and cleared frequently. If you are busy and forget to read and clean them up, the two files will expand very large in a short time, it cannot be opened. The following describes how to clean the access. log and error. log files. I suspect there are other crawlers. I will be crawling several of my websites tomorrow.
The methods to optimize access. log and error. Log are as follows:
Customlog "| D:/thridparty-system/Java/apache2/bin/rotatelogs.exe D: /thridparty-system/Java/apache2/logs/Access _ % Y _ % m _ % d. log 86400 480 "common
Errorlog "| D:/thridparty-system/Java/apache2/bin/rotatelogs.exe D: /thridparty-system/Java/apache2/logs/error _ % Y _ % m _ % d. log 86400 480 ″
Everything is so simple, so that these two log files will start a new file every day, so that a single file will not be too large and cannot be opened, and the log information cannot be seen.

A solution to the increasingly expanding access. log of Apache Log Files

Change customlog logs/access. Log common in httpd. conf

Customlog "| C:/Apache/bin/rotatelogs C:/Apache/logs/% Y _ % m _ % d. Access. Log 86400 480" common

Restart Apache

C:/Apache/is the path to install Apache.
In this way, a log file is generated every day.

A solution to the increasingly expanding access. log of Apache Log Files

 

View and analyze Apache logs

 

Assume that the Apache log format is:
118.78.199.98--[09/Jan/2010: 00: 59: 59 + 0800] "Get/public/CSS/index.css HTTP/1.1" 304-"http://www.a.cn/common/index.php”" Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; sv1; gtb6.3 )"

Problem 1: In apachelog, find the top 10 IP addresses with the most accesses.
Awk '{print $1}' apache_log | sort | uniq-c | sort-Nr | head-N 10

Awk first captures the IP addresses in each log. If the log format is customized, you can use-F to define the delimiter and print the specified columns;
Sort for the first time to sort the same records together;
Upiq-C combines repeated rows and records the number of repeated rows.
Head to filter the top 10;
Sort-nr is sorted in reverse order by number.

The commands I reference are:
Display 10 most commonly used commands
Sed-e "s/| // n/g "~ /. Bash_history | cut-D ''-F 1 | sort | uniq-c | sort-Nr | HEAD

Question 2: Find the maximum number of accesses in the Apache log for several minutes.
Awk '{print $4}' access_log | cut-C 14-18 | sort | uniq-c | sort-Nr | HEAD
[09/Jan/2010: 00: 59: 59;
Extract 14 to 18 characters from cut-C
The remaining content is similar to question 1.

Question 3: Find the most visited page in the Apache log:
Awk '{print $11}' apache_log | SED's/^. * cN /(. */)/"// 1/G' | sort | uniq-c | sort-Rn | HEAD

Similar to Problems 1 and 2, the only special feature is to replace "http://www.a.cn/common/index.php?" with the internal content of" http://www.a.cn (/common/index. php) "with the SED replacement function )"

Question 4: In the Apache Log, find the maximum number of times of access (the most load) (in minutes), and then check which IP addresses have the most access at these times?
1. view the Apache process:
PS aux | grep httpd | grep-V grep | WC-l

2. View TCP connections on port 80:
Netstat-tan | grep "established" | grep ": 80" | WC-l

3. Check the number of IP connections on the current day in the log to filter duplicates:
Cat access_log | grep "19/May/2011" | awk '{print $2}' | sort | uniq-c | sort-NR

4. What are IP addresses with the highest number of IP connections doing on that day (originally spider ):
Cat access_log | grep "19/May/1::00" | grep "61.135.166.230" | awk '{print $8}' | sort | uniq-c | sort-Nr | head-N 10

5. Visit the top 10 URLs on the page on the current day:
Cat access_log | grep "19/May/2010:00" | awk '{print $8}' | sort | uniq-c | sort-Nr | head-N 10

6. Use tcpdump to sniff access to port 80 to see who is the highest
Tcpdump-I eth0-tnn DST port 80-C 1000 | awk-F ". "'{print $1 ". "$2 ". "$3 ". "$4} '| sort | uniq-c | sort-NR

Then, check the log to see what the IP address is doing:
Cat access_log | grep 220.181.38.183 | awk '{print $1 usd/t "$8}' | sort | uniq-c | sort-Nr | less

7. view the number of IP connections in a certain period of time:
Grep "2006:0 [7-8]" www20110519.log | awk '{print $2}' | sort | uniq-c | sort-Nr | WC-l

8. the 20 most connected IP addresses in the current Web Server:
Netstat-NTU | awk '{print $5}' | sort | uniq-c | sort-n-r | head-N 20

9. view the top 10 IP addresses with the most visits in the log
Cat access_log | cut-D ''-F 1 | sort | uniq-c | sort-Nr | awk '{print $0}' | head-N 10 | less

10. view more than 100 IP addresses in the log
Cat access_log | cut-D ''-F 1 | sort | uniq-c | awk '{if ($1> 100) Print $0}' | sort-Nr | less

11. view the most recently visited files
Cat access_log | tail-10000 | awk '{print $7}' | sort | uniq-c | sort-Nr | less

12. view the pages that have been accessed for more than 100 times in the log
Cat access_log | cut-D ''-F 7 | sort | uniq-c | awk '{if ($1> 100) Print $0}' | less

13. List objects that have been transferred for more than 30 seconds.
Cat access_log | awk '($ NF> 30) {print $7}' | sort-N | uniq-c | sort-Nr | head-20

14. List the most time-consuming pages (more than 60 seconds) and their occurrence times
Cat access_log | awk '($ NF> 60 & $7 ~ //. Php/) {print $7} '| sort-N | uniq-c | sort-Nr | header-100