Syslog information loss in Redhat 6.3

we use Linux syslog to record the debug log of the product.

A running file is called. After running the command, look at the debug log information, even from a log after the log has been lost. After several attempts, it was found that the log was lost every time after a fixed log.

This blog post let us come together to explore the end.

I. Problem-finding

I made the following attempt before I found the real problem:

(1) Does the process have some sort of logic exit after fixing log? Or will a signal in the statement after the fixed log print cause the process to terminate? Print a message to the screen at the end of the program to see the program execute properly and exit.

(2) Whether the debug log object has changed. Or has the debug level changed in execution? This information is printed in the same program, and no exception is found.

(3) GDB Debug View program Walk branch logic

If none of the above methods are found, the other is actually another idea: will syslog discard some log information? But I was excluded from the beginning, there are two reasons: the first on the REDHAT4/5 will not appear this problem; the second Redhat 6 platform products have been published for at least a year, if debug log is always missing should not wait until I find out. however. It is because of some habitual thinking. Or some seemingly rational but not rigorous judgment. Cause the truth to be belated.

Then. I took a first look at the/var/log/messages file and saw an error message such as the following. and 6292 is the process ID I ran before.

Imuxsock begins to drop messages from PID 6292 due to rate-limiting

well, obviously. So quickly using the Google artifact, found the reason. This is related to the mechanism of Rsyslog in Redhat 6. And these mechanisms can be configured.

two. Rate limit configuration for Rsyslog in Redhat 6.3

The so-called rate limit refers to the maximum number of log messages that syslog agrees to print in a fixed period of time (the extra log information is discarded). In Redhat 6. The following two configuration items are determined by configuration file/etc/rsyslog.conf:

$SystemLogRateLimitInterval [Number1]: Number1 The time interval size for the set limit

$SystemLogRateLimitBurst [Number2]: Number2 is the maximum amount of log information that is output during the time interval of the set limit.

After Setup, it is assumed that the log information that exceeds the number of Number2 will be removed within each Number1 interval.

The default Number1 is 5 seconds and Number2 is 200. But suppose we don't want to. If there is a loss in the printed log, it can be added or set in/etc/rsyslog.conf :

$SystemLogRateLimitInterval 0

of course after setting up, be sure to remember to start Rsyslog service again (with command: service rsyslog restart) Oh ~ ~ ~

This feature was added to the version number after Note:rsyslog 5.7.1, and Redhat 6.3 was Rsyslog 5.8.10.

References:

1. https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=753363

2. http://lists.adiscon.net/pipermail/rsyslog/2011-April/028307.html

3. http://www.rsyslog.com/tag/rate-limiting/

4. http://www.rsyslog.com/changelog-for-5-7-1-v5-devel/

Syslog information loss in Redhat 6.3