Detailed Rsyslog/python/loganalyzer logging and viewing service-side/client logs

RSYSLOG is an efficient logging system and is the default journaling system currently used by Ubuntu and CentOS.

Loganalyzer is a PHP-written Web front-end that you can use to analyze and view the logs generated by RSYSLOG.

After research, I am prepared to use these two systems directly. This article has documented the problems I have encountered in configuring both systems.


Introduction to Rsyslog Configuration

Rsyslog is the process for collecting syslog and can be used to replace syslogd or syslog-ng. In these syslog handlers, the individual considers Rsyslog to be the most powerful. Its features include: Support output log to various databases, such as Mysql,postgresql,mongodb,elasticsearch, etc. through Relp + TCP to achieve reliable data transmission (based on this combination of rich filter conditions can build a Reliable data transmission channels for other applications; Fine output format control and powerful filtering of messages; High precision timestamp; queue operations (memory, disk, mixed mode, etc.); Support data encryption and compression transmission.

The relevant RFC reference RFC3164, RFC5424, RFC5425, RFC5426.

This article covers only the configuration of Rsyslog, where the correctness of the configuration instructions is based on the official document and Wiki as a reference.

Configuration file/etc/rsyslog.conf

Please refer to the official documentation for the configuration document here. Http://www.rsyslog.com/doc/rsyslog_conf.html

The Rsyslog configuration file is typically/etc/rsyslog.conf. Module-related configuration specifies that the specified and functionality is available only after the corresponding module is loaded ($ModLoad). Configuration rows can be split into multiple lines using "\" At the end of a line if it is too long. Annotations support two types of syntax, one starting with # to the end of the line and the other in the C language format.





The order in which the instructions are processed is the file content from beginning to end.





Use the command rsyslogd-f/etc/rsyslog.conf-n1 test profile check.





# rsyslogd-f/etc/rsyslog.conf-n1


Rsyslogd:version 7.4.4, config validation run (Level 1), Master config/etc/rsyslog.conf


Rsyslogd:warning: ~ action is deprecated, consider using the ' stop ' statement instead [try Http://www.rsyslog.com/e/2307 ]


Rsyslogd:immark:mark message period must is not is 0, can not run


Rsyslogd:end of config validation run. Bye.





The send test for the log can use the Logger command, which supports sending syslog via UNIX socket,udp/tcp, and so on.





The following is a description of the commonly used configuration directives and is mainly described in Legacy rsyslog format. Most of the official documents are currently described in rainerscript format. Some of the configuration directives are described in separate chapters.





$ActionFileDefaultTemplate [TemplateName] # defines the log output template for file actions





$ActionFileDefaultTemplate Rsyslog_traditionalfileformat





Dec 17:16:58 linux-64 logger[16898]: Test syslog





$ActionFileDefaultTemplate Rsyslog_fileformat





2013-12-30t17:19:50.926770+08:00 linux-64 logger[7757]: Test syslog





$ActionForwardDefaultTemplate [templatename] # Log Output template that defines the default forwarding action





$ActionExecOnlyWhenPreviousIsSuspended [On/off] # value is on indicates that the next action is performed only if the previous action was suspended (failed).





File System related Configuration





$DirCreateMode, $DirGroup, $DirOwner # Set directory creation mode, owner and group





$FileCreateMode, $FileGroup, $FileOwner # Set file creation mode, owner and group





$Umask 0062 # Set file creation mask, equivalent to man 1 Umask





$DynaFileCacheSize # Control the number of dynamic files kept open, for each action


$IncludeConfig # Import other files into the configuration, equivalent to the #include in C


$MainMsgQueueSize # main Message Queue size


$MaxMessageSize # Log maximum size, too large value to consider transport protocol, such as UDP


$ModLoad XXX # module loading





Duplicate Message control


$RepeatedMsgReduction on


$REPEATEDMSGCONTAINSORIGIONALMSG on





Log Receive rate control





The interval at which the Interval set rate is calculated, 0 is closed, and Burst sets the number of logs allowed within the interval. Imuxsock for logs received through the system log socket; Systemlog for other input sources.


$SystemLogRateLimitInterval 0


$SystemLogRateLimitBurst 0


$IMUXSockRateLimitInterval 0


$IMUXSockRateLimitBurst 0





Dec 22:02:36 linux-64 Rsyslogd-2039:imuxsock begins to drop messages from PID 6927 due to rate-limiting


Dec 22:02:39 linux-64 Rsyslogd-2039:imuxsock lost 133250 messages from PID 6927 due to rate-limiting





Module





The complete list of modules is here. Http://www.rsyslog.com/doc/v8-stable/configuration/modules/index.html

The following is a simple introduction to the common types of modules.

input module (im prefix)





Imfile-Convert text file to log





IMRELP-Reliable receive logs via RELP.





$ModLoad Imrelp # Loading module


$InputRELPServerRun 514 # on 514-port monitor





IMUDP-Receive logs via UDP





$ModLoad IMUDP


$UDPServerRun 514





IMTCP-Receive logs via TCP





$ModLoad imtcp


$InputTCPMaxSessions 500


$InputTCPServerRun 514





Imptcp-with IMTCP, but high performance customization for Linux





$ModLoad imptcp


$InputPTCPServerRun 514





Immark-Periodic output tag information





Set to 60, producing a mark log per minute





$MarkMessagePeriod 60





Imklog-Get kernel log (see related information through DMESG)





Imuxsock-unix socket





By default, logs are obtained from/dev/log.





# Netstat-nxp|grep/dev/log


UNIX 3 [] Dgram 7439519 9074/rsyslogd/dev/log





$ModLoad Imuxsock # needs to being done just once


$SystemLogRateLimitInterval 0 # Turn off rate limiting


$InputUnixListenSocketCreatePath on # Turn on for next socket


$InputUnixListenSocket/var/run/sshd/dev/log


$InputUnixListenSocketHostName jail1.example.net


$AddUnixListenSocket/jail/1/dev/log


$InputUnixListenSocketHostName jail2.example.net


$AddUnixListenSocket/jail/2/dev/log





Impstats-rsyslog internal Data Cycle statistics





output module (om prefix)





Omfile-Output to File





Support static file, support dynamic file through template;





OMFWD-built-in modules, forwarding





*.* @192.168.2.11:10514


*.* @@192.168.2.11:10514





OMSNMP-SNMP Trap Output





$ModLoad OMSNMP


$actionsnmptransport UDP


$actionsnmptarget localhost


$actionsnmptargetport 162


$actionsnmpversion 1


$actionsnmpcommunity Public


*.*: Omsnmp:





Omrelp-relp Output Module





$ModLoad Omrelp


*.*: omrelp:loghost.example.com:20514





OMUSRMSG-built-in modules, sending logs to designated users





# Emergencies are sent to everybody logged in.


*.emerg:omusrmsg:*





Omprog-Send log to program processing





For the convenience of program parsing, the output of log is best output in JSON format.





$ModLoad Omprog


$ActionOMProgBinary/home/app/rtdatamq/save_log_data.py


:p Rogramname,startswith, "Log4report": Omprog:; Rsyslog_traditionalfileformat





Ommail-Send mail





Output to database, etc.


Ommysql-mysql


Ompgsql-postgresql


Ommongodb-mongodb


OMLIBDBI-Universal Database Module


Omhdfs-hadoop ' s HDFS


Omelasticsearch-elasticsearch





Other Modules





Message filtering (FM prefix), message resolution (PM prefix), string generation (SM prefix), library module, and so on.

Rules

The rules are made up of filters and actions.

Filtering Device





Facilities/Priority Filters





The format is as follows:





<FACILITY>.<PRIORITY>





which





Represents the subsystem that generates the log. The value range is Auth,authpriv,cron, daemon,kern,lpr,mail,news,syslog,user,uucp,local0 to LOCAL7, reference Man 3 syslog. The corresponding values refer to/usr/include/sys/syslog.h, note that the shift operation needs to be ignored.





Represents the level of the log, with a value range (from low to high, corresponding to the value of 7-0) Debug,info,notice,warning,err,crit,alert,emerg. Refer to Man 3 syslog.





Before the level can add the appropriate modifiers, such as Plus = To select only the priority of the log, Plus! Indicates that all logs that are not equal to the priority are selected, and no symbol is selected to select that priority and above the log. * Can be used to represent all the log subsystems and/or message levels. None of the keywords indicates a log with no level specified. If you want to define multiple settings/priorities, use, separate. If you want to define multiple filter conditions, use; Separated.





Examples are as follows:





kern.* # Select all levels of kernel log


Mail.crit # Select all mail system related logs at level crit and above


Cron.! Info,!debug # Select all cron log information to exclude logs with info and debug priority


*.=debug # Select all Debug-level logs


*.*;auth,authpriv.none # Select All levels of logging, as well as authentication-related no-level logs





Attribute-based Filters





The format is as follows:





: <property>, [!] <compare_operation>, "<STRING>"





which! Represents the inverse of the matching result.





The filter can filter the properties of all logs (relative to the traditional syslog program), for example: TimeGenerated or Syslogtag, all attributes are listed below, and the complete list can refer to the Available Properties section here. Some of the properties are as follows:





Attribute Name Description


MSG Journal Body


Host name in the hostname log


Fromhost messages received from this host may not be the start of the sending host


Fromhost-ip fromhost IP


Syslogtag log labels, such as named[12345]


ProgramName the static portion of the log label, such as named


PRI portion of the PRI log


A textual representation of the Pri-text PRI, such as Syslog.info


Syslogfacility Log Category


Syslogfacility-text the text representation of the log category


Syslogseverity Log Level


Syslogseverity-text Log-level text representation


TimeGenerated log receive time, or understood as timereceived


Report time in the timereported log, or build time


$now Current time, YYYY-MM-DD


$year Current year, YYYY


$month Current month, MM


$day Current log, DD


$hour Current hour, 24-hour format, HH


$hhour the current half hour, 0-29 corresponds 0,30-59 corresponds to 1


$qhour current 1/4 hours, 0-3


$minute the current minute, MM





The difference between timegenerated and timereported please refer to What is the difference between timereported and timegenerated?





Each attribute can be used to define a filter by using the following comparison action and a specified value.


Operator description


The Contains property contains the specified string


The IsEqual property equals the specified string


The StartsWith property starts with the specified string


Regex POSIX BRE Regular expressions


Ereregex POSIX ERE Regular Expressions





Examples are as follows:





: Msg,contains, "Error" # Select the log that contains the error


: Hostname,isequal, "host1" # Select Log with host name Host1


: Msg,!regex, "fatal. * ERROR" # Select the log that does not match the specified regular expression





Filter based on Rainerscript





The format is as follows:





If <EXPRESSION> then <ACTION>





where,<expression> represents an expression, for example: "$msg startswith ' devname ' or $syslogfacility-text = = ' Local0 '". If the expression result is true, the corresponding action is performed.





Action





Save Log to specified file





For example:





Cron.*-/var/log/cron.log





If "-" before the file path indicates that the log file is not synchronized (Fsync) each time the log is output. The file path can be either a static file or a dynamic file. Dynamic files are added by the template before. Defined.





Send logs over the network





The format is as follows:





@[(<options>)]<host>:[<port>]





@ Indicates the use of UDP protocol. @@ 表示 The use of the TCP protocol. <options> can be: z<number> means to use zlib compression, number indicates compression level. Multiple options are used, delimited.





For example:





*.* @192.168.0.1 # Use UDP to send logs to 192.168.0.1


*.* @ @example. com:18 # 18 ports sent to "example.com" using TCP


*.* @ (Z9) [2001::1] # Use UDP to send messages to 2001::1, enable zlib 9-level compression





Send a message to a specific user





For example





: Msg,contains, "error" liuzx





Send the log containing the error to the user liuzx. Multiple users, separated, * represents all users. The user's MESG (Reference man 1 mesg) permission needs to be set to Y.





The above wording is different from the Rsyslog version, may have the following warning message, it is best to make adjustments.





Action ' LIUZX ' treated as ': omusrmsg:liuzx '-Please change syntax, ' LIUZX ' won't be supported in the future





More action





Refer to the documentation for the output module. For example Omprog output log to specify the standard input of the program, Ommysql output to the MySQL database, ompgsql output log to PostgreSQL, ...





Discard Log





For example





cron.* ~





Discard all information, that is, the action after the configuration does not see the log.





Depending on the Rsyslog version, the ~ is modified to stop if you have the following warning message.





Warning: ~ action is deprecated, consider using the ' stop ' statement instead [try http://www.rsyslog.com/e/2307]





For each filter condition, you can specify multiple actions, one for each action line. In this case, you can also use the "&" to represent the previous line of filtration examination. For example:





: Msg,contains, "error" liuzx


& @192.168.1.1





Contains an "error" Log that is sent to the user liuzx and sent via UDP to 192.168.1.1.

Log Output Template





Templates make it more necessary to control the style of log output. The format is as follows:





$template <template_name>, "text%<property>% more Text", [<options>]





$template is a template directive. <TEMPLATE_NAME> is the template name. The text between "" is a template format. The text that is contained in% corresponds to the related property. <options> specify some options for modifying the template functionality, such as SQL or stdsql format text as SQL queries.





Dynamic file output





The output file name is determined by the log and/or System properties.





$template dynamicfile, "/var/log/test_logs/%timegenerated%-test.log."


*.* ? Dynamicfile





Use timegenerated to generate the filename, and use the template to precede it with the.





Other examples are as follows:





$template dailyperhostlogs, "/var/log/syslog/% $YEAR%/% $MONTH%/% $DAY%/%hostname%/messages.log"





Control log output format based on attributes





Use the following format to customize the format of the log by making various modifications to the properties in the template:





%<propname>[:<fromchar>:<tochar>:<options>]%





<propname> property names, available property names refer to the above.





<fromChar> and <toChar> represent the scope of operations on property value strings. Set <fromChar> to r,<tochar> is a regular expression that defines a range through a regular expression.





<options> represents the property options. The complete list can refer to the Property Options section here.





Some examples are as follows:





Full message text for%msg% # log


%msg:1:2% # The first two characters of the log message text


%msg:::d The full message text of the rop-last-lf% # Log, and remove the last line feed


%timegenerated:1:10:date-rfc3339% # The first 10 characters of the timestamp and formatted according to the RFC3999 standard





Here are some examples of templates.





Level of output log, category, time error when receiving log, host name, message label, message body, plus line break:





$template verbose, "%syslogseverity%,%syslogfacility%,%timegenerated%,%hostname%,%syslogtag%,%msg%\n."





Output log source, time and log label, body, and Beep (\7):





$template wallmsg, "\r\n\7message from syslogd@%hostname% at%timegenerated% ... \ r \ n%syslogtag%%msg%\n\r"





Format the log for direct SQL operations:





$template Dbformat, "INSERT into systemevents (message, Facility,fromhost, Priority, Devicereportedtime, Receivedat, Infounitid, Syslogtag) VALUES ('%msg% ',%syslogfacility%, '%hostname% ',%syslogpriority%, '%timereported:::d ate-mysql % ', '%timegenerated:::d ate-mysql% ',%iut%, '%syslogtag% '), SQL





Output in JSON format for ease of program resolution:





$template Jsonformat, "{\ message\": \ "%msg:::json%\", \ "fromhost\": \ "%hostname:::json%\", \ "facility\": \ "% Syslogfacility-text%\ ", \" priority\ ": \"%syslogpriority-text%\ ", \" timereported\ ": \"%timereported:::d ate-rfc3339 %\ ", \" timegenerated\ ": \"%timegenerated:::d ate-rfc3339%\ "}\n"





Note that the content of the message will be one more space before the first, and its explanation please refer to here.





Rsyslog also provides predefined templates (prefixed by Rsyslog_), referring to the Reserved Template Names section here, which is defined as follows:





Rsyslog_fileformat





"%timestamp:::d ate-rfc3339%%hostname%%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::d rop-last-lf%\n\"





Rsyslog_traditionalfileformat





"%timestamp%%hostname%%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::d rop-last-lf%\n\"





Rsyslog_forwardformat





"<%pri%>%timestamp:::d ate-rfc3339%%hostname%%syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"





Rsyslog_traditionalforwardformat





"<%pri%>%timestamp%%hostname%%syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"





Using these templates, you can attach "; Template_name" after the action, for example:





:p rogramname,startswith, "cron"-/var/log/cron; Rsyslog_traditionalfileformat





Deep Understanding





Queues are an important basis for understanding Rsyslog internal principles and configurations, referring to understanding rsyslog queues and turning lanes and Rsyslog Queues-an. The following figure outlines the data flow diagram within the Rsyslog.





Queue-related configuration directives:





$WorkDirectory/rsyslog/work # Intermediate File Store path


$ActionQueueType LinkedList # [Fixedarray/linkedlist/direct/disk]


$ActionQueueFileName relpact # define queue file name


$ActionResumeRetryCount-1 # Number of retries,-1 indicates infinite retry


$ActionQueueSaveOnShutdown on # Rsyslog the contents of the queue when it is closed to prevent data loss


*.*: omrelp:172.19.2.10:2514











syslog System Log Configuration











Linux saves detailed records of events that occur in the system, which are called log files or message files. You can refer to the log file to determine the current state of the system, observe the intruder trail, and look for data related to a particular program (or event). SYSLOGD and KLOGD (monitor the Linux kernel-submitted messages) daemon is responsible for recording, sending the information generated by the system or tool, both of which are/etc/syslog.conf. When the system kernel or tool generates information, it sends the information to the SYSLOGD or KLOGD daemon by calling the relevant function. The syslogd and klogd daemon processes the whereabouts of the message according to the configuration information in the/etc/syslog.conf. A detailed description of the Syslog protocol is in RFC3164.


The Logrotate tool is used to periodically rename, compress, and post system log files, which ensures that log files do not take up too much disk space.


1, configuration file/etc/syslog.conf: Syslog.conf is a syslogd process configuration file that will be read when the program starts, and the default location is/etc/syslog.conf. The blank lines in this configuration file and the rows that begin with "#" are ignored. The "Facility.level" section is also called a selector (seletor). One or more whitespace separators are used between the seletor and the action. It specifies a series of logging rules. The rules are formatted as follows:


Facility.level Action


The selector (Seletor) consists of the facility and level two parts, with a period (.) Connection.


(1) Facility: Specifies the device that generates the log, which can be one of the following keywords:


Keyword value interpretation


Kern 0 Kernel information, first passed through KLOGD


User 1 information generated by your program


Mail 2 e-mail-related information


Daemon 3 information related to the inetd daemon


Auth 4 certified activities reported by PAM_PWDB


Syslog 5 Information generated by the Syslog


LPR 6 information related to print services


News 7 information from the press server


UUCP 8 Information generated by UUCP (UUCP = Unix to UNIX copy)


Cron 9 information about cron and at


Authpriv 10 authentication activities including private information (such as user name)


FTP 11 Information about FTP


12-15 System Retention


Local0 ~ Local7 16-23 is used by custom programs, such as using LOCAL5 as an SSH function


Mark Syslog internal function, for generating time stamps


* Wildcard symbol for all functions except Mark


In most cases, any program can send log messages through any facility, but it is generally in accordance with established rules. For example, only the kernel can use the "Kern" facility. Note that syslog retains the number of device numbers from "local0" to "LOCAL7" to accept log information sent over remote servers and network devices. Routers, switches, firewalls, and load balancers can easily configure a separate log file for each of them.


(2) Level: Specifies the priority of the message, which can be one of the following keywords (descending order, less severity):


Keyword value interpretation


Emerg 0 System is not available


Alert 1 conditions that require immediate modification


Crit 2 (Critical) error conditions that prevent certain tools or subsystem features from being implemented


ERR 3 Block tool or error condition implemented by some subsystem features


Warning 4 Alert Information


Notice 5 common conditions of importance


Info 6 message For information


Debug 7 does not contain additional information about a function condition or problem


None (masking all messages from the specified device) has no precedence, and is typically used for scheduling errors


* All levels except None


The facility part can be multiple devices separated by commas (,), and multiple seletor can also be grouped together by semicolons (;). Note that multiple combinations of selectors, followed by the previous one, allow some precedence to be excluded from the schema. Messages at the specified level and more severity levels are operated by default, but can be modified with 2 operators. Equal to operator (=) means that only the message at this level is operated, unequal operator (!) Represents a message that ignores this level and the severity level. These two operators can be used at the same time, but "!" Must appear in front of "=".


(3) Action action: action determines where the SYSLOGD and klogd daemons send log messages. There are several options:


Normal file: Use the absolute path of the file to indicate where the log file is located, for example:/var/log/cron.


Terminal Equipment: Terminals can be/dev/tty0~/dev/tty6, or can be/dev/console.


User list: For example, the action is "root Hackbutter", which writes the message to the computer screen of user root and Hackbutter.


Remote host: The SYSLOGD daemon that sends information to other hosts on the network in the form "@hostname."


(4) syntax description of the configuration file:


* When used as a device or priority, you can match all the devices or priorities.


* Sends a message to all logged-in users when used as an action.


Multiple selectors can be in the same row and separated by semicolons, and the latter will cover the front. such as Uucp,news.crit.


When the keyword none is used as a priority, all messages from the specified device are masked.


By using the same selector and different actions, the same message can be logged to multiple locations.


The configuration lines that follow in the syslog.conf file do not overwrite the previous configuration line, and each row specifies that the actions are operated independently.


(5) The default/etc/syslog.conf content in my Ubuntu:

# /etc/syslog.conf    syslogd Profile #  More information reference syslog.conf (5) # #  is the first of several standard log files, Messages sent by the corresponding device # #  authentication activity AUTH,AUTHPRIV Log messages are sent to Auth.log auth,authpriv.*             /var/log/auth.log #  all messages except Auth and Authpriv are sent to the syslog *.*;auth, Messages from Authpriv.none        -/var/log/syslog # cron and at are sent to the Cron.log #cron . *                /var/log/ All messages Cron.log # inetd are sent to Daemon.log daemon.*           All messages   -/var/log/daemon.log the #  kernel (such as KLOGD messages) are sent to Kernlog kern.*       All Messages           -/var/log/kern.log #  Print Service are sent to Lpr.log lpr.*                 -/var/log/ Lpr.log #  Mail all the eliminationSend to Mail.log mail.*                 -/var/log/mail.log All messages #  the user program to the User.log user.*                 -/var/log/user.log # #  The log records of the mail system. Split into different files so that it's easier to write scripts #  parse these files # #  messages sent to Mail.info,warn higher level messages sent to Mail.warn, #  Err or higher level message to Mail.err mail.info            -/var/ Log/mail.info mail.warn            -/var/log/ Mail.warn Mail.err            /var/log/mail.err #  inn News system logging: Message from news server # news.crit             /var/log/news/news.crit news.err            / Var/log/news/news.err news.notice             -/var/log/news/news.notice # #  Some captured log files # # 
Debug-level messages for all devices except Auth,authpriv,news,mail are sent to debug *.=debug;/    auth,authpriv.none;/     news.none;mail.none    -/var/log/debug #  except Auth,authpriv,cron, Info,notice for all devices other than Daemon,mail,news, # warn level messages are sent to messages *.=info;*.=notice;*.=warn;/     auth,authpriv.none;/    cron,daemon.none;/    mail,news.none         -/var/log/messages # #  All device Emerg level messages (which the system cannot use) are sent to all logged-in users # *. emerg                * # #  Expect some messages to be displayed on the console, but only on the Virtual Console: Daemon,mail any messages, news dangers, #  errors, notification messages, debugging, information, notification, warning messages for any device # #daemon, mail.*;/#     news.=crit;news.=err;news.=notice;/#    *.=debug;*.=info;/#  & nbSp; *.=notice;*.=warn    /dev/tty8 #  Named pipe/dev/xconsole is used for Xconsloe tools. In order to use it, you must take the-file option to invoke XConsole #    $ xconsole -file /dev/xconsole [...]   Note: Adjust the list below, otherwise if you have a fairly busy site, you will become Mad daemon.*;mail.*;/    news.err;/     *.=debug;*.=info;/    *.=notice;*.=warn    |/dev/xconsole

Some applications also log their health to a specific log file, which is not controlled by the syslog.conf file. such as file/var/log/apache2/access.log, catalogue/var/log/apache2,/var/log/cups,/VAR/LOG/GDM,/var/log/mysql,/var/log/samba, etc.


After the/etc/syslog.conf is modified, the Syslog service can be restarted using the service syslog restart, and sudo/etc/init.d/sysklogd restart is used in Debian/ubuntu. View the latest records in the log file available Tail-f/var/log/messages.


2, logging information to the remote Linux server:Logging your system information to a remote server will be a good security measure. If you log information about all of your company's servers to a centralized syslog server, this will facilitate you to manage the events that occur in each server. This can also make masking errors and malicious attacks more difficult because malicious deletion of individual log files cannot occur at the same time on the log server, especially when you restrict user access to the log server.


Configure the Syslog server: By default, Syslog does not accept remote client information, and to activate it, the/etc/sysconfig/syslog file needs to be modified in Redhat/fedora, Syslogd_ The-r option is added to the options parameter to allow logging of remote machine information. In Debian/ubuntu, you modify the syslogd parameters in the/etc/default/syslogd file, or you can edit the Syslog startup script/etc/init.d/sysklogd directly and define the SYSLOGD parameters as "-r". Restart the Syslog service after the modifications are completed.


Configure client: The Syslog server can now accept syslog information. You also have to set up remote Linux clients to send information to the server. Assume that the client host name is Smallfry and can be implemented by editing the/etc/hosts file.


(1) Determine the IP address and valid domain name of the remote server.


(2) Add a description of the remote server in the/etc/hosts file of the client flight: IP address full domain name server name "Alias", such as 192.168.1.100 bigboy.my-site.com Bigboy Loghost, This sets an alias Loghost for the server host Bigboy.


(3) Edit the client's/etc/syslog.conf file so that the syslog information can be sent to your new set of log server Loghost.


*.debug @loghost


*.debug/var/log/messages


Reboot the Syslog service, and your machine will now send all the "debug" level and above level information to both the server Bigboy and the local file/var/log/messeges. You can do a simple test to see if the syslog server is receiving your information. For example, restart the LPD printer daemon and see if you can see the relevant log messages on the server.


3, the configuration file/etc/logrotate.conf:It is a general configuration file for Logrotate. You can use it to set which log file is recycled and how often it is reused. The cycle parameters can be set weekly (weekly) or per day (daily). The rotate parameter defines how many log file backups are retained, and the Create parameter specifies that a new empty log file be created after each round of loops. The default/etc/logrotate.conf content in my Ubuntu is:

# More details refer to "Man logrotate"
# Weekly Circular update log file
weekly
# Keep 4 copies of log files, if the log file is logfile, its backup
# is logfile.0~ Logfile3
rotate 4
# Create a new blank log file after each round of loops
creating
# If you want to compress the log file, uncomment the following
#compress
# Some software packages that use syslog will place log rotation information in this directory, so include
INCLUDE/ETC/LOGROTATE.D
# No packages have wtmp or btmp, we will rotate here
/var/ log/wtmp {
Missingok # If the log file is missing, use the next and do not send any error messages
Monthly # rotation cycle for the monthly
create 0664 ro OT utmp # When creating a new log file, the permission is 0664, the owner is root, the group is utmp
rotate 1
}
/var/log/btmp {
Missingok
month Ly
Create 0664 root utmp
rotate 1
}
# System-specific logs can be configured here

Note In the Debian/ubuntu system, the/ETC/CRON.DAILY.SYSKLOGD script will read the/etc/syslog.conf file and rotate all the set log files according to it. So, for general system logs, you don't have to repeat the settings in the/ETC/LOGROTATE.D folder. Because it does not run once in 24 hours in the/etc/cron.daily folder. In the Fedora/redhat system, this script is replaced by/etc/cron.daily/logrotate, and it does not read the syslog configuration file. So you have to set it up in/ETC/LOGROTATE.D.


The logrotate settings will only take effect if you use the Logrotate-f command. If you want logrotate to read only specific profiles instead of all, run the configuration file name as a logrotate parameter, such as Logrotate-f/etc/logrotate.d/syslog. If you have a high level of access to your Web pages, the log files will become unusually large. You can compress the log file by editing the logrotate.conf file to add the compression option compress. These log files will be compressed with gzip, and all files will have a. gz extension. It is still easy to view the contents of these files, as you can quickly display their contents on the screen with the Zcat command.


One drawback of the syslog server is that it does not filter out messages from bad sources. Therefore, when your server is in an insecure network, it is a wise move to use TCP wrappers software or firewalls to limit acceptable sources. This will help limit the effectiveness of the denial of service attack, which is designed to fill your server's hard disk or add additional system resources, which could eventually cause the server to crash.


The use of more and more extensive syslog-ng programs now combines the features of Logrotate and syslog to provide users with more customization capabilities and more functionality. If you want to use it, you can go to www.balabit.com to download the installation, its configuration file is/etc/syslog-ng/syslog-ng.conf.











My system configuration:





CentOS 6.3


RSYSLOG 5.8.10


Loganalyzer 4.1.1 (V4-beta)





First, filter logs to separate files





In the two libraries of Git and server.git and serverctrl.git, I used the server-side hook (the service side hook) to automate the deployment. These two hook programs generate log information.





The following settings are done in three things:





Writes the facility as LOCAL5, PRIORITY the log of any value to the All.log;


Writes the log of the program name Hhl-git-serverctrl to Serverctrl.log;


Writes the log of the program name Hhl-git-server to Server.log.





-





local5.*/var/log/hhl/all.log


:p rogramname,isequal, "Hhl-git-serverctrl"/var/log/hhl/serverctrl.log


:p rogramname,isequal, "Hhl-git-server"/var/log/hhl/server.log





I put these in a separate configuration file, stored in the/etc/rsyslog.d/folder. It will be automatically loaded when the RSYSLOGD service restarts.





Second, Loganalyzer does not show facility and PRIORITY








This is because the log file defaults to the old format, and Loganalyzer does not fully support this format. The RSYSLOG WIKI provides a solution. You need to make the following changes:





1. Modify/etc/rsyslog.conf to find the following:





$ActionFileDefaultTemplate Rsyslog_traditionalfileformat





To replace it with:





$ActionFileDefaultTemplate Rsyslog_syslogprotocol23format





This operation replaces the log format with the format defined in RFC 5424. Remember to restart RSYSLOGD when you have finished replacing.





2. Ditto, add this template to the configuration file





$template Traditionalformatwithpri, "<%syslogpriority%>%syslogseverity%%timegenerated%%HOSTNAM E%%syslogtag %%msg:::d rop-last-lf%\n "





3. Enter the Admin Center interface of Loganalyzer to modify Source to use RSyslog Format23 (RFC 5424).








third, write logs using Python





Use Python's logging package to write logs directly to RSYSLOG. In the configuration file above, I specify that facility is LOCAL5, so the same value is given here.





Also note that Formatter the second paragraph of the first argument is the program name, RSYSLOG can rely on that name to filter the log and write to a different file.





#!/usr/bin/env python


Import logging


From logging.handlers import Sysloghandler





Log = Logging.getlogger (' git ')


Log.setlevel (logging. DEBUG)


LOG_HDLR = Sysloghandler (FACILITY=SYSLOGHANDLER.LOG_LOCAL5, address= '/dev/log ')


Log_format = logging. Formatter (


'% (asctime) s hhl-% (name) s-server[% (process) d]:% (message) s: ', '%b%e%h:%m:%s ')


Log_hdlr.setformatter (Log_format)


Log.addhandler (LOG_HDLR)





If you want to add the print effect at the same time, add Streamhandler.





#!/usr/bin/env python


LOG_HDLR2 = logging. Streamhandler (Sys.stdout)


Log_hdlr2.setformatter (Log_format)


Log.addhandler (LOG_HDLR2)