Linux log Parsing

Linux log parsing can be said to be a monitoring system record, and the system will basically record every action. In this way, the information is very comprehensive and important. Generally, only the root user can perform inspections! You can use the logon file (Log File) to solve most of the Linux problems based on the error messages on the screen and the error messages matching the logon file! Therefore, log files are very important. As a qualified linux system engineer, log files must be mastered. Common Logon files include/var/log/secure: files that log on to the system to access data, such as pop3, ssh, telnet, and ftp; /var/log/wtmp: records the information data of the login user. Because this file has been encoded, you must use the last command to retrieve the file content;/var/log/messages: it is particularly important that almost any error message (or important information) is recorded here;/var/log/boot. log: records the startup or shutdown messages displayed when a service is started./var/log/maillog or/var/log/mail /*: record the user record of email access or exchange (sendmail and pop3);/var/log/cron: record the contents of the crontab routine service. /Var/log/httpd,/var/log/news,/var/log/mysqld. log,/var/log/samba,/var/log/procmail. log: several different network service record files! One of the record programs in the logon file: syslogd usually records data after syslog: the date and time of the event; Host Name of the event; the service name (such as samba and xinetd) or function name (such as libpam ..) that starts this event ..); the daemon configuration file of syslogd is/etc/syslog. the conf content syntax is as follows: Service name [. =!] Name of the file or device or host of the information level message record # For example, mail.info/var/log/maillog_info service name: the meaning of the message generated by this service. Syslog recognizes the following services: auth, authpriv: Authentication-related mechanisms, such as telnet, login, and ssh. cron: routine commands such as cron/at generate information records; daemon: messages related to various daemon; kern: the location where the core (kernel) generates messages; lpr: prints related messages! Mail: as long as the message records related to sending and receiving all belong to this, news: something related to the news group server, and syslog: syslogd information generated by this program! User, uucp, local0 ~ Local7: some messages related to the Unix like machine itself. The information level system divides messages into seven major levels, from unimportant to important information levels in sequence: info: only some basic information descriptions; notice: information that needs to be noticed more than info; warning or warn: warning message, which may be faulty, but will not affect the operation of a daemon. Err or error: There are some major error messages, so we need to find the cause. Crit: an error message that is more serious than an error. crit is short for the critical point (critical). It is already very serious! Alert: Warning and warning. It is already very problematic and more serious than crit! Emerg or panic: pain level, meaning that the system is almost ready to become a machine! The error message is very serious. In addition to these classified messages, there are two special levels: debug (error detection level) and none (logon level not required). When error detection is required, or you can ignore some service information! Before the message level, there are [. =!]! It means :.: indicates that a higher level (including this level) than the subsequent one is recorded. For example, mail.info indicates that the information of mail is higher than info) will be recorded .. =: Indicates that the required level is followed by the level !.!: Not equal. The file name recorded by the log file or the common location of the device or host: the absolute path of the file: Usually the file in/var/log! Printer or other: for example, the/dev/lp0 printer device (even if a hacker can delete the log file, but the printed log information cannot be deleted) User name: displayed to the user! Remote Host: for example, @ test.adsldns.org can be supported by the other host! *: Represents all online users. It is similar to the meaning of the wall command! Check the relevant information from Fedora Core Release 4 without enabling the network service [root @ linux ~] # Vi/etc/syslog. conf # kern. */dev/console # All messages generated by the kernel are sent to the console! Disabled by default. *. Info; mail. none; authpriv. none; cron. none/var/log/messages # do not record known services in this file, and record known services in a separate log file for future query, otherwise, the file messages is too messy. This file is very important, and all unknown information will be recorded in this file. Therefore, if there is a problem, it will be better to find this file. Authpriv. */var/log/secure # This is the file that needs to be recorded after identity confirmation. Mail. *-/var/log/maillog # This file is recorded if it is related to mail (pop3 or sendmail! Cron. */var/log/cron # related to routine commands. *. Emerg * # Any warning message is displayed to all online users! That * is the current online user. Uucp, news. crit/var/log/spooler # records information with a level where the news error is higher than the crit, and writes it to spooler! Local7. */var/log/boot. log # Write the messages in the boot to/var/log/boot. log! Syslog. conf varies greatly in each version. Therefore, the data recorded in each login file is not fixed. For example, write all the information to/var/log/admin. log! [Root @ linux ~] # Vi/etc/syslog. conf *. info/var/log/admin. log if the server hard disk capacity is large enough, this is also a good strategy. [Root @ linux ~] #/Etc/init. d/syslog restart [root @ linux ~] # Ll/var/log/admin. log-rw ------- 1 root 122 Oct 23/var/log/admin. log, pay attention to the permission analysis logon file dmesg [root @ linux ~] # Dmesg | more because the system has not mounted the hard disk during the boot process, it is impossible to directly read the data into the log file, but for the convenience of debugging, therefore, the information in the boot process should be recorded. At this time, the system will open a cell block for ram to store this data! The Boot Record File is/proc/kmsg! Almost all the core information can be viewed using dmesg. For example, if you want to know whether the NIC has been caught at startup, dmesg | grep 'eth '! Last [root @ linux ~] # Last-n number [root @ linux ~] # The last-f filename parameter-n: last reads the data for this month. If the data volume is too large, you can use-n to strictly limit the number of records displayed. For example, 20 pieces of data: last-n 20 or last-20. -F: Read/var/log/wtmp by default at last, but we can read different login file information through-f! For example, read the last month's data and only read 5 pieces of data [root @ linux ~] # Last-n 5-f/var/log/wtmp.1dmtsai2 pts/2 Mon Oct 24-() dmtsai2 work: 0 work Mon Oct 24 gone-no logoutdmtsai2 work: 0 work Mon Oct 24-() dmtsai2 pts/2 Mon Oct 24-() dmtsai2 work: 0 work Mon Oct 24) [root @ linux ~] # Lastloglastlog only reads the information in/var/log/lastlog. It displays the last logon time of each account in all accounts on the current system! Linux System LOG File introduction! UNIX network administrators rely mainly on system logs to obtain intrusion traces. of course, there are traces of third-party tools recording intrusion into the system. UNIX systems store LOG files in the following common locations: /usr/adm-earlier versions of UNIX/var/adm-new versions use this location/var/log-some versions of Solaris, linux BSD, free BSD uses this location/etc-most UNIX versions put utmp here, some also put wtmp here, syslog. conf files below vary depending on your directory: acct or pacct -- Record access_log for the command used by each user -- mainly when the server runs ncsa httpd, record the sites connected to your server aculog -- save the MODEMS record lastlog you dialed -- Record the recent LOGIN record of the user and the initial destination of each user, sometimes it is the final failure LOGIN record, when a user logs on to the u Nix system. The registration program searches for the user's uid in the lastlog file. If the program finds the user's uid, unix displays the Last login time and tty (terminal number) loginlog -- Record some abnormal LOGIN records messages -- Record records output to the system console, in addition, syslogs are used to generate security records, which record some sulogs that use UCP system attempts to enter the restricted range, and record the records that use the su command. it is usually in/var/adm/sulog. if you use the su command on the machine, do not forget to clear it. utmp-records all users currently logged on to the system. This file is constantly changing as the user enters and leaves the system. it also keeps a long history for users in the system. utmp logs are usually stored in the/var/adm/utmp directory. you can use the w and who commands to view the file. Other commands can also access the file. for example, finger root can be used. currently, utmp generally has utmpx files as log records.. Utmpx -- UTMP extension wtmp -- records user logon and exit events. it is similar to the utmp log file, but it increases with the increase in the number of logins, and some system ftp access is also recorded in this file, it also records the normal system exit time and can be accessed using the ac and last commands. syslog: The most important log file. You can use the syslogd daemon to obtain log information. Generally, you can view/etc/syslog. conf. we can know what syslog records. lack of time, it transmits most of the messages to/var/adm/message. /dev/log -- a UNIX domain socket, message/dev/klog generated by processes running on the local machine -- port 514 of a device that receives messages from the UNIX kernel -- an INTERNET socket, receives syslog messages from other machines over UDP. Uucp-the recorded UUCP information can be updated by a local UCP activity or modified by an action initiated by a remote site. The information includes sent and received calls, sent requests, and senders, sending time and sending host lpd-errs -- logs for processing Printer fault information ftp logs -- execute ftpd with the-l option to obtain the logging function httpd logs -- the HTTPD server records every WEB access record history log-this file stores the vold record of the user's recent command input. log -- Record the error records encountered when using external media ==================== other types of log files-= ================================= some types of LOG files do not have specific titles, but starting with a specific flag, you can find the following flag in the front header. This generally indicates that this is a LOG file and you can edit it: xfer -- indicates an attempt to transfer a prohibited file. rexe -- indicates that the attempt to execute an unsupported command also has Many other types of LOG files exist, mainly caused by third-party software, or even the fucking network administrator sets an "eye" on his system, therefore, you need to take a closer look at what you think may be a LOG file. Many administrators prefer to put LOG files in the same directory for management. Therefore, you need to check whether other LOG files are stored in the directory where the LOG files you find are located. If yes, you know how to do this. Another thing you should pay attention to is the file related to the LOG user MAIL. This file name can be varied, or sometimes it is part of the syslog file. You need to know the information recorded by syslog. You can view syslog. conf. The directory of this file is in/etc. We usually view syslog. conf file to view the log configuration. example: cat/etc/syslog. in conf, the sunos operating system is connected to/var/log and/var/adm, And the/usr/adm is/var/adm. under/var/log and/var/run, the redhat is a log sample in sun os5.7. in addition, various shells also record the history of commands used by users. It uses files in the user's main directory to record the history of these commands. Generally, the file name is. sh_history (ksh ),. history (csh), or. bash_history (bash. # Ls/var/admacct log messages.1 passwd sulog vold. logaculog messages messages.2 sa utmp wtmplastlog messages.0 messages.3 spellhist utmpx wtmpx # ls/var/logauthlog syslog syslog.1 syslog.3sysidconfig. log syslog.0 syslog.2 syslog.4 contains the log sample in redhat9.0. # ls/var/logboot. log dmesg messages.2 secure uuucpboot. log.1 htmlaccess. log messages.3 secure.1 wtmpboot. log.2 httpd messages.4 secure.2 wtmp.1boot. l Og.3 lastlog netconf. log secure.3 xferlogboot. log.4 mailllog netconf. log.1 secure.4 xferlog.1cron maillog netconf. log.2 sendmail. st xferlog.2cron. 1 maillog.1 netconf. log.3 spooler xferlog.3cron. 2 maillog.2 netconf. log.4 spooler.1 xferlog.4cron. 3 maillog.3 news spooler.2cron. 4 maillog.4 normal. log spooler.3daily. log messages realtime. log spooler.4daily. sh messages.1 samba transfer. log # ls/var/run Atd. pid gpm. pid klogd. pid random-seed treemenu. cachecrond. pid identd. pid netreport runlevel. dir utmpftp. pids-all inetd. pid news syslogd. pid is usually the log we want to clear is lastlogutmp (utmpx) wtmp (wtmpx) messagessyslogsulog. Generally, we just need to erase the above logs. next I will talk about the relevant information and clearing methods of the above logs. for more detailed information and other logs, please view the relevant information. I have already made a simple statement on the log function above. What are the logs recorded in these log files? The following is an example of follow me: SunOS 5.7 login: gaoPassword: No directory! Logging in with home =/Last login: Sun Feb 4 22:18:25 from 219.31.36.7Sun Microsystems Inc. sunOS 5.7 Generic October 1998 $ then register the program to update the lastlog file with the new Logon Time and TTY information, and the program will update utmp wtmp. file. shell record :. sh_history (ksh ),. history (csh), or. bash_history (bash) is the history of shell execution. record the commands executed by the user. it usually exists in the user's home directory. don't forget to go to the root directory. 1. logs are all text files. the most stupid way is to use a text editor to edit log files. delete related records. to wipe footprints and hide yourself. for example, vi is used, but it is very stupid to do so. it is too troublesome and the workload is too large. 2. use rm-f to delete logs. for example, rm-f/usr/adm/lastlog is stupid. it is easier for administrators to detect intrusions. however, it is better to protect yourself. it can be used on less important machines. 3. clear with> er. for example: cat>/usr/log/lastlog-> here, enter what you want to write. it's better to pretend to be something or not. ^ d-> here ^ d is to press ctrl + d. #.