Introduction to emergency response tools in UNIX systems

IncidentResponseToolsForUnix, PartOne: SystemToolsbyHoltSorensonlastupdatedMarch27, 2003 Source: http://www.securityfocus.com/infocus/16792003-10-20

Original article: Incident Response Tools For Unix, Part One: System Tools
By Holt Sorenson
Last updated March 27,200 3

Translation: Refdom
Source: http://www.securityfocus.com/infocus/1679
2003-10-20

(Note: This is a basic emergency tool introduction document. Suitable for those with less experience. Limited by level. please note the shortcomings .)

The entire article is divided into three series to introduce tools useful for OpenBSD, Linux, or Solaris system emergency response and forensics. This part focuses on system tools. The second part focuses on file system tools, and the last part focuses on network tools. This article uses OpenBSD 3.2, Debian GNU/Linux 3.0 (woody), RedHat 8.0 (psyche), and Solaris 9 (aka Solaris 2.9 or SunOS 5.9.

Depending on the operating system you are using, many of the software discussed in this article may not be installed by default. In this case, obtain these tools according to the reference section below.

The Soapbox

Tools that can be used to handle intrusion threats are not discussed in this series. This article only covers the tools used after intrusion. Intrusion is usually prevented. Use technologies such as system update and installation of the latest patches, configure the system according to the minimal service, use the OS designed to be secure, and use the core patches that can reinforce the system, these techniques to prevent intrusion may prevent you from using the tools described in this series of articles.

The reader should give commands. Once attackers gain control of the system, the system should not be trusted. Most of the tools we discuss are operated on by users. A spoofed core module may not allow you to normally check the system, for example, if the system has been intruded. These spoofed modules use multiple methods to hide themselves. for example, they are designed to prevent themselves from being easily detected when the system is running. This means that you should not trust the output results of the tools you are using. This means that suspicion, research, and caution should be taken in response to intrusion events.

The tools you use to analyze the system should be trusted and not modified. Some technologies are discussed below, such as saving tools in offline read-only media, so that you can trust them more than the binary programs on the intruded systems.

You have done everything right. you can perform the following exercises and your system has been infiltrated. What should we do now?

Breaking Out the Toolbelt

Now we should first check the tools we need later. Here is not the information provided by man (1), especially because the command parameters vary with systems. Checking such information is a kind of exercise left for readers.

Vmstat-this command allows you to quickly view memory, CPU, and disk subsystems. Generally, vmstat executes a short period of time to view the trend of sub-system utilization. Vmstat often helps us know where to go when there are system problems.

Mpstat-this command is available on both Linux and Solaris. you can use it to view statistics on CPU utilization. Mpstat provides an option to view statistics of a specified CPU in a multi-processor system. Vmstat does not have this function.

Iostat-displays subsystem-related statistics more detailed than vmstat.

Sar, sa, lastcomm, and last-these are checking historical data and recent system events. Sar is a Solaris and Linux system performance analysis tool. The performance data that can be checked is similar to the display of vmstat, mpstat, and iostat. Sar data is stored for a period of time, so you can view the past information. Lastcomm can be used to display the commands recently executed by the system. These can be used in system audit. Sa can be found in * BSD and Linux, which gives users more options in system audit to collect information.

Ps-displays the processes executed by the system and their information based on the process status.

The top-display information is similar to the ps, but top can understand the CPU consumption and update the display based on the time specified by the user.

Lsof-lists opened files and displays all files currently opened by the system. Almost everything in a Unix system can be seen as a file, so lsof also shows important content in the system status.

File-determine what a file is. different file formats can be in hexadecimal format.

Readelf-displays the details of the ELF (executable link and format) header of the binary file. This content can determine the executable provided functions.

Od-output the file content in the format specified by the user. Od is helpful for viewing the original content with some explanations in the file content.

Ldd-read the content of the ELF header and display the object library on which executable files depend.

String-ASCII string in the real file. It is very useful for searching readable strings in binary files.

Find-is used to find the specified object in the file system.

Strace-This tool starts or is appended to a currently running process and displays all system calls made by this process. This can be used to determine the program running behavior and determine whether the program is suitable. Strace exists on Linux. On Solaris, it is truss. * ktrace provided by BSD can achieve similar functions.

Sudo-The administrator can give the user the ability to execute commands with the permissions of other users, rather than the user password.

Grep-used to query data in the specified mode. Grep uses matching rules.

Less-page scheduling program, used to display text by page.

In the reference section "CD-ROM and Floppy distributions" [3], some sites explain how to compile these tools for different operating systems and place them on the CD-ROM. Now, check the running system.

Luck is preparing for the opportunity.

Luck is not feasible for technology. Attackers can use many methods to compromise OS security. Vulnerabilities that threaten system security may be OS providers or applications. When the vulnerability has not been published on a large scale, or has not been known to its related providers, attackers already know in IRC that the system can be intruded. As a security administrator, you must always be outstanding in this game. However, attackers only need to perform the operation once.

However, you need to make a lot of preparations. You should have an emergency response policy. You must have a CD-ROM or floppy disk or something to store the tools needed in the checking system, instead of using the tools in the compromised system. If you are using a floppy disk and keep them in write protection mode, you can obtain excellent resources in Forensic [5.

Now go back to the machine. Record the event response date and time in the record. And record all the details and time of each step. Mount the CD-ROM or floppy disk, and use software recorded on these media, remember to save the relevant information to a safe place in a volatile order [6.

The important reason for the order of volatility is that the quantity and quantity of information related to the intrusion event will decrease rapidly as time increases. Events created by experienced attackers may lead to steep curves and faster downgrades.

Learn about yourself

You may have spent a lot of time on the system before the intrusion event. You understand the internal and external aspects of these systems, have performed a lot of documentation work, and have backed up. On the system, process audit (or * BSD system audit) is enabled, and real-time system data (sadc [8]) is performed to save system performance data. When managing logs, you may not have enough space to frequently store the data. you can send syslog data to a secure log server.

When you receive a call from the webmaster, he finds that the CPU load on a web server is very high, and the server views only a few thousand pages a day. The webmaster is sure that the server has any problems.

Vmstat and mpstat (only on the * BSD system) indicate that the CPU is consumed by one or more processes in the user space, but the memory and I/O subsystems are not in use yet. Iostat also shows that the disk system is abnormal.

$ Vmstat 1 4
Procs memory swap io system cpu
R B w swpd free buff cache si so bi bo in cs us sy id
1 0 0 376 7756 29772 0 0 7 3 570960 441 87 5 8
5 0 0 376 6728 29772 0 0 0 570960 498 0 0
6 0 0 376 7240 29772 0 0 0 570960 97 3 0
6 0 0 376 7604 29772 0 0 0 570960 97 3 0
$ Mpstat 1 4
20:51:21 CPU % user % nice % system % idle intr/s
20:51:22 all 100.00 0.00 0.00 0.00 479.00
20:51:23 all 100.00 0.00 0.00 0.00 496.00
20:51:24 all 100.00 0.00 0.00 0.00 499.00
20:51:25 all 97.00 0.00 3.00 0.00 481.00
Average: all 98.00 0.60 1.40 0.00 486.60
$ Iostat-dk 1 4
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
Dev3-0 0.00 0.00 0.01 73 1296
Dev3-0 0.00 0.00 0.00 0 0
Dev3-0 0.00 0.00 0.00 0 0
Dev3-0 0.00 0.00 0.00 0 0

Sar (* BSD sa) shows that the CPU has been used since. Lastcomm shows how many times the FTP client has been running as root at. The last Command shows the recent logon, but does not show where the root has logged in during this time period. In addition, this server uses sudo to manage root permissions. Here there are two system administrator accounts that can be logged on with root, but they have not logged on, especially near AM 3. Based on this, you decided to use the binary tool prepared on CD. Run top-d1 and it is found that the apache process occupies 100% of the CPU.

Now we use grep to check the error log of apche. This GNU tool can have more useful parameters,-, -B and-C allow you to specify the starting point of the matched row, which can be viewed before or around. Grep can use the extended pattern matching expression through-E. Check system logs. However, nothing strange is found in these logs.

Check again. Run ps-eflcyL (Solaris 9), ps-eflcym -- headers (Deb3.0, RH8.0), or ps auwxhkwvl (OBSD 3.2 ). Find the process apache, and this time there is a problem. There is only one apache process and multiple httpd processes. Httpd is actually an Apache server, because httpd is a binary file executed in the apachectl file and runs apache as a pre-derived file. Therefore, if the binary file is renamed as "apache ", there should also be multiple processes. This is the problem.

Using lsof-p [9], we found that the apache process opened a file named john. pot. You can use google to search for a password cracking tool named "John the Ripper", which puts the CPU at full capacity.

Now let's check this so-called apache program. File apache indicates that this file is executable by ELF. Perform further mining and run readelf-a apache to make sure this is an ELF executable program.