UNIX Emergency Response Security Strategy

Remember: the operations on the affected system may change the existing evidence or lead to the loss of sensitive information!

{Initial response }}

Objective: To obtain the loss-prone data in the system before judicial authentication and replication, and preliminarily determine the emergency situation.

====================

Create a response Toolkit

====================

We investigate that the system must execute commands with highly trusted programs, and add backups and fixes to create a toolkit. Even on non-UNIX/LINUX systems, creating a Toolkit should be the first step in response.

First, we need to compile the tools required during the response on the System of the corresponding architecture, and the compilation program needs to consider the system compatibility issue.

We usually need the following tools:

Ls dd des file pkginfo

Find icat lsof md5sum nc

Netstat pcat perl ps strace

Strings truss df vi

Cat kstat ifconfig chkrootkit

More gzip last w rm

Script bash modinfo lsmod

You can add a Toolkit as needed. However, a toolkit can only be used to complete the work on a specific platform, putting the tool compiled on multiple platforms into the same toolkit will become messy. When creating a response toolkit on Linux, you can use the Cstatic parameter of gcc to compile the source code, or use ldd to check the dynamic Connection Library and create a library file directory on the storage medium of the response toolkit, COPY Copies of dynamic connection libraries required by all tools, and set environment variables. This process is similar to creating a Linux boot disk.

================

Easy to lose data

================

Data that is easy to lose includes: the currently opened socket, process list, RAM content, and locations of non-linked files. * Unix features: unix allows a process to be deleted when it is being executed! A non-linked file is a file marked as deleted when the process accessing the file is aborted. When the system is shut down normally or suddenly shut down abnormally), all files marked as deleted will disappear. Therefore, you cannot shut down a file marked as deleted!

========================

Execute a trusted shell

========================

Use our own response toolkit to load the file system of the media, mount Ct auto/dev/sda1/mnt/usb or

Mount Ct iso9660/dev/cdrom/mnt/cdrom press Ctrl + Alt + F1 ~ F6. log on to the console as root. Be sure to distinguish the commands in the original environment variables from the command set with the same name as the current response toolkit to prevent potential binary Trojan Horse attacks.

============================

View users logging on to the system

============================

[Root @ ay4z3ro foo] # w

19:50:48 up 43 min, 2 users, load average: 0.00, 0.00, 0.00

User tty login @ IDLE JCPU PCPU WHAT

Root: 0 :08? Xdm? 11.10 s 0.43 s gnome-session

Root pts/0 1.00 s 0.21 s 0.01 s w

The output header line shows the current system time, the system running time, the current number of logged-in users, the average system load in the last 1 minute, 5 minutes, and 15 minutes. The USER field displays the username currently logged on. The TTY field shows the session control terminal. tty indicates logging in from the console. pts/typ indicates a network connection, because X is a C/S application, therefore, the shell window I opened in GNOME is displayed as pts. If you do not log on FROM the local machine, the output also contains the FROM field, indicating the domain name or IP address of the source address of the session. LOGIN @ displays the local start time of the connection. The IDLE field shows the length of time since the previous process was running. JCPU displays the time used by all processes associated with tty or pts. The PCPU field shows the CPU time used by the current process in the WHAT column. The WHAT column displays the processes currently running by the user.

======================

View the system process list

======================

Ps Ceaf is used in Solaris, while ps Caux is used in FreeBSD and Linux.

[Root @ ay4z3ro foo] # ps aux

User pid % CPU % MEM VSZ RSS TTY STAT START TIME COMMAND

Root 1 0.1 0.2 1356 496? S init

Root 2 0.0 0.0 0 0? SW [keventd]

Root 3 0.0 0.0 0 0? SWN [ksoftirqd_CPU0]

Root 4 0.0 0.0 0 0? SW [kswapd]

Root 5 0.0 0.0 0 0? SW [bdflush]

Root 6 0.0 0.0 0 0? SW [kupdated]

Root 7 0.0 0.0 0 0? SW <[mdrecoveryd]

Root 11 0.0 0.0 0 0? SW [kjournald]

Root 114 0.0 0.5 2108 1304? S pm devfsd/dev

Root 209 0.0 0.0 0 0? SW [khubd]

Root 338 0.0 0.0 0 0? SW [kjournald]

Rpc 620 0.0 0.2 1496 520? S [portmap]

Root 636 0.0 0.2 1452 624? S syslogd-m 0

..................... Omitted below)

The START field in the Ps command output shows the START time of the program, which is helpful for detecting the attack time. Sometimes suspicious processes can be identified only by time. In Linux, you can also use strings Cf/proc/[0-9] */cmdline to view the complete command line parameters of processes running in the system, but this is not completely credible. Because attackers do not even need to insert kernel modules, they can trick us by adding statements to the application layer encoding.

====================

LKM Rootkit Detection

====================

What's worse than the backdoor of the kernel module? Solaris, Linux, and almost all UNIX Systems Support LKM (Loadable Kernel Modules), which cannot be detected by common methods, which brings great challenges to emergency response. For us, the solution is to find the lkm rootkit and familiarize ourselves with and dissect them. Sometimes, although the lkm rootkit is successfully loaded, "exception" may occur in some details of the system, or even cause the system to crash after running for a period of time. In addition, although the lkm activity is in the ring0 core State, attackers often leave traces somewhere in the system, for example, an attacker may rewrite/etc/modules to enable the system to automatically mount the embedded Kernel Backdoor after every shutdown or restart. conf or/etc/rc. local. kstat/ksec is a convenient tool for detecting lkm. The former is used in Linux, and the latter is used in * BSD.

[Root @ ay4z3ro kstat] #./kstat

Usage:./kstat [-I iff] [-P] [-p pid] [-M] [-m addr] [-s]

-I iff may be specified as "all" or as name (e.g. eth0)

Displays info about the queried interface

-P displays all processes

-P pid is the process id of the queried task

-M displays the kernel "s LKMs" linked list

-M addr is the hex address of the queried module

Displays info about the module to be found at addr

-S displays info about the system cballs "table

Among them, the-s parameter is the most useful. It displays information about the system call entry and can detect the most popular kernel backdoors knark and adore on the market, however, theoretically, he cannot detect all lkm rootkits. kstat/ksec site: http://www.s0ftpj.org is actually familiar with kernel attacks know that the way Kstat simply checks sys_call_table [] has now been completely surpassed by the attacked side.

Now madsys, The Forum Security edition moderator, has an article on Phrack61:

Finding hidden kernel modules (the extrem way) -- link:

Http://www.linuxforum.net/forum/gshowflat.php? Cat = & Board = security & Number = 434871 & page = 0 & view = collapsed & sb = 5 & o = all & fpart =

======================================

Detect open ports and associated processes

======================================

[Root @ ay4z3ro foo] # netstat Canp

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

Tcp 0 0 0.0.0.0: 111 0.0.0.0: * LISTEN 620/

Tcp 0 0 0.0.0.0: 6000 0.0.0.0: * LISTEN 908/X

Tcp 0 0 0.0.0.0: 22 0.0.0.0: * LISTEN 880/sshd

Udp 0 0 0.0.0.0: 111 0.0.0.0: * 620/

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags Type State I-Node PID/Program name Path

Unix 2 [ACC] stream listening 2753 756 // tmp/. font-unix/FS-1

...... Omitted below)

On Solaris, HP-UX, AIX, FreeBSD, Linux, lsof tools can be used to list all running processes and their open file descriptors, including regular files, library files, directories, UNIX streams, socket. If you only want to display the network socket process:

[Root @ ay4z3ro foo] # lsof Ci

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

Portmap 620 rpc 3u IPv4 2598 UDP *: sunrpc

Portmap 620 rpc 4u IPv4 2609 TCP *: sunrpc (LISTEN)

Sshd 880 root 3u IPv4 2885 TCP *: ssh (LISTEN)

X 908 root 1u IPv4 2945 TCP *: x11 (LISTEN)

Special attention should be paid to the strange process and the opened original socket.

======================================

Find whether an illegal sniffer is running in the system

======================================

To achieve this, we need to check whether the NIC is in promiscuous mode: [root @ ay4z3ro foo] # The ifconfig Ci eth0 grep PROMISC flag does not appear on all * NIX. You can use the lsof + ps command to determine whether the system is running a sniffer. Or use a third-party tool, such as AntiSniff.

========================

Check/proc file system

========================

What makes sense in the/proc/$ PID/directory is: exe link, fd subdirectory, and cmdline file.

[Root @ ay4z3ro 880] # ls-al

Total 0

Dr-xr-x 3 root 0 Sep 20 :53 ./

Dr-xr-x 62 root 0 Sep 20 :07 ./

-R -- 1 root 0 Sep 20 19:54 binfmt

-R -- 1 root 0 Sep 20 :54 then line

Lrwxrwxrwx 1 root 0 Sep 20 :54 cwd-> //

-R -------- 1 root 0 Sep 20 :54 environ

Lrwxrwxrwx 1 root 0 Sep 20 :54 exe->/usr/sbin/sshd *

Dr-x ------ 2 root 0 Sep 20 :54 fd/

-R -- 1 root 0 Sep 20 :54 maps

-Rw ------- 1 root 0 Sep 20 :54 mem

-R -- 1 root 0 Sep 20 19:54 mounts

Lrwxrwxrwx 1 root 0 Sep 20 :54 root-> //

-R -- 1 root 0 Sep 20 :54 stat

-R -- 1 root 0 Sep 20 :54 statm

-R -- 1 root 0 Sep 20 19:54 status

The Exe link allows us to restore deleted files as long as these files are still running. To back up a deleted executable file, you only need to use the cp command to create a copy on the file system. Check the fd subdirectory to identify all files opened by the process. If you know something about programming in unix, you can easily find out whether to write a file or open a network connection. The contents of the cmdline file are the complete command lines of the process. The following statement is an attacker's deception, strcpy (argv [0], "any_string"); this file shows an illusion. Even so, we still need to check this file.

========================================

Obtain the creation, modification, and access time of all files.

========================================

Ls CalRu>/mnt/usb/access

Ls CalRc>/mnt/usb/modification

Ls CalR>/mnt/usb/creation

================

Obtain system logs

================

Most UNIX logs are stored in the/var/log and/var/adm directories. The specific locations of logs of Various UNIX-derived systems are different. Before that, it is necessary to know the log storage location for a specific system. Important binary log files:

Utmp, which can be accessed using w tools;

Wtmp, accessed with the last tool;

Use the lastlog tool to access lastlog;

Process accounting logs, which can be accessed using the astcomm Tool

Common ASCII text log files:

Apache log --/var/log/httpd/access_log;

Ftp Log-xferlog;

Command history file;

/Var/log/messages;

Check the configuration files of/etc/syslog. conf and other daemon to locate other logs.

======================

Obtain important configuration files

======================

Check the configuration files for backdoor locations, unauthorized trust relationships, and unauthorized user IDs. /Etc/passwd to Find Unauthorized user accounts and permissions. Elementary intruders will add users with uid = 0, and some will change a humble ordinary account without a shell in the system into a user who can log on to obtain shell to execute commands, then he can get rootshell through a suid-bit ksh or other backdoors placed locally.

/Etc/shadow ensures that every user has password authentication. Of course, it is very easy for an attacker to add an md5 hash to his account.

/Etc/groups.

/Etc/hosts to list local DNS entries.

/Etc/hosts. equiv: Check the trust relationship.

~ /. Rhosts, check the trust relationship based on the user, "++" is a very abuse of backdoor and we believe everyone knows.

/Etc/hosts. allow &/etc/hosts. deny check tcpwrapper rules.

/Etc/rc *, check the Startup File.

Crontab file to list scheduled events.

/Etc/inetd. conf to list the services monitored by the port.

==============

Dump system RAM

==============

It is mainly to transfer the/proc/kmem or/proc/kcore file from the system, which contains the system ram content in an discontinuous manner.

{In-depth investigation }}

================

Check System Logs

================

UNIX has many logs, which provide important clues for emergency response. Most log files are in the public directory, usually/var/log,/usr/adm,/var/adm, and some logs are in the/etc directory that is not allowed to be accessed. For more information, see the operating system documentation. The syslogd daemon provides very powerful log functions. For example, when a kernel module is loaded for registration, its configuration file is/etc/syslog. conf,

Generally, it provides the most useful logs: messages, secure, syslog. in syslog. each line in conf contains three fields: the facility field indicates the subsystem that generates the log file, the priority field indicates the Severity Level of the event, and the action field indicates how to record the log, it supports remote network recording. TCP wrapper logs also use syslogs, which may contain telnet, ssh, ftp, and other remote logon information. In these logs

There are many valuable entries: the logon attempt date, host name, accessed service type, and source IP address. Other network logs, such as web, ftp, and SQL, usually provide more detailed information. Apache default logs are stored in/usr/local/apache/logs. The most useful logs are access_log, and ssl_request_log and ssl_engine_log can also provide valuable information. It may contain scan records before the attack.

The Su command log records the operation of executing the su command every time: time, date, success, terminal device, user ID. Some UNIX have separate su logs, and some are saved in syslog.

Log on to the user: The utmp or wtmp file stores information about the user currently logged on to the system. This file varies with different UNIX versions,

The name and storage location are different. The basic information stored is the user name, used to log on to the terminal and the logon time. Files are stored in binary format.

To query the utmp and wtmp files, use the appropriate client, such as w, who, finger, and last.. login entries with unknown username and failure.

Cron logs record the content of scheduled jobs, usually in/var/log/cron or a file named cron in the default log directory. Process accounting. If the system has an acct or pacct log file, you can use the lastcomm or acctcom command to view it. The log is a binary file. Shell history: [root @ ay4z3ro foo] # less ~ /. Bash_history

If. bash_history is linked to the/dev/null file, or the value of $ HISTFILE and $ HISTFILESIZE in the environment variable is 0, someone is definitely not active.

Most intruders modify or delete logs. Although theoretically there is almost no trace in addition to planting lkm rootkit, in actual intrusion, the aftermath is actually a huge project, not only does it rely on intruders to be familiar with the system, but it is easy to neglect when dealing with too much tedious content. For example, when you get the rootshell, unset HISTFILESIZE is displayed. When you exit, you forget to restore the rootshell and leave a trace. There are many examples of this type. The log clearing tool is dead. It only clears pre-defined projects. Although you can also modify the source code, it still cannot be changed at random. The safest way is manual work, which increases the burden on intruders. Out of laziness, the lack of understanding of the system or various reasons often leave something valuable to us. Therefore, checking logs is very important for emergency response.

====================

Execute keyword search

====================

Keyword Search is a part of this process regardless of the operating system for emergency response. For a specific event, there may be some IDs. phrase is closely related to this event. You can find more information by executing a keyword search. The keyword can be a long ASCII string, including the attacker's backdoor password, user name, MAC address, or IP address.

For example, search for all files in the case of the ay4z3ro string in the file system:

[Root @ ay4z3ro foo] # The grep Cr Ci ay4z3ro/strings command is used to display printable characters in the file. For example, srings/bin/login is used to display unencrypted plaintext passwords in the login backdoor, encoded or encrypted hash ). The Find command is used to Find any file name that matches a regular expression. For example, search for "…" in the entire file system "..." File or directory:

[Root @ ay4z3ro foo] # find/-name "\ .." Cprint

In addition, the find command can match the following features: Access time modification, file owner, file string, and file name string. The "-exec" option of the "Find" command is used in combination with grep and strings to reflect the genius of unix, which makes it very convenient to start a job.

======================

Determine the emergency time

======================

* If IDS exists, ensure that the IDS system time is consistent with the affected system time.

* You may be pleasantly surprised to find the files created and modified before and after emergencies in the system.

======================================

Restore deleted files and data

======================================

This process requires the operator to have a certain understanding of the unix/linux File System architecture. I am not going to introduce the data structure of the file system and the implementation of its code here, but to briefly explain it, deleting a file only sets the data and pointer of the inode node to 0, and does not delete the data block pointed to by the node before the next data overwrite. To restore the deleted file, use the inode node information to recreate the file size and data block list. Find the file node information:

[Root @ ay4z3ro foo] # ls Ci/tmp/x. d

82241/tmp/x. d

Indicates that/tmp/x. d is located at the 82241 node.

TCT (The Coroner "s Toolkit) is a very useful Toolkit, where icat can view all The files on a specified node.

[Root @ ay4z3ro TCT] #./icat/dev/hda5 82241.

If the program is still running, you can use the lsof command to reference the NODE column to find the NODE information.

To restore a file, you only need:

[Root @ ay4z3ro TCT] #./icat/dev/hda5 NODE> some. recovery

There are many useful tools in TCT, for example:

[Root @ ay4z3ro TCT] #./ils CA/dev/hda5 grep "501"

The command line above finds all deleted files related to UID = 501 users.

TCT link: http://www.fish.com/forensics/

================

Check special files

================

SUID and SGID files:

---------------

[Root @ ay4z3ro/] # find/-type f (-perm-04000-o-perm-02000)-exec ls-lg {}

Find the S-bit programs that are not originally included in the S-bit, unusual, or placed in a strange place. Renaming/bin/ksh and placing it in the/tmp (777) directory is a backdoor.

Uncommon and hidden files and directories:

---------------------------

Files starting with "." in UNIX systems are hidden. If the "-a" parameter is not added to ls, it will not appear in the file list.

Renaming a directory with hidden characters is a hacker technique. For example, some file names are as follows: ". ^ T ","... "(Note that there is a space behind the three points )......

This can fool many system administrators.

What should we do? As follows:

[Root @ ay4z3ro foo] # ls Cal "cat Ctve"

The cat command parameter allows it to display non-printable characters, display tabs, and place a $ at each end. Therefore, the preceding directory is displayed as: ". ^ T $ ","... $ ".

To enter the previous directory, press CTRL + V Before T, instead of using the ^ symbol. Enter the next Directory: [root @ ay4z3ro foo] # cd "… "

Configuration file:

---------

If an attacker has a very skilled system management skill and steals the bar, it is easy to do something in the configuration file. For example, you can modify or add/etc/services and/etc/inetd. conf. However, for skilled system administrators, such backdoors are easy to find. There are also some by-products of exploit, such as the remote overflow of the Sadmind RPC of Solaris2.6. The default attack program is in the/etc/inetd. conf

Add the counterfeit ingreslock service to bind/bin/sh to port 1524. Attackers with poor levels usually do not pay attention to this issue.

STARTUP file:

----------

The preceding inetd. conf file contains cron files and directories/var/spool/cron and/usr/spool/cron To save cron jobs for different users. The files in this directory are named after the user account, and the tasks in the directory run with the user privilege. The root file in this directory should be our concern. Some people like to start a Trojan horse to run a bindshell in the middle of the night, and then shut down the opened port in a few hours.

/Etc/rc * is a script that runs automatically upon startup and is often exploited by attackers. The remaining User Startup files, such as login, profile,. bashrc,. cshrc, And. exrc, may be inserted with the trojan statement, which has been used in the intrusion events of apache.org.

/Tmp directory:

-------------

The default permission of the/tmp directory is 777, which is often used by attackers. Many hacking tools also use this directory to store intermediate files. If the attacker is not careful,

The tail will be left here.

======================

Check user accounts and groups

======================

Some accounts are set for the system, and there is no shell or login, attackers may exploit this. People who have just learned about UNIX intrusion may look like this:

Echo "aya: x: 0: 0: intruder !! : //:/Bin/sh ">/etc/passwd. Even worse, the passwd file is damaged because one"> "is missing, so you cannot log on to the file ?, I have seen such a guy. To be honest, I did it myself, but it happened on my own Linux, so it didn't make a big mistake. Sophisticated users will not do this. They may pick out an inconspicuous one from a bunch of users and take it as their own. After obtaining the shell through remote legal login, the local server will be upgraded to root. I believe this problem is a good solution for us.

================

Illegal process identification

================

Check the binary files related to the listening service and running processes to view/etc/inetd. during conf, you may find that the valid service is listening on the valid port, but the binary file of the process may have been replaced. Therefore, make sure that the running rootkit (lrk4, lrk5 ......)

[Root @ ay4z3ro tool] #./chkrootkit is a tool used to check integrity. If you have used tripwire before, you can use tripwire for verification. Or use the MD5 checksum function of rpm.

================

Survey System Vulnerabilities

================

Check the versions of various services, applications, kernels, and patches, check the list of known vulnerabilities on bugtraq, find system vulnerabilities, and discover potential and possibly ignored vulnerabilities from the front. This requires the responder to have the skills related to a professional intruder! Trying to act as an intruder can also find out how the other party enters the system.

================

Analyze trust relationships

================

First, the trust relationship in UNIX was once a weakness of attacks. Secondly, if the relationship is exploited, the scope of the affected system will be expanded. At this time, the trusted system is considered insecure and included in the scope of the response object.

================

Hacker analysis tools

================

If, fortunately, the intruders leave behind or we use some clever way to restore the tools and code used by the intruders during the activity, we can further analyze the code. If it is the source code, it can be read directly. Of course, it is the best source code for LKM RootKit, haha ). The unskillful guy even runs the original tool, such as sniffit, without changing the name. This is a better way to deal with it.

If we get a copy of the binary file of a running process, we can use gdb and other debuggers for disassembly and tracking debugging. However, if a very good attacker compiles his program like this:

[Root @ ay4z3ro edevil] # gcc CO4 edevil. c Co edevil uses the-O4 parameter to optimize the compilation, which reduces the instruction and removes the symbol information in the binary file with strip:

[Root @ ay4z3ro edevil] # strip./edevil makes our work very difficult. The File command can display the File type information, whether the File has been strip, and so on. The Strings command can be used to display ASCII Strings in executable files, such as a line controlled by the printf statement in the Local Buffer Overflow exploit, messages that are processed incorrectly, the returned information of the default-h parameter. In addition, you may also obtain functions, variable names, file names used before compilation, compiler versions for creating the file, etc. By using these keywords for online search, you may find the source code of the tool.

Similarly, we can dynamically analyze binary files and use strace tools to track system calls. Strace displays the file access, network access, memory access, and many other system call information generated during file execution. Generally, by observing critical system calls, we can determine what the program has done. It is also possible to reconstruct the running scenario of the file. Strace provides us with great convenience. Throughout the response process, we can also use it to do many things.

{Remarks }}

Emergency response does not need to follow a fixed pattern. A novel idea can be used appropriately. If a competitor is an extremely high-level attacker, the strategy should be changed as appropriate. For the sake of security, finally, reinstalling the system may also be necessary.

To the broad masses of "machine-related people:

Even if some people think that their intrusion skills are good, in fact, the defense side is always more or less out of your expectation, a little sloppy may embarrass you. Even if you think you have cleared all the logs on the target system, you may not be able to "process" the logs on the front-end routers or IDS or Firewall, or even the trace records you have not noticed, for your own security, it is necessary to find a fast line and use multiple springboards to launch a real attack. You cannot expose yourself from "start" to "end!