Post-invasion process for Linux systems

Background--

Operating system: ubuntu12.04_x64

Operating business: Corporate business Systems, crawlers, data queues.

Server hosted in the field room.

Fault causes-

Suddenly, frequently received a group of server ping Monitoring unreachable mail, quickly log on to the Zabbix monitoring system to view traffic status.

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/7F/92/wKioL1cjFOTyIO2sAABkSBOorW8950.png "title=" 1.png " alt= "Wkiol1cjfotyio2saabksboorw8950.png"/>

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M01/7F/92/wKioL1cjFPGSppbDAADd32iT3AI593.png "title=" 2.png " alt= "Wkiol1cjfpgsppbdaadd32it3ai593.png"/>

Visible traffic has reached about 800M, certainly not normal, immediately try SSH login system, unfortunately, this situation is difficult to log on the system operation.

What should we do?

Troubleshooting--

The first reaction is to cut off the external network immediately, through the intranet connection to view. However, the traffic will disappear, but it is also difficult to find the source of the attack.

So contact the computer room to help solve, authorized computer room technology login to the system, first through the W command to see if there is an abnormal user login, and then use the tool to find out that the connection occupies a large amount of traffic, I used the Iftop tool.

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/7F/92/wKioL1cjFT6T1dttAAOY0JDSTUc606.png "title=" 3.png " alt= "Wkiol1cjft6t1dttaaoy0jdstuc606.png"/>

Computer room technology gave me a picture, see the Local has been the HTTP way to 104.31.225.6 this IP send packets, and continue.

OK, let's try to shield this IP first:

Iptables–a output–d 104.31.225.6–j DROP

Whoa, shove! The miracle appeared, the flow went down, can connect normally, the face gradually smile.

After a while, the unfortunate thing happened, traffic came up again, rub! What a situation! The mood suddenly tense up.

and quickly contact the computer room technology, the last operation.

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/7F/92/wKioL1cjFXDzrYVgAATklwVCE3Q320.png "title=" 4.png " alt= "Wkiol1cjfxdzryvgaatklwvce3q320.png"/>

Dumbfounded, the purpose of IP has changed, this can do, it is impossible to seal it!

Calm down, carefully think about the next, the local outsourcing, the local will certainly have a program to send! How can I find this?

Start looking for Trojans--

First filter the port through the Netstat tool to see the running process ID:

Netstat–atup |grep 15773

There is nothing ah, another port to try, the same effect!

Let the computer room technology to observe the next connection state, the original is a short connection, will quickly release the port, so the connection status of the port can not be seen.

For a normal long connection, you can use lsof–i:15773 to find the PID, and then lsof–p the PID to find the open related file.

I want to be quiet.

All right! decided to first cut off the external network, the intranet SSH into the system, first find the program, walk up!

See if there are open or suspicious ports or connections through Netstat–antup first.

See if there are any suspicious processes through PS–EF.

Looked carefully, did not find suspicious.

Is it implanted a rootkit Trojan horse program, say not good, try it!

Want to determine if the system has been implanted a rootkit can use the Md5sum checksum to execute file judgment, first find a version of the operating system, get to the tool to execute the MD5 value of the file, and then get the suspicious tool to execute the file MD5 value, compare the two value is the same, if the same description of the tool is trustworthy, If it is not the same, it is likely to be replaced. In addition, the generic tool executable file size is dozens of k to hundreds of K.

In fact, I did not use the MD5 way to determine whether the tool can be trusted, because the same version of the operating system is not easy to find, slightly different, the tool may have been updated, MD5 value is different. I directly use Du–sh/bin/lsof view, found size 1.2M, obviously there is a problem.

So directly download the normal system in the Netstat, PS and other tools uploaded to the black system to use, and then replace the unusable.

Sure enough, the miracle appeared, after the implementation of the PS–EF, under the most below a few lines of suspicious procedures. Here, I would like to, unfortunately, the SSH client turned off, did not leave.

In memory, this is probably the case:

Pid/sbin/java.log

Pid/usr/bin/dpkgd/ps–ef

Pid/usr/bin/bsd-port/getty

Pid/usr/bin/.sshd

See these, feel very strange, how there will be a java.log execution of the file in the run, after looking for colleagues to verify that there is no running such, they say no, that good first kill and delete again.

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M00/7F/94/wKiom1cjFOKzFjoUAAAUIuvKFbk803.png "title=" 5.png " alt= "Wkiom1cjfokzfjouaaauiuvkfbk803.png"/>

And then how did I execute the command, PS–EF, the path of the command is not/bin/ps, causing my suspicion, immediately into this directory to view.

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M01/7F/92/wKioL1cjFbyQ_aCFAAApwXDfVN4723.png "title=" 6.png " alt= "Wkiol1cjfbyq_acfaaapwxdfvn4723.png"/>

Rub, and a few more, the preliminary judgment is that the tool has been replaced.

There is also a how to call Getty, again normal system inside the contrast process, found no this. Estimate is a hacker left, labor anger, rather than kill 100, also do not let one!

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/7F/92/wKioL1cjFdihG2q7AAAUUc4e0Fo320.png "title=" 7.png " alt= "Wkiol1cjfdihg2q7aaauuc4e0fo320.png"/>

Kill the process and delete the directory.

. sshd process? Obviously very suspicious, it is ssh back door, first kill Delete to say!

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M00/7F/92/wKioL1cjFcrSFWh7AAAQ7XfUmrs431.png "title=" 8.png " alt= "Wkiol1cjfcrsfwh7aaaq7xfumrs431.png"/>

then execute the ps–ef command look, strange, java.log process again up, there is a self-starting setting? So to the/ETC/INIT.D under the view, there is an exception script, in the normal system is not, open looked under, sure enough is the start Trojan Horse program script. Delete the script and delete the Java.log once again, no longer appears. 650) this.width=650; "src=" Http://img.baidu.com/hi/jx2/j_0069.gif "alt=" J_0069.gif "style=" font-family: ' Song body '; />

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/7F/92/wKioL1cjFe2hrmIpAAB33YwJaxI695.png "title=" 9.png " alt= "Wkiol1cjfe2hrmipaab33ywjaxi695.png"/>

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/7F/94/wKiom1cjFTDjQKW6AAAh8i07Fx8226.png "title=" 10.png "alt=" Wkiom1cjftdjqkw6aaah8i07fx8226.png "/>

Deleted/sbin/java.log file after a while, what's going on? It is estimated that Getty the ghost, the same purge, no longer automatically generated.

OK, can open the outside network, observed a will network traffic no longer soar, the mood like to see Beauty as happy! 650) this.width=650; "src=" Http://img.baidu.com/hi/jx2/j_0061.gif "alt=" J_0061.gif "/>

Blog Address: http://lizhenliang.blog.51cto.com

Summary--

Ls/usr/bin/dpkgd/#替换的工具, the system comes with a tool that is not normally in this directory and is not available

Netstat lsof PS SS

/sbin/java.log #判断是发包程序, automatically generated after deletion

/usr/bin/bsd-port #判断是自动生成java. Log or Back door program

/usr/sbin/.sshd #判断是后门程序

Trojan Code Program: Http://pan.baidu.com/s/1b3yOVW

Careful, direct execution of Java.log may cause Linux to be virtually inaccessible at a moment!

Think carefully, if there are other Trojan programs to do? If it is an XSS attack, what happens to application layer vulnerability intrusion?

In view of these problems, from our company's point of view, try not to re-install the system, the business is too complex. It's not too realistic to find the point of intrusion. Let's take one step at a glance!

Let the hacker take the opportunity to invade the reason:

1. Operation and maintenance of network security implementation of low-intensity

2. No relevant security testers, no timely detection of application layer vulnerabilities

Wait a minute...

For this attack, summed up the following protection ideas:

1. After the Linux system is installed, the firewall is enabled, allowing only trusted sources to access the specified services, removing unnecessary users, shutting down unnecessary services, and so on.

2. Collect logs, including system logs, log logs, program logs, etc., to identify potential risks in a timely manner.

3. Real-time collection for user login, including login time, password retry number and user execution command record, etc.

4. Monitor the changes of sensitive files or directories, such as/etc/passwd,/etc/shadow,/web,/tmp (generally uploading files for power), etc.

5. Process status monitoring to record and notify new or suspicious processes.

6. On-line server system, Web program process Security vulnerability Scan.

Or that sentence, there is no absolute security, only to minimize the attack surface, provide system protection ability.

Network security, starting from me! 650) this.width=650; "src=" Http://img.baidu.com/hi/jx2/j_0019.gif "alt=" J_0019.gif "/>

This article is from the "Li Zhenliang Technology Blog" blog, make sure to keep this source http://lizhenliang.blog.51cto.com/7876557/1769028

Post-invasion process for Linux systems