Linux server is hanging Horse solution (i)

Linux server is hanging Horse solution ( i )

phenomenon: The server Web site appears occasionally open, not frequent. The reason for the lookup is that using the SSH tool connection will often break down and not work.

Resolution process:

Fortunately, using a different server to ssh to another intranet IP connection, yes. First of all , it is suspected that the external network IP card problem.

How to view excessive traffic:

1 using ifstat

wget http://distfiles.macports.org/ifstat/ifstat-1.1.tar.gz

Ifstat-a join Monitoring Lo

2 using iftop to Monitor that port traffic

P can display the connection port

3 Monitor each process traffic using nethogs

Yum change rpel source

wgethttp://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

Yum Install Nethogs

Nethogs eth0

Comparison of 3 tools using the effect:

1. Ifstat

650) this.width=650, "width=" 546 "height=", "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ( "/e/u261/lang/zh-cn/images/localimage.png") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

650) this.width=650; "Width=" 547 "height=" 227 "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/images/localimage.png") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

2.iftop

650) this.width=650; "Width=" 697 "height=" 262 "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/images/localimage.png") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

3 Nethogs

D user    program                                                                                                                                                                                          dev         sent      received       

?    root     124.172.118.226:6750-14.18.143.59:80                                                                                                                                                                        0.000       0.200 KB/sec

?    root    124.172.118.226:2508-14.18.143.59:80                                                                                                                                                                        0.000       0.200 KB/sec

?    root    124.172.118.226:15291-14.18.143.59:80                                                                                                                                                                        0.000       0.200 KB/sec

?    root    124.172.118.226:52445-14.18.143.59:80                                                                                                                                                                        0.000       0.200 KB/sec

?    root    124.172.118.226:62124-14.18.143.59:80                                                                                                                                                                       0.000       0.200 KB/sec

?    root    124.172.118.226:22214-14.18.143.59:80                                                                                                                                                                       0.000       0.200 KB/sec

?    root    124.172.118.226:7915-14.18.143.59:80                                                                                                                                                                        0.000       0.200 KB/sec

?    root    124.172.118.226:51793-14.18.143.59:80                                                                                                                                                                       0.000       0.200 KB/sec

?    root    124.172.118.226:18692-14.18.143.59:80                                                                                                                                                                       0.000       0.200 KB/sec

through the above analysis, the cause of the website can not open because there is a process to send a large number of packets to an IP port, causing the server network congestion, but through the above tools, found this trojan quite cunning, can not be used in the process.

After using netstat,SS to get the Trojan through the port process failed

can use the top tool, this trojan in the large number of contracts will certainly cause the consumption of resources

There are a number of suspected two processes identified by the lock:

650) this.width=650; "width=" 634 "height=" 227 "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/images/localimage.png") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

Locate the directory that the process executes through the PS command:

/usr/bin/sshupdate-bootsystem-insserv

/tmp/guibger

the agent process flashes through continuous observation , after the Find/-name agent is used

# ll/usr/bin/bsd-port/

Total dosage 1120

-rwxr-xr-x. 1 root root 1135000 month 11:20agent

-rwxr-xr-x. 1 root root 4 month 11:20 agent.conf

-rw-r--r--. 1 root root 11:50 conf.n

-rw-r--r--. 1 root root 0 months 9 19:36 Getty

at this point, the relevant can process are found, through the test, the network congestion is deleted Sshupdate-bootsystem-insserv,guibger Two processes, network traffic immediately normal. The agent is suspected to be a communication process with hackers, to receive commands (guessing) or to monitor the process.

finding these 3 processes does not mean the end, because they can very well be powered on from the boot program, so to find their boot from the configuration file, I implemented this function through a script:

#!/bin/sh

echo >/tmp/find_init.log

function ergodic () {

Forfile in ' ls $ '

Do

if[-D $ "/" $file] # True if file exists and is a directory

Then

Ergodic$1 "/" $file

Else

Localpath=$1 "/" $file # get the full directory of files

Localname= $file # Get the name of the file

# do your own work .

Echo $path

rootkit_init= ' Cat$path | grep sshupdate | Head-n 1 '

If[-Z $rootkit _init];then

echo "Sed-i ' s# $rootkit _init# #g ' $path" >>/tmp/find_init.log

Fi

Fi

Done

}

Init_path= "/ETC/INIT.D"

Ergodic $INIT _path

Cat/tmp/find_init.log

This script function is very simple, by traversing all the files in the/ETC/INIT.D directory, using grep to search the process name keyword, the files containing these processes to find out.

The results are as follows:

Sed-i ' s#/usr/bin/sshupdate-bootsystem-insserv# #g '/ETC/INIT.D/DBSECURITYMDT

Sed-i ' s#/usr/bin/sshupdate-bootsystem-insserv# #g '/etc/init.d/insserv

There's really a self-starter configuration that quickly removes the

When you delete this Trojan command, you will encounter an issue that cannot be removed, which is simple:

Lsattr/usr/bin/sshupdate-bootsystem-insserv

View shadow permissions for a command

-------I------e-sshupdate-bootsystem-insserv

The discovery was restricted to delete operations

Chattr-i/usr/bin/sshupdate-bootsystem-insserv

Remove the shadow permissions, and then delete, complete.

Finally, the reason is hackers on the Linux server Hanging Horse, because it is convenient to develop on-line products, close the iptables, this lesson is very profound, so use iptables It is necessary to restrict the port of the server if it is possible that SELinux is turned on. Of course, it is also important to change each account password regularly!

This article is from the "Gu Shao" blog, please be sure to keep this source http://2847513.blog.51cto.com/2837513/1599726

Linux server is hanging Horse solution (i)