Real record Linux virus causes bandwidth to run out of the process

Case description

The morning received an IDC phone call, said one of our network segment IP non-stop outsourcing, should be attacked, specifically which IP do not know, let us check.

The logical analysis and solution

First of all we have to determine which machine is the network card in the outgoing package, fortunately we have zabbix monitoring, I just one of the checks, found that there is a flow full, the problem should appear on the machine above.

650) this.width=650; "title=" qq20160111114619.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;padding-bottom:0px;padding-top:0px;padding-left:0px;margin:0px; Border-left:medium none;padding-right:0px; "alt=" wkiol1atjjkjec0xaagyrs78_8a989.jpg "src=" http://s5.51cto.com/ Wyfs02/m02/79/81/wkiol1atjjkjec0xaagyrs78_8a989.jpg "width=" 650 "/>

I log in to the machine inside, check the network card traffic, my goodness, incredibly ran this multi-flow.

650) this.width=650; "title=" qq20160111105341.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;padding-bottom:0px;padding-top:0px;padding-left:0px;margin:0px; Border-left:medium none;padding-right:0px; "alt=" wkiom1atjiqskuopaagbjctazko745.jpg "src=" http://s3.51cto.com/ Wyfs02/m00/79/82/wkiom1atjiqskuopaagbjctazko745.jpg "width=" 650 "/>

This machine is mainly running a Tomcat Web service and Oracle database, the problem should not appear in the Web services and databases above, I checked the Web log, no anomalies found, view the database is also normal, there is no error log, view the system log, Also did not see anything unusual, but the system log is cleared, I quickly check the current running process, to see if there is any abnormal process, a view, sure enough to find a few abnormal process, not carefully see it really does not look out, these processes are not normal.

650) this.width=650; "title=" qq20160111105525.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;padding-bottom:0px;padding-top:0px;padding-left:0px;margin:0px; Border-left:medium none;padding-right:0px; "alt=" wkiom1atjubtjnjnaamiwbl8wo8464.jpg "src=" http://s1.51cto.com/ Wyfs02/m02/79/82/wkiom1atjubtjnjnaamiwbl8wo8464.jpg "width=" 650 "/>

What is this process, I ps-ef every time is not the same, has been changing, process number one has been in the change, I want to see the process open what file is OK, at the moment, think of here, I suddenly realize that this should be some sub-process, by a master process to manage, so look at these sub-processes is useless, Even if I kill them, there will be a new generation, the Catch thief first, we go to find the main process, I use the top D1 real-time view of the process using resources, to see if there is an abnormal process to occupy CPU memory and other resources, found a strange process, usually not seen. This should be the Trojan master process we are looking for.

650) this.width=650; "title=" qq20160111110002.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;float:none;padding-bottom:0px;padding-top:0px;padding-left:0px; Margin:0px;border-left:medium none;padding-right:0px; "alt=" wkiol1atkayhrwwnaah6n-9vj9c172.jpg "src=" http:// S4.51cto.com/wyfs02/m01/79/81/wkiol1atkayhrwwnaah6n-9vj9c172.jpg "width=" 650 "/>

I try to kill this process, killall-9 Ueksinzina, but after killing Ps-ef to see or have those sub-processes, didn't kill? Again top D1 view, found there is a other main process, it seems to kill is not killed, if so easy to kill is not a Trojan horse.

650) this.width=650; "title=" qq20160111110624.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;float:none;padding-bottom:0px;padding-top:0px;padding-left:0px; Margin:0px;border-left:medium none;padding-right:0px; "alt=" wkiom1atj9-bkvzzaaf_awqtcq8107.jpg "src=" http:// S4.51cto.com/wyfs02/m01/79/82/wkiom1atj9-bkvzzaaf_awqtcq8107.jpg "width=" 650 "/>

Let's see what he really is, "which obgqtvdunq" found this command under/usr/bin, after several kills and then re-generated under the/usr/bin directory, think of what program should be listening to the status of this process may also have any scheduled tasks, The discovery process died in the re-execution, I looked at the current idea of the/etc/crontab timed tasks and/etc/init.d startup scripts, both found to be problematic.

Can see there is a scheduled task gcc4.sh, this is not our set, look at the content is more strange, this should be the listener is dead and then started, we have to delete the relevant configuration, and delete/lib/libudev4.so.

650) this.width=650; "title=" qq20160111113648.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;float:none;padding-bottom:0px;padding-top:0px;padding-left:0px; Margin:0px;border-left:medium none;padding-right:0px; "alt=" wkiol1atnukbypf6aadwle0requ493.jpg "src=" http:// S2.51cto.com/wyfs02/m00/79/81/wkiol1atnukbypf6aadwle0requ493.jpg "width=" 650 "/>

This file was also found under the/etc/init.d/directory.

650) this.width=650; "title=" qq20160111111029.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;float:none;padding-bottom:0px;padding-top:0px;padding-left:0px; Margin:0px;border-left:medium none;padding-right:0px; "alt=" wkiom1atnroxw_edaaictxycr80284.jpg "src=" http:// S4.51cto.com/wyfs02/m02/79/82/wkiom1atnroxw_edaaictxycr80284.jpg "width=" 650 "/>

Inside the content is the boot start information, this we also to delete.

650) this.width=650; "title=" qq20160111111054.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;float:none;padding-bottom:0px;padding-top:0px;padding-left:0px; Margin:0px;border-left:medium none;padding-right:0px; "alt=" wkiom1atnrpwqm-qaacax6gdhq8925.jpg "src=" http:// S2.51cto.com/wyfs02/m01/79/82/wkiom1atnrpwqm-qaacax6gdhq8925.jpg "width=" 650 "/>

The above two is a boot on the start of the Trojan Horse, a Trojan horse program died after the start of the Trojan, but at present we kill the Trojan horse did not die, but immediately change the name switch to another program file run, so we directly kill is no use, Our goal is to prevent the new program file generation, first we cancel the program execution permissions and the program files into the directory/usr/bin directory lock.

chmod 000/usr/bin/obgqtvdunqchattr +i/usr/bin

Then we kill the process "killall-9 obgqtvdunq", and then we look at the/etc/init.d/directory, we see that he generated a new process, and the directory changes to the/bin directory, and the same as above, cancel the execution of permissions and lock the/bin directory, Don't let him build it here, kill it and then look at it and then create a new file, this time he is not in the environment variable directory, in/TMP, we have the/tmp directory locked, and then end the process.

650) this.width=650; "title=" qq20160111111943.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;padding-bottom:0px;padding-top:0px;padding-left:0px;margin:0px; Border-left:medium none;padding-right:0px; "alt=" wkiol1atoijgvyf9aacpcncglgm837.jpg "src=" http://s1.51cto.com/ Wyfs02/m00/79/81/wkiol1atoijgvyf9aacpcncglgm837.jpg "width=" 650 "/>

So far, there is no new Trojan process generation, the principle is to end the Trojan horse program, the back of the work is to clear the files produced by these directories, after I find, first clear the/ETC/INIT.D directory under the Trojan start script, and then clear/etc/rc#.d/directory below the connection file.

650) this.width=650; "title=" qq20160111112458.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;padding-bottom:0px;padding-top:0px;padding-left:0px;margin:0px; Border-left:medium none;padding-right:0px; "alt=" wkiol1atorhqyujgaaisuyq8roc792.jpg "src=" http://s2.51cto.com/ Wyfs02/m02/79/81/wkiol1atorhqyujgaaisuyq8roc792.jpg "width=" 650 "/>

Later I look at the change time of the file in/etc directory, found the SSH directory also has a newly generated file, do not know if there is a problem.

650) this.width=650; "title=" qq20160111131519.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;padding-bottom:0px;padding-top:0px;padding-left:0px;margin:0px; Border-left:medium none;padding-right:0px; "alt=" wkiol1ator-a5rbgaade2frr-km832.jpg "src=" http://s4.51cto.com/ Wyfs02/m01/79/81/wkiol1ator-a5rbgaade2frr-km832.jpg "width=" 650 "/>

Cleanup almost after we have to clean up just a few files generated, a directory clear, such as "Chattr-i/tmp", and then delete the Trojan file, and so on delete/bin,/usr/bin directory under the Trojan, to this Trojan clean up.

fast-Clean Trojan flow

If the name of the Trojan is Nshbsjdy, if top is not visible, you can view it under the/ETC/INIT.D directory.

1, first lock three directories, can not let the new Trojan file generation

chmod 000/usr/bin/nshbsjdychattr +i/usr/binchattr +i/binchattr +i/tmp

2. Delete Scheduled tasks and files and boot files

Delete Scheduled tasks and files Rm-f/etc/init.d/nshbsjdyrm-f/etc/rc#.d/Trojan Connection file

3, kill the Trojan process

Killall-9 Nshbsjdy

4, clean up the Trojan process

Chattr-i/usr/binrm-f/usr/bin/nshbsjdy

Once the processing is complete, check the above directories again, especially the most recently modified files under the/etc directory.

5, if it is a rootkit trojan, you can use the following software to check

Software chkrootkit: Software rkhunter:

The installation is very simple, I use rkhunter simple check, did not find any major problem, but this does not mean that there is no problem, because our detection command is also dependent on some system commands, if the system command is infected that is not detected, preferably a system of command backup a check, No more, just back up the data reload.

650) this.width=650; "title=" qq20160113093120.jpg "style=" Border-top:medium none;border-right:medium none; Vertical-align:top;border-bottom:medium none;padding-bottom:0px;padding-top:0px;padding-left:0px;margin:0px; Border-left:medium none;padding-right:0px; "alt=" wkiol1avqpwts0kyaafvquwvc-8650.jpg "src=" http://s3.51cto.com/ Wyfs02/m01/79/94/wkiol1avqpwts0kyaafvquwvc-8650.jpg "width=" 650 "/>

This article is from the "it--Brother" blog, please make sure to keep this source http://402753795.blog.51cto.com/10788998/1744285

Real record Linux virus causes bandwidth to run out of the process