Master showdown-story about hacking of blog servers

Master showdown-story about hacking of blog servers
 
Every hero needs to confront the wall on the road to growth. either you succeed, stand on the top of the world, and gain top-level knowledge; or be beaten down by it to become one of all beings, and then get used to it.
I am no exception.
Not long ago, I had just built my own "ladder" on my server. This is the story from "ladder.
The opening night is deep, and I am still sitting on the computer, thinking about one thing: Why suddenly, the "ladder" that has always been running well, suddenly so stuck? Has it been detected and restricted by the "wall?
I am unwilling to think, this is impossible, I am hiding so well, so low-key, not me.
I took out my favorite tool: ping.
I started my detection journey.

100 packets transmitted, 40 received, 60% packet loss, time 10015ms

60% packet loss, which is too exaggerated. This network environment is a thousand times worse than the traffic environment of the imperial capital. I am thinking.
Try traceroute, another powerful tool. Only the first hop is accessible. F ** K is the world's first local area network, and the operator dares to illegally discard my detection package, this detection is fruitless.
However, when I watched my "ladder" shake in the clouds (packet loss), I decided: not in confrontation, success, and extinction.
When everything was done, the service provider of "ladder" Suddenly sent a crucial email to my favorite Gmail, as it said:

Your Linode, linode202373, has exceeded the notification threshold (5) for outbound traffic rate by averaging 138.14 Mb/s for the last 2 hours (Your ladder has been in two consecutive hours, continuous packet sending at an average speed of over 138.14 Mb/s)

Ah, I am about to become the sixth kind of keen sense of the experts. I feel that I may be mistaken for the "wall" and I may be hacked.
A burst of hunger detection made me wake up a lot. As a result, some experts found that the "light food" therapy was very effective and gave me a step away from the experts.
"I must drag my opponent out.

ssh root@myblog.me

I started to connect to my server. The first step is to start from the logon log.
This is a very clever practice:

# whoroot pts/2 2015-01-20 3:00 (xx.xx.xx.xx)

Well, I am the only one who hasn't caught the current hacker. "I don't think he will be so stupid. Let me hold him on the spot." I thought to myself.

# lastroot     pts/2        li657-42.members Tue Jan 20 03:03 - 03:41  (00:38)root     pts/1        183.37.59.101    Tue Jan 20 02:54 - 05:12  (02:18)root     pts/0        li657-42.members Tue Jan 20 02:45 - 04:51  (02:05)reboot   system boot  3.18.1-x86_64-li Tue Jan 20 02:45 - 20:30  (17:45)ruby     pts/0        119.139.89.32    Sat Jan 17 01:03 - 01:03  (00:00)ruby     pts/0        119.139.88.22    Mon Jan 12 17:05 - 19:30  (02:24)ruby     pts/0        183.39.218.147   Sat Jan 10 16:31 - 16:33  (00:02)ruby     pts/0        183.39.218.147   Sat Jan 10 16:29 - 16:29  (00:00)reboot   system boot  3.18.1-x86_64-li Sat Jan 10 16:26 - 02:44 (9+10:17)wtmp begins Sat Jan 10 16:26:21 2015

Continue to skillfully check who has logged on to the system and detected the IP source through ip138.com. The results are all my IP addresses, no one !!!
Why am I wrong? Isn't it hacked?
No. Check the logon log again.

# Grep 'sshd'/var/log/auth. logJan 18 18:56:42 localhost sshd [16157]: Failed password for invalid user from 180.150.177.103 port 39118 ssh2Jan 18 18:56:42 localhost sshd [16157]: Received disconnect from 180.150.177.103: 11: bye [preauth] Jan 18 18:59:05 localhost sshd [16194]: Connection closed by 180.150.177.103 [preauth] Jan 18 19:01:26 localhost sshd [16219]: invalid user from 180.150.177.103Jan 18 19:01:26 localhost sshd [16219]: input_userauth_request: invalid user [preauth] Jan 18 19:01:26 localhost sshd [16219]: pam_unix (sshd: auth): check pass; user unknownJan 18 19:01:26 localhost sshd [16219]: pam_unix (sshd: auth): authentication failure; logname = uid = 0 euid = 0 tty = ssh ruser = rhost = 180.150.177.103Jan 18 19:01:28 localhost sshd [16219]: failed password for invalid user from 180.150.177.103 port 45735 ssh2Jan 18 19:01:29 localhost sshd [16219]: Received disconnect from 180.150.177.103: 11: Bye [preauth] Jan 18 19:03:52 localhost sshd [16248] invalid user from 180.150.177.103Jan 18 19:03:52 localhost sshd [16248]: input_userauth_request: invalid user [preauth] Jan 18 19:03:52 localhost sshd [16248]: pam_unix (sshd: auth): check pass; user unknownJan 18 19:03:52 localhost sshd [16248]: pam_unix (sshd: auth): authentication failure; logname = uid = 0 euid = 0 tty = ssh r... (Omitted tens of thousands of rows)

When I saw so many ssh Brute-Force Logon logs, I felt a bit cool and confused on the Internet. I always had to be careful with various hidden devices in the distance. Please calm down and look for them.
After tens of minutes, except for Logon failures of various user names and passwords, no logs of Successful Logon using the password were found.
I secretly thought, "the opponent is not weak either ."
However, there is another clue that I have not checked: "The current server traffic is still high"
It's time to use the core weapons in my hands.
I found that the current idea is still very clear. I need to find out which process is causing high traffic, and then analyze whether it is a hacker-infected Trojan through its behavior?
OK, start action:
Download iftop and open the system traffic panel. After 10 seconds, the traffic panel will display the traffic correctly. My traffic peak will reach 100 mb/s. adjust some command parameters to display port and IP information.

iftop -nP

The traffic panel shows that the outbound traffic is from my server. It is sent out from Port 30157. After a few seconds, it will switch to a port in the range of 30000-50000 to continue sending. is UDP traffic. I asserted.
Sure enough, I checked the port opened through netstat-anp and found that there was no TCP status. I confirmed my assertions.
(UDP traffic can be stateless and can be quickly switched. On the contrary, you can capture the link status through tools)
However, unfortunately, these two tools do not show which process is the ghost.

ps aux

There were not many processes, and I soon finished reading them. There were two suspicious processes:

ruby      5162  0.0  5.0 286128 102200 ?       Sl   02:58   2:20 /usr/sbin/httpd -c ./init -d /home/ruby/lib/2111      3033  0.0  5.0  1  2017 ?       Sl   02:58   2:20 /tmp/freeBSD /tmp/freeBSD 1

At, I use nginx instead of apache. httpd is very suspicious here. At, freeBSD is obviously a disguise, and the process permissions are suspicious.
So far, it has been confirmed that my server has been hacked. Next, it is time to fight.
There is a suspicious directory in the process information at point 1st. In/home/ruby/lib, this is my personal directory. How can it appear in the parameters here?
Go in and see cd/home/ruby/lib.

# tree.├── 2│   └── muhrc├── config├── cron.d├── dir├── f├── h├── h.c├── init├── inst├── ips├── log├── mess├── muhrc├── restart├── run├── run2├── servers└── y

I was surprised. The sixth sense tells me that this name is used to being a hacker. This is not what I wrote. Take a closer look.

# cat y#!/bin/shif test -r /home/ruby/lib/pid; thenpid=$(cat /home/ruby/lib/pid)if $(kill -CHLD $pid >/dev/null 2>&1)thenexit 0fificd /home/ruby/libkillall -9 atd./run &>/dev/null

Oh no, a Trojan program is very eye-catching. This is a very obvious reverse connection Trojan:
As long as you open the server, it will start and automatically connect to the server specified by the hacker, report that the hacker has been online, and then wait for the command. once a command is received, the corresponding program will be run with the permissions you control. terrible.
The configuration file is as follows:

# cat muhrcnickname = "Dan";altnickname = "Dan";username = "xxx";realname = "dan e pe linode :)";password = "xxx";listenport = 41000;awayreason = "mie";servers {      "Tampa.FL.US.Undernet.org":6667,      "budapest.hu.eu.undernet.org":6667,};logging = false;channels = "#olimpia";connectcmd = "PRIVMSG x@channels.undernet.org : login ";away = "mie";norestricted = true

This is the IRC address of the reverse connection of the opponent. "This guy is really awesome." I thought, "He knows, I cannot track such an IRC address ."
Now, the fact of being hacked has been confirmed, but I am more worried about it:
Does he control my root permissions?
Once the root permission is controlled, it is difficult for the system to clean up, because it can insert its own trojan in any location, such as when the driver is started, replace a command and hide it in a directory.
In addition, you can manually clear the logs so that you cannot know what the other party has done. In this way, you can only reinstall the system.
How he hacked into my server
It is terrible to develop effective defense strategies without knowing how the other party is hacked.
However, I know that I am getting closer and closer to the truth.
He continued to look at his Trojan program and found a very interesting thing: h.c.
The comment says:

psf -- Process Stack Faker (a.k.a. Fucker)Coded by Stas; (C)opyLeft by SysD Destructive Labs, 1997-2003Tested on: FreeBSD 4.3, Linux 2.4, NetBSD 1.5, Solaris 2.7Compile with:# gcc -O2 -o h h.c# strip h

Haha, psf can be translated as a process stack Forge. As the name suggests, it is a small tool for the command output of the top command to cheat ps.


You can find that:

 

Without the root permission, the process you specified can be forged into any process name. ps, top, many process monitoring tools will be cheated.

The principle is as follows:

In the main function (as follows,

int main(int argc, char *argv[])

You can continue to call the following execv interface, while the path can be different from the agrv [0] in main () and can be specially constructed. This can cause exceptions in many process monitoring tools, displays specially crafted parameters.

int execv( const char *path, char *const argv[])

When I saw this tool, I put all the previous concerns down: using this trick to lie to me, it means that you probably did not have the root permission.

I seem to have heard each other's sigh, but it does not seem like a little laughter.

No matter how much, the next step is about his life. I am thinking.

Mining

Check the core information of the system:

# Check whether the user information is normal cat/etc/passwd # Check whether the system file is replaced by find/-user 122 | xargs ls-l

Everything is normal, so the system is probably not passive to the root, it is time to find out the reason for being hacked.

Start with the clue of another process.

Process Number 111, which is very special. You can see from the user information just now that this user number belongs to: elasticsearch.

As it turns out, this prompt is too important. At this moment, I still remember that in order to install railsgirlschina.com, I used campo3 and installed it on elasticsearch. Previously, Rei sent a dedicated email to tell me: this product may have a remote execution vulnerability.

No, I have enabled the ufw firewall.

# ufw statusStatus: inactive

What, not open? My server went bare for so long in the harsh internet environment. I checked the operation logs and found that the firewall has not been opened since January 8.

Besides, I closed it myself. It's no wonder that his log of successful intrusion was exactly in January 8.

At this moment, the hacker's intrusion method is almost completely clear: the vulnerability scan tool is remotely executed by elasticsearch to scan my server, after running the permission escalation tool, we found that the/home/ruby/directory is writable.

At this time, he cleverly disguised his Trojan. Although he did not have the root permission, he could still easily kill my blog process (although he did not ). it can also enable my server to become an accomplice, a secondary springboard, or launch DDOS attacks against innocent people at any time.

I am also lucky that elasticsearch does not run under the root permission, but uses user No. 111. So the best result of this attack is that, get the permissions of my ruby user, include my server into his zombie, and lay the groundwork for his large-scale operations in the future.

Some energy needs to be added. I was thinking that the brewed noodles had reached the mouth.

"If I become a world-class hacker, I must use his reverse proxy trojan horse to hack into the dark." I am thinking about it together. "To Do This, please clean up these Trojans first ."

Clear

Close Source

Open the firewall: ufw enable.

Clear crontabs: crontab-l, rm/var/spool/cron/crontabs/ruby

Kill the Trojan process: kill pid

Handle Vulnerabilities

According to the official elasticsearch processing recommendations, set the default listening IP address to 127.0.0.1, and disable the dynamic execution script capability: script. disable_dynamic: true (all in its configuration file)

Kill the elasticsearch process and sub-process that have been successfully attacked: kill-9 xxx

Reinforcement

It seems that I am a lot worse than a master. I am thinking that the following measures should be taken to remedy the problem:

User directory permission

Previously, useradd's default user directory permission 644 was used to allow access by any user, which allowed the hacker to successfully intrude into the directory. I disabled it: chmod 700/home/ruby

Web vulnerability scan

I need to perform some vulnerability scanning on my server locally.

Firewall Enabled

Add ufw enable to/etc/rc. local.

Upgrade System

Speaking of this, the two commands should be done immediately:

apt-get updateapt-get upgrade

Postscript

As the traffic drops normally, my favorite ping shows a perfect display.

PING myblog.me (xx.xx.xx.xx): 56 data bytes64 bytes from xx.xx.xx.xx: icmp_seq=0 ttl=52 time=84.207 ms64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=52 time=80.165 ms64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=52 time=83.242 ms64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=52 time=86.241 ms64 bytes from xx.xx.xx.xx: icmp_seq=4 ttl=52 time=86.799 ms......

My "ladder" came back stably.

I felt that I was a step closer to the Master. Suddenly, the display started to appear slowly.

64 bytes from xx.xx.xx.xx: icmp_seq=4 ttl=52 time=86.799 msRequest timeout for icmp_seq 0Request timeout for icmp_seq 1Request timeout for icmp_seq 2Request timeout for icmp_seq 3......

I know. The other party is back.