Cause
Recently we have an Ubuntu Aliyun server has been prompted to chicken behavior, prompted for several days, the beginning and no attention, and then a few days after the discovery should be a big problem ah. It's possible that the service was compromised!!!
Looking for clues
At first I was completely freaked out, Linux is not very familiar, only simple commands, install deployment Redis,mongo these things. OK, only Baidu Google!
Looking for suspicious processes
Ps-ef
However, the results seem to have no clue, very unfamiliar with Linux common process!
Find the relevant log clues
Linux has a lot of log files, unified storage under the/var/log, here I want to see if someone cracked the account to invade the server
Cat/var/log/faillog-Login Failure log
Cat/var/log/auth.log--Verifying log
Indeed found some clues (after the solution was found to be not so, for the time being has not been studied)
A large number of failed are found in Auth.log, which means that someone is trying to break the password violently, the most suspicious of which is a large number of sessions opened for the user root by (uid=0) (at first I think it was an intrusion?) )。 Baidu went down and found a few clues:
Aliyun official issued a notice of the dirty cow loophole https://bbs.aliyun.com/read/297492.html
A hacker versus http://ruby-china.org/topics/23848
After looking carefully, I found the groove! The hacker duel This and my situation is same AH ~ a few days ago in order to deploy exceptionless, migrate Elasticsearch to Linux. Did not pay attention to the security of El.
Looking for a Trojan horse
Look at the process again, this time with documentation help, a general understanding, and a comparison of the process of taking another Linux server. Immediately locate the suspicious process.
Through the elastic+ start of these several processes is apparently Trojan process, according to the above documents, can be a preliminary explanation Trojan did not get root permissions, and run under the elastic user rights.
Find the process, how to find the file? Baidu!
cd/proc/31598
Ls-l exe
The horizontal slot is also deleted! But it's also located in a suspicious place./tmp
Go to/TMP now to view
These files should be a trojan, down to open looked down, is indeed a Trojan! Also Baidu to some information, these are the broiler program! Document
Kill the Trojan.
Found the Trojan, and finally to kill it, but cautious I still do some other lessons, to prevent the indiscriminate killing, resulting in more damage to the Trojan. (Baidu, however, found that Linux is just too weak)
Stop Elasticsearch
Service Elasticsearch Stop
Replace Elasticsearch configuration, this is the key to this vulnerability!
Script.disable_dynamic:true--Change from flase to True
I was thinking about how a dynamic scripting capability would be considered? The perfect loophole, just like the last Redis default no security verification problem!!!
Delete Temp
Rm-rf/tmp
I'm furious! But the impulse is the devil, RM-RF please be careful!!! The sad story of the entire server is that it provoked.
Mass Kill process
Kill-9 $ (ps-ef | grep elastic | grep-v grep | awk ' {print $} ')
Re-review the list of processes to make sure the Trojan does not reboot
Ps-ef
The result seems to be optimistic, but is it really dead? Still need time to test?
Restart Elastichsearch
Service Elasticearch Start
To be more secure, the server has tightened the password and swept it with ClamAV.
Sequel?? I hope we don't have a sequel,!!!.
Summarize
This security incident, I think probably like me a few Linux novice, along with. NET across the platform, a lot of. NET application will rely on more Linux environment components, Linux application security must also be more important to us (although Linux is not in the industry as a claim than Windows security, but Linux should say this is Elasticsearch pot!!!) )
The above is a small set to introduce the Ubuntu server is black experience, I hope to help everyone, if you have any questions please give me a message, small series will promptly reply to everyone. Here also thank you very much for the cloud Habitat Community website support!