A remote execution vulnerability in elasticsearch

Cause:

The company uses the cloud host service of ucloud. This morning, we were suddenly told that the outbound traffic of a server surged, And the outbound packet sending volume reached 1 million in a short time, all of which were UDP-type, the first feeling is: Ah, no, it's not hacked, it's a zombie!

Exploration:

Immediately log on to the corresponding server, first use iftop to view the traffic status

650) This. length = 650; "src =" http://s3.51cto.com/wyfs02/M01/42/73/wKiom1PXewmxnBlAAAIc3Cx_xKI222.jpg "Title =" 002.png" width = "700" Height = "120" border = "0" hspace = "0" vspace = "0" style = "width: 700px; Height: 120px; "alt =" wkiom1pxewmxnblaaaic3cx_xki222.jpg "/> 650) This. length = 650; "src =" http://s3.51cto.com/wyfs02/M00/42/73/wKioL1PXfFTz7OPKAAJYayVxA5Q867.jpg "Title =" 003.png" width = "700" Height = "120" border = "0" hspace = "0" vspace = "0" style = "width: 700px; Height: 120px; "alt =" wkiol1pxfftz7opkaajyayvxa5q867.jpg "/>

It can be seen that the outbound traffic is very scary, with a total traffic of MB in one minute. Check the two IP addresses, one in the United States and the other in Zhejiang Telecom;

Check the running process and find out the suspected process:

650) This. length = 650; "src =" http://s3.51cto.com/wyfs02/M02/42/73/wKioL1PXffLTYiecAAFKNFx1Tg0944.jpg "Title =" 004.png" width = "700" Height = "97" border = "0" hspace = "0" vspace = "0" style = "width: 700px; Height: 97px; "alt =" wkiol1pxffltyiecaafknfx1tg0944.jpg "/>

[. Ecc6dfe919a382] This process also wants to pretend to be a system process, which has a lot of doubts, And/tmp/FreeBSD is also a very strange thing. The user corresponding to the UID 498 is elasticsearch, remember that elasticsearch + logstash was deployed yesterday to implement the log statistics system. It's not because elasticsearch has a bug. Continue to check the cause.

650) This. length = 650; "src =" http://s3.51cto.com/wyfs02/M00/42/73/wKiom1PXf96xKGEEAAO1e8i1ER4561.jpg "Title =" 005.png" width = "700" Height = "230" border = "0" hspace = "0" vspace = "0" style = "width: 700px; Height: 230px; "alt =" wkiom1pxf96xkgeeaao1e8i1er4561.jpg "/> suspect that/tmp/FreeBSD is the trojan program. Unfortunately, it has been deleted and cannot be viewed.

Cause:

The Culprit found out, detailed reasons need to be investigated, so now the most important thing is to solve the problem, quickly kill the relevant process, and check again iftop found that the traffic quickly fell back, further confirms our judgment;

Next, we need to find out the cause of the hijacking and the specific hijacking method to eliminate the problem. However, the external search engine quickly located the cause of the problem, elasticsearch remote arbitrary code execution vulnerability:

• Elasticsearch has the scripting function to easily reprocess the queried data. elasticsearch uses the mvel script engine, which does not provide any protection, or sandbox packaging, so you can directly execute any code .;

• In versions earlier than elasticsearch 1.2, the dynamic Script Function is enabled by default. Therefore, you can directly execute arbitrary code through HTTP requests .;

• In fact, the official website is aware of this vulnerability, which is described in the document:

• First, you shocould not run elasticsearch as the root user, as this wocould allow a script to access or do anything on your server, without limitations. second, you shoshould not expose elasticsearch directly to users, but instead have a proxy application inbetween.

Find the reason, and solve it.

Solution:

Method 1: manually disable the elasticsearch script execution function, that is, add the following line to the elasticsearch. yml configuration file of each elasticsearch node.

script.disable_dynamic: true

Restart es;

Method 2: Upgrade elasticsearch to version 1.2 or later. Because the remote script execution function is disabled by default in es1.2, compatibility with logstash needs to be considered;

Follow-up:

1. According to official information, to ensure the security of elasticsearch, you cannot start elasticsearch as the root user. You can also use a proxy (or Server Load balancer) to open non-9200 ports (such as 9300) to the outside world ), forwarding to port 9200 on the Intranet can effectively prevent malicious port scanning;

2. Because it has been infected once, before re-enabling es, you need to fully check whether there are other risks to the compromised host. This involves the content of web security scanning, I don't know yet, so I hope more experienced predecessors can ask for advice!

This article from the "little ninja" blog, please be sure to keep this http://xxrenzhe.blog.51cto.com/4036116/1532469