Analysis and solution of the problem of high bandwidth occupancy rate of Ambari server network port

Ambari is hortonworks out an open source Hadoop management system, is written in Python, the current market is open source Hadoop management system seems to be the only one, although ambari problems, but also not good use, but there is no way.

Recent surveillance systems often warn that a URL is always unreachable, just a URL to a Ambari server.

Then log on to the server for a probe.

Use Iftop to view the network condition, found that the network occupancy rate is high, reached 700Mbps, and has been very persistent, the maximum bandwidth of the network card is 1000Mbps, was used off the more than half, no wonder will alarm, this is not normal.

Looking closer, we find that Ambari server and each slavenodes have a lot of packet transmissions, and it's definitely about a service. Suspicion is ganglia.

On the safe side, I logged on to another Ambari server and found that the network occupancy rate was low and 1Mbps was not there.

Use tcpdump to grab the bag and then analyze it with Wireshark.

Tcpdump-i bond0 ' src host slavenode '-W traffic.cap-g 60-w 1

-g:override previous saved file very n seconds.

-w:keep N Files

Grasp the package, the package to their computer, with Wireshark analysis, found that almost all packages are metrics related, inference is definitely ganglia, because another Ambari server does not install ganglia.

Although the ganglia service has been stopped, but it is estimated that the agents end in the continuous hair metrics to the server side, resulting in a high network occupancy.

So delete ganglia, in the Ambari Web UI can not delete ganglia, only call Ambari API to do.

#STOP the GANGLIA SERVICE
Curl-u admin:admin-h "X-requested-by:ambari"-X put-d ' {"Requestinfo": {"context": "Stop Service"}, "Body": {" ServiceInfo ": {" state ":" Installed "}}} ' https://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_NAME>/ Services/ganglia

#STOP the server and MONITOR on the GANGLIA server
Curl-u admin:admin-h "X-requested-by:ambari"-X put-d ' {"Requestinfo": {"context": "Stop Component"}, "Body": {" Hostroles ": {" state ":" Installed "}}} ' Https://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts /<ganglia_server_fqdn>/host_components/ganglia_server
Curl-u admin:admin-h "X-requested-by:ambari"-X put-d ' {"Requestinfo": {"context": "Stop Component"}, "Body": {" Hostroles ": {" state ":" Installed "}}} ' Https://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts /<ganglia_server_fqdn>/host_components/ganglia_monitor

#STOP the GANGLIA MONITOR on ***every*** node (REPEAT for each node WHERE GANGLIA is monitoring):
Curl-u admin:admin-h "X-requested-by:ambari"-X put-d ' {"Requestinfo": {"context": "Stop Component"}, "Body": {" Hostroles ": {" state ":" Installed "}}} ' Https://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts /
#STOP each SERVICE COMPONENT:
Curl-u admin:admin-h "X-requested-by:ambari"-X put-d ' {"Requestinfo": {"context": "Stop All Components"}, "Body": {"Serv Icecomponentinfo ": {" state ":" Installed "}}} ' Https://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_NAME >/services/ganglia/components/ganglia_server
Curl-u admin:admin-h "X-requested-by:ambari"-X put-d ' {"Requestinfo": {"context": "Stop All Components"}, "Body": {"Serv Icecomponentinfo ": {" state ":" Installed "}}} ' Https://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_NAME >/services/ganglia/components/ganglia_monitor

#RERUN to CHECK all components is STOPPED:
Curl--user admin:admin Https://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_NAME>/services/GANGLIA

#REMOVE the GANGLIA SERVICE:
Curl-u admin:admin-h "X-requested-by:ambari"-X DELETE http://<AMBARI_NODE>:8080/api/v1/clusters/<CLUSTER_ Name>/services/ganglia

Delete Ambari, and then use Iftop to see the network occupancy rate, return to normal level.

This article from "Linux operation and Maintenance" blog, declined reprint!

Analysis and solution of the problem of high bandwidth occupancy rate of Ambari server network port