Actual combat: Use Hyperic HQ to diagnose the website cannot access the problem

Problem description

has been using Hyperic HQ CRP monitoring two websites:

Www.GoodU.info: "If I smell", record some good articles that are inadvertently seen every day, typesetting concise, easy to read, "You never know what type of article the next article." ”

Www.wongjingwingchun.com: "Huang Yong Chun will", a Yong Chun boxing members of the website, by Huang Wing Chun descendants free halfway, here to do an advertisement, welcome to participate.

Weekend leisure, suddenly received alarm mail, monitoring of the site can not access.!!??

Environment description

The use of the mainstream technology is basically.

CENTOS6, 64bit,; Apache Httpd 2.2.15, MySQL 5.1.x; Drupal;

The HQ Agent 5.8 is installed on the server;

Analysis process

The following picture is after the post-recovery screenshot, please focus on the middle red dot time section:

Turn on monitoring tools http://demo.innovatedigital.com:7080/

CPU utilization and load, before the incident, the abnormal movement, rapid increase.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/47/F7/wKiom1QFMNbDitA0AAdTHf9UAuk059.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image005.png "alt=" Wkiom1qfmnbdita0aadthf9uauk059.jpg "/>

Memory swap area, free memory and swap area use size, fast increase. Here, taking into account the Linux memory management mechanism, the focus is free memory (+buffers/cache), rather than used memory (small change). It is possible that a program suddenly takes up a lot of memory.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/47/F7/wKiom1QFMP-ihUMLAAddesPLhxk188.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image003.png "alt=" Wkiom1qfmp-ihumlaaddesplhxk188.jpg "/>

Looking at the httpd, the byte throughput and request volume per minute did not change significantly.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/47/F8/wKiom1QFMSOjFmk0AAanxOcNqho481.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image007.png "alt=" Wkiom1qfmsojfmk0aaanxocnqho481.jpg "/>

Busy Servers and Busy Workers increased, but Keepalive did not change significantly. The cause of the problem may not be related to HTTP.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/47/F8/wKiom1QFMU7A9Z9FAAZQBJWsT1g948.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image009.png "alt=" Wkiom1qfmu7a9z9faazqbjwst1g948.jpg "/>

Also need to observe the database, MySQL, its process CPU utilization is not changed, memory is reduced, it is estimated that the other process with a lot of memory, the MySQL has been squeezed, freed up some memory. Slow query number has increased.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/47/F8/wKiom1QFMX2yW2CZAAbwwb9kjh4290.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image011.png "alt=" Wkiom1qfmx2yw2czaabwwb9kjh4290.jpg "/>

So which process suddenly takes up too much memory and CPU? Look at the top results of each time point collection, a lot of RDP, what is this? Never installed, is it rdesktop or ...? Before the problem occurred a few points in time, there were these phenomena before, and not. 650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/47/FA/wKioL1QFMZWDp443AAe6ka_1VJs315.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image001.png "alt=" Wkiol1qfmzwdp443aae6ka_1vjs315.jpg "/>

Log in to the host and discover that there is an. rdp directory under/root/that contains a lot of files containing frequently used passwords. Was it black? Which prawn is interested, can leave a message to request.

Analyze the secure log and extract a few lines as follows:

16:04:05 www sshd[1842]: Failed password for root from 111.74.238.167 Port 1237 Ssh2 ... 16:39:46 www sshd[2432]: Failed password for root from 202.109.143.35 port 4421 ssh2 ... 17:01:44 www sshd[2756]: Failed password for root from 117.21.173.175 port 2182 ssh2 ... 22:10:29 www sshd[5437]: Failed password for root from 222.186.34.36 port 3580 ssh2 ...

Where do I find the IP?

111.74.238.167,202.109.143.35,117.21.173.175 Jiangxi Province Jian Telecom 222.186.34.36 Jiangsu province Zhenjiang Telecom

is not the Nanxiang Technical school in Shandong?

Preliminary conclusions

According to the above, it is likely that the host password is guessed, and thus be exploited like the above several addresses.

Measures

The first is to change the password, close SSH login, use VNC.

In order to be able to find a problem that the site cannot access earlier, a service is defined in HQ that polls a particular Web page to determine whether a problem has occurred based on the return time and the data returned. Specific as follows:

Define an HTTP service, named Goodu.info Connection test, specific parameters: hostname www.goodu.info; PORT: 80; time-out: 10 seconds; PATH:/gc/;  Method:get;   Pattern: As I smell; follow redirect:yes; It means to visit HTTP://WWW.GOODU.INFO/GC regularly if the page returned is "If I smell" four words, it is considered normal.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/47/FA/wKioL1QFMbni2KsVAAK86CrAeLI974.jpg "width=" 301 " height= "title=" Image017.png "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "alt=" wkiol1qfmbni2ksvaak86craeli974.jpg "/> 650" this.width=650; "src=" http ://s3.51cto.com/wyfs02/m01/47/fa/wkiol1qfmc6aoj7qaavk9bpvgqy970.jpg "width=" 276 "height=" "title=" image019.png "Style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Alt=" Wkiol1qfmc6aoj7qaavk9bpvgqy970.jpg "/>

Then, define some alarms and so on. After the website is restored, look at some graphs:

Mysql Run Status:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/47/FA/wKioL1QFMeyB8ZF9AAWDiv6AGow799.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image013.png "alt=" Wkiol1qfmeyb8zf9aawdiv6agow799.jpg "/>

Host running Status:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/47/F8/wKiom1QFMgXT4ht6AAXmBCAaito429.jpg "style=" padding:5px;margin:10px 0px;border:1px solid RGB (221,221,221); Background-color:rgb (244,247,249); "Title=" Image015.png "alt=" Wkiom1qfmgxt4ht6aaxmbcaaito429.jpg "/>

Message

Come on, you guys, hack shrimp to a station mercy;

What do you mean by being a master of RDP? Hope to draw more conclusions.

Http://hq.innovatedigital.com

This article is from the "HYPERICHQ Application Monitoring" blog, so be sure to keep this source http://hyperichq.blog.51cto.com/1250795/1547718

Actual combat: Use Hyperic HQ to diagnose the website cannot access the problem