A murder caused by a simple password

A murder caused by a simple password

Tian Yi ( [email protected] )

is trying to board bricks, a burst of telephone bells remembered, colleagues emergency informed, forum visits very slow, other technical staff have been tossing half a day to find reasons, need to deal with as soon as possible.

first a brief look at some of the situation, the operation of the colleague said that a few days ago the administrator account was leaked (how leaked? Set a simple password, really like the trunk of the brick to take the past to shoot him), was a tool to automatically post, good guy, a night sent tens of thousands of posts. In the morning in the QQ Group to see them chatting, said it took a lot of time to delete these spam posts. The Forum's discuz program did not make any changes, suspected to be malicious attacks. Forum can not access already large half a day, too late to learn more about the situation, rolled up the sleeves, open dry.

Before logging into the system, consult the data to confirm the logical topology of the entire forum, as shown in:

650) this.width=650; "title=" 21-1.jpg "src=" http://s3.51cto.com/wyfs02/M01/6F/ED/ Wkiom1wtpgqywxlbaafcq1vd19a822.jpg "alt=" Wkiom1wtpgqywxlbaafcq1vd19a822.jpg "/>

Note: This figure omits the management network (by VPN management of the server)

Follow this topology to step through the inspection and troubleshooting steps as follows:

(i) Load balancer check

1 , check the system load, memory usage: The output indicates that it is normal;

2 , check disk space usage, no partitions are found to be stuffed;

3 , check the system log, in order to detect abnormal output;

4 , check application run, execute command Ipvsadmin , there are forwarding numbers and the numbers are constantly changing, which confirms that the application keepalived and LVs are normal.

The above points determine that there is no problem with load balancing, analog failover, and user requests can automatically switch to the secondary load balancer.

(ii) Check the application server

1 , view the load on several servers, Load good value dozens of, serious exceeding;

2 , view disk space usage, for discovery usage greater than 80% the partition;

3 , check the system log, did not find the file system corruption, kernel exceptions, such as problems;

4 , view memory usage, and consume swap partitions without running out of memory Swap the situation;

5 , check the application PHP and Nginx , Nginx no exception, and PHP occasionally have slow log output;

preliminary judgment, is PHP caused by a load exception. The system load quickly drops to its normal level when PHP is closed for a moment. Suspected of malicious attack, by looking at the TCP status, there is no valuable clue.

650) this.width=650; "title=" 21-2.jpg "src=" http://s3.51cto.com/wyfs02/M01/6F/EA/ Wkiol1wtqcywfcpbaacplzmo4hc215.jpg "alt=" Wkiol1wtqcywfcpbaacplzmo4hc215.jpg "/>

if attacked, the same IP of the TCP the number of connections should probably be a large value. The general experience, is thousands of levels. Try to use the iptables to prohibit a few connections to a number of source IP, the effect is not obvious, the load is still very high. Turn off php, load down, start again, load quickly and climb up again. It seems that the problem is not here yet.

(iii) Check the file servers that are closely associated with the application server

Forum program is to NFS is shared with multiple application servers in order to reduce data synchronization between servers and, if not in a shared form, each application server should keep a copy of the exact same data file. Because some data files change over time, these update data must be synchronized in real time in order to ensure user access consistency.

1. Check the system load, memory usage, found all normal;

2. Check disk space usage and do not find the blocked partition;

3. Check the system log, no abnormal output was found;

4. Check NFS -related processes and configurations, no exceptions found;

A situation in which a forum access exception has occurred before is due to the sharing of file systems ( NFS ) partition file system corruption, causing NFS client (that is, PHP Application Server) cannot write to the cache. In order to confirm this speculation, both the file server and the client have been tested for creating and deleting files, everything is OK.

(iv) Check the cache server.

the application that the data cache uses is memcached , this kind of server system, only runs such an application, because is the memory storage data, therefore generally, the suspicion has the fault, simply restarts memcached , so that it can be re-cached.

1. Check the system load, memory usage, all normal;

2. Check System log, no exception;

3. Check the network condition, also very normal;

Restart memcached , the load on the application server doesn't help, and it seems that the problem is not here.

(v) Check the database server.

There is only one left, and the problem is clearly related. Take a sip of haze air, log in to the master database and from the database server system, perform the following routine troubleshooting.

1. Check the system load, memory usage, all normal;

2. Check disk space usage and do not find the disk partition that is stuffed;

3. Check the system log, no abnormal output was found;

4. Check network connectivity, the number of primary server connections is very small, and the number of connections from the server, which at least indicates that at the system level, the primary database server is normal, from the server system, at the system level, it is basically considered normal.

Forum program used by this cluster Discuz The database itself supports read and write separation, so the primary server's task is to assume the task of data update, and query a class of work, then by the read-only state from the server ( slave ) Bear. We will first layer the primary server application level to troubleshoot, the steps are as follows:

( 1 ), check MySQL error log and full query log, for the detection of abnormal output;

( 2 ), check MySQL process is normal, execute mysql–p log in and get into MySQL client command line, daemon normal;

( 3 ), MySQL Client Execution Commands show full processlist; output A few lines, all in an acceptable range;

As a result, there is no exception to the database master server.

Now, the bottom of the rope, the problem is mostly the problem from the database server. System-level checks to find useful clues. Check mysql error log, no exception; check mysql slow query log, sure enough to have output, but also constantly have new output.

650) this.width=650; "title=" 21-3.jpg "src=" http://s3.51cto.com/wyfs02/M00/6F/EA/ Wkiol1wtqfycmg-4aaheywojllg056.jpg "alt=" Wkiol1wtqfycmg-4aaheywojllg056.jpg "/>

a pieceSQLThe statement is long, confined to space, not intended for full output. LoginMySQLclient, executingshow full processlist;Oh, long time, the output is over, the number of threads exceeds +(I set theMySQLthe maximum number of connections is +), and the output of the content, most of the slow query inside the Dongdong. Withmysql–p–e "show processlist;" >/root/sql_output theseSQLform a text file that is forwarded to the programmer. They looked at it for a long while and did not come to a conclusion. Some people suggest that killing looks unusual.SQLstatement, but after killing, or not. People still suspected of being attacked throughDNSThe request is first directed to the the, play a certain role in filtering the attack. Seeing from the surveillance, there was a real result.ccThe attack was intercepted. However, the load on the application server does not play a role, and the forum is still in the open state.

Login to the forum backstage with the Super system administrator's account, after closing the forum, the server load returns to normal level immediately, then open the forum, the load soared. Scary Ah, incredibly technical level to find out where the problem lies. Toss a day, the head is big. In the QQ Group shouted a few words: "Who changed the program did not?" Who's doing the backstage setup? ... .. " , someone popped up and said, "When you delete a spam sticker, you set the number of posts that the page displays for quick deletion."

650) this.width=650; "title=" 21-4.jpg "src=" http://s3.51cto.com/wyfs02/M02/6F/EA/ Wkiol1wtqiijeokgaaeaakjeodq947.jpg "alt=" Wkiol1wtqiijeokgaaeaakjeodq947.jpg "/>

My math is taught by political teachers, can not be more than a few plate multi-person visit time, specific to show how many records, but we all believe that this must be a very scary number.

650) this.width=650; "title=" 21-5.jpg "src=" http://s3.51cto.com/wyfs02/M01/6F/EA/ Wkiol1wtqljruk0zaagpx-uedns077.jpg "alt=" Wkiol1wtqljruk0zaagpx-uedns077.jpg "/>

Discover the problem and realize the seriousness of the problem.

650) this.width=650; "title=" 21-6.jpg "src=" http://s3.51cto.com/wyfs02/M02/6F/EA/ Wkiol1wtqptsrum6aahlped69no887.jpg "alt=" Wkiol1wtqptsrum6aahlped69no887.jpg "/>

Forum Background Change This setting, a few seconds later, the application server load down, the number of connections from the database also tends to normal, the forum can be the same as before, you can browse, post ...

All this, in retrospect, is some forum admin setting the damn simple password.

This article is from the "Sery" blog, make sure to keep this source http://sery.blog.51cto.com/10037/1676615

A murder caused by a simple password