A CP command raises a large number of MongoDB slow queries

Encounter problems: early morning received the alarm, a MONGODB server CPU load more than 8. Because it did not affect the business, the next morning began to check the reasons.

Check the reason:

1. First understand what the application on this server is

The DB server is primarily used by MongoDB only. Then start querying the MongoDB log:

Through the early morning log mongodb.log view, there are a lot of slow queries, but in fact these tables are very small, only hundreds of rows of data, and the table has an index, but only a query spent 60~80s, slow query after the log shows that the node is not detected by other nodes (MongoDB copy set form).

650) this.width=650; "Src=" Http://s5.51cto.com/wyfs02/M00/89/A9/wKiom1gZXljjDea6AAmHVICatn8483.png-wh_500x0-wm_3 -wmp_4-s_673307001.png "title=" 3.png "alt=" Wkiom1gzxljjdea6aamhvicatn8483.png-wh_50 "/>

Because a small table of queries need to determine about 70s, and MongoDB deployment is the copy set form (other query nodes are normal) can be judged, it may be the performance of this DB affects MongoDB, not the performance impact of MongoDB itself.

2. Then look at what tasks to run in the wee hours.

Crontab-l found that there was a task in the wee hours. is a script for cutting logs. This is probably the log CP to another directory, and then empty the current log, continue to record the new day of the log.

But this log is usually small, and it runs for a long time. You can only try it. Look at the log directory

650) this.width=650; "Src=" Http://s3.51cto.com/wyfs02/M00/89/A7/wKioL1gZWOKxEuHYAAATiMt--TM154.png-wh_500x0-wm_3 -wmp_4-s_1606677349.png "title=" 1.png "alt=" Wkiol1gzwokxeuhyaaatimt--tm154.png-wh_50 "/>

See the log is suddenly so big. Was it due to the time of the CP file at night?

The almost-judgment problem occurs when the CP causes the IO load to be too high, which in turn causes the CPU to load too high.

3. Reproduce the problem

Because the operation was operated at night, it did not affect the online business. So manual operation CP, reproduce the problem.

CP has a file that is nearly 3g, such as seeing too many I/O requests generated and the I/O system is already full load, the disk may have bottlenecks.

Then the CPU load also rises to 10 (CPU load per minute) around.

650) this.width=650; "Src=" Http://s1.51cto.com/wyfs02/M00/89/A7/wKioL1gZWfCD_61uAAGL7nuZJOM045.jpg-wh_500x0-wm_3 -wmp_4-s_1076327921.jpg "title=" db104.jpg "alt=" Wkiol1gzwfcd_61uaagl7nuzjom045.jpg-wh_50 "/>

Solve the problem:

Separate the larger logs from the MongoDB server.

Change the truly log script cut to the system logrotate cut. The logrotate will operate in the idle state of the system.

But why copying a 3g file can result in a CPU load of up to 10. This results in a straight down of MongoDB query performance.

So we contacted a cloud, how the performance of a common cloud disk is so low!

The above-mentioned thinking of the problem, the beginning is not very smooth. At first the file was small and there was no guessing that it was a log copy. Do not use inertial thinking when checking the problem.

This article is from the "Meng Sir accumulation" blog, please be sure to keep this source http://mengsir.blog.51cto.com/2992083/1868392

A CP command raises a large number of MongoDB slow queries