ArticleDirectory
- The stress test ran for 8 hours, and the result data began to suffer from 5th hours. The response time was more than doubled (the source image cannot be found, and the shape is the same)
A series of data fluctuations found during performance tests in recent projects are recorded below for future search.
The stress test ran for 8 hours, and the result data began to suffer from 5th hours. The response time was more than doubled (the source image cannot be found, and the shape is the same)
The data before and after the fluctuation is very stable. Check the log and find that no logs are generated after the problem occurs. Check the size of each log file through LS-SH and find that one of the log files is 208 GB, the whole home space is full and cannot be entered when logging continues. After deleting the log, run the stress test again and the problem disappears.
The performance test results show that the interface response time is very different from the expected results. First, check the environment parameters used for the test. There is no problem. Check the data to find that the slow response time of the entire interface is not caused by data fluctuations, but is always slow. eliminate this problem. You can use visualvm to directly connect to the Performance Testing JVM and view the interface information. It is found that one step in the interface call takes more than 90% of the time. The problem is located here. Through detailed troubleshooting of this method, we find that there is a database call in it. However, it usually takes several milliseconds to query an index of tens of thousands of data tables, so it is not slow to this level. I checked it in detail and found that no corresponding index was created for the performance test database, but it was created in both the test environment and the online environment. Therefore, the cause is that the index tree has been built. After the index is built, the data meets the expectation of merging the trunk, and then the performance test is re-performed. It is found that the data has a periodic fluctuation and a large fluctuation occurs every five minutes, the full GC, thread pool size adjustment, cache expiration, Apache configuration, and other issues were ruled out. The final problem was focused on a newly added JVM monitoring. A 5-minute event was found after a detailed search. Thread data is collected and threadmxbean is called. dumpallthreads (booleanlokcedmonitors, Boolean lockedsysnchronizers) method, the passed parameter is threadmxbean. dumpallthreads (True, true), which needs to traverse the monitor and synchronizer information of all threads, leading to blocking of all threads (normally there will be no such obvious impact, during performance testing, the concurrency is set to 30, and the number of requests per second is several hundred, so the impact will be greater ). Modify the parameter to threadmxbean. dumpallthreads (false, false.