Stress Testing tps performance degradation solution

Stress Testing tps performance degradation solution
Stress Testing tps performance degradation problem solution background

During the force measurement test, the tps has been declining. to reproduce the problem, a simple transaction is developed for testing. The test code is as follows:

Record the transaction script and put it in LoadRunner11 for testing. In this scenario, 10 users can start and run at the same time. It can be seen that tps starts to decline in 1 minute and continues to decline in the later stage.

At this time, no exceptions are found, such as server logs, javacore, heapdump, and gc. But the test results continue to decline. Record the server's request processing time, and found it to be very stable. It was initially suspected that the client was under insufficient pressure, but there was no specific evidence, so the pressure test tps decline was put on hold.

Turning Point

This week, a colleague started a stress test. After talking about the problem, a similar test was also conducted on the site, but the test results were exactly the same. Later, guangfa used the vuser running by process in LoadRunner. The test result is as follows. As you can see, the entire process lasted for three hours, and the tps remained stable at around 950. Now there was a breakthrough.

Analysis

Why are the results of running vusers by process in LoadRunner different from those running vusers by thread.

The two types in LoadRunner are described as follows:

? Run vuser by thread

Start multithreading to run more vusers on each server load generator computer.

? Run vuser by Process

Disable multithreading and run each vuser as a complete process. Used for non-thread security protocols

The Internet has the following explanation:

Thread resources are allocated from process resources. Therefore, multiple threads in the same process have shared memory space.

The memory usage of the mdrv.exe (lr program used to run the vuser) process when the vuseris run by using the god javavisualvm.pdf. We can see that fullgc has been executed 94 times in 7 minutes, after careful consideration, we can find that the new generation is 896 k + 64 k * 2 = 1024 k, the maximum is 4992 k, the old generation is 4 m, and the maximum is 59.125 m. In this case, we assume that the jvm memory allocation is too small.

Optimization

Now adjust the VM memory and use the same scenario for testing. The jvm parameters and test results are as follows:

Jvm parameters:-Xmn128m-Xmx384m-Xms384m-XX: PermSize = 64 m-XX: MaxPermSize = 128 m

As expected, the test runs stably. The memory usage is as follows:

We can see that the memory usage of the old generation has been stable at 1.457 m, and the persistent generation memory has been stable at 5.616 m,

Full GC is not performed throughout the process, and garbage collection is all performed in the young generation.

Based on this test, we continue to adjust the virtual machine parameters in this scenario, increase the number of young generations, and reduce the number of old and persistent generations. The same scenario is used for testing. The jvm parameters and test results are as follows:

Jvm parameter:-Xmn320m-Xmx384m-Xms384m-XX: PermSize = 12 m-XX: MaxPermSize = 12 m

Continue to check the memory usage during the test.

We can see that during the period of, the young generation executes gc for 2304 times, which takes 45.703 s. Compared with the previous test run for, the young generation executes gc for 3228 times, which consumes 35.965 s.

Conclusion

The continuously decreasing tps in this stress test is not caused by the processing performance of the server and client, but by the unreasonable memory allocation of the stress test client.

If a similar situation occurs during subsequent tests, you can choose to run the vuser by process or adjust the memory parameters according to the actual transaction scenario to ensure that subsequent tests are normal.

Appendix

Memory usage of one of the processes when running vuser by process: