Constantly optimizing configurations and improving performance-my one-time testing experience

Recently, the partner wants to perform a performance test on our application server. The basis of the test is the partner's original business system with huge functions and complex structures, the purpose is to check whether the application server can meet the needs of the business system and make a general evaluation of the application server. Due to limited hardware resources, the hardware server is configured with two quad-core Zhiqiang e5520, 8 GB memory, 2003 GB hard disk, and a common PC server. The operating system uses Windows Server SP2.

In the early stage of the test, when the client concurrency is 100, the server basically failed to respond after less than an hour of running. Now, the JVM heap of the application server is full, and very stable (almost no garbage collection), it seems that the memory is exhausted when the client is 100 concurrent. But the memory has never been released. Think about the possible reasons:

1. Memory leakage exists in the application system. As a result, the JVM cannot execute garbage collection in a timely manner, resulting in memory depletion.
2. The application system puts too much big object data in the session, resulting in high memory usage.
3. The JVM parameter settings are inappropriate, and the JVM cannot recycle garbage in time (this is summarized during this test, which is also the most common problem, think about some of the problems we encountered during previous development. It may have a huge relationship with this)
4. Application Server bug

Analysis: the customer described the application as a provincial business system. The maximum concurrency in the production environment can reach about 1000, and there is no problem during operation. Moreover, the business system has been running stably for a long time, it seems that the possibility of both 1 and 2 is not high. At the same time, check the server resource usage at this time, and find that the CPU (2 4-core hyper-threading CPU) usage is extremely low, always between 3%-7%, and one of them has a core, the utilization rate has been above 90%, or even often reaches 100%, while other cores are very idle and no computation is being performed. At this time, I have never used any Java monitoring and management tools, so I am confused. By looking at some previous test reports on the application server, we found that almost all tests have made a lot of changes to Java running parameters, in particular, many Running Control Parameters for Java virtual machines have been specified. After learning about the functions of several parameters and specific server conditions, you may try again. Therefore, some JVM stack and garbage collection parameters are added to the Java startup parameters. After several adjustments, the following parameters are used: "-XX: newsize = 500 m-XX: maxnewsize = 500 m-XX: newratio = 2-XX: Export vorratio = 8-XX: + useparallelgc-XX: parallelgcthreads = 8 "(the significance of these parameters will be gradually supplemented in future articles, or you can search for them). After this parameter change, the system has eight cores, when idle (without receiving large concurrency), it is estimated that these cores are constantly collecting garbage. Since then, there is no such situation as memory usage and no response from the system. Even when the concurrency reaches 300, the system still runs very efficiently. So far, we have concluded that all Java systems are running on JVM, and the performance of JVM directly affects the performance of the entire system. A good JVM is the basis for a good Java application system. When your Java application encounters a performance bottleneck, first consider optimizing the JVM startup parameters.

In addition, to ensure the system performance test efficiency and prevent JVM from performing fullgc, you can disable the JVM fullgc, that is, add "-XX: + disableexplicitgc" to the startup parameters ".

In the performance test, the system is slow, but the resource usage of the server where the application server is located is very low. The slow response speed causes the test to be deadlocked again. At this time, it was still an accident that solved the puzzle. Using the database connection pool monitoring tool built in the application server, it is found that there are a large number of waiting threads in the database connection pool, up to 200 at most. Is the database connection pool not large enough? As a result, the capacity of the connection pool is doubled, and the maximum number of connections reaches 300. After the data source is restarted, the system performance immediately increases and the bottleneck disappears.

From this point of view, the two most common problems affecting the performance of Java applications are: JVM efficiency and database connection pool. If you configure the most suitable JVM, you will get half done, and add a suitable database connection pool (you can also use spring and other management connection pools or even write relevant database connection management tools by yourself ), the rest is only the development of an excellent application system.

In addition, a good management and monitoring tool can also become an auxiliary means to improve performance. Jconsole tools above jdk5.0 have intuitively displayed JVM memory and thread information, which can basically meet the needs of general systems. Java virtual machines of IBM and Oracle also provide many performance monitoring and management tools. If the application system encounters a performance problem during running, or if the memory is always full and the CPU is idle, you can use the manual GC button provided by jconsole, execute GC once and check the memory again. The problem may be solved at this time.

Finally, I would like to explain that there is no better balance than the most. This is the original idea of development. Now it seems to be the principle of the entire software field. Some performance-first parameter configurations will be performed for this stress test. In the actual test process, database connection is always invalid due to network or other reasons. When idle databases are re-enabled, the connection validity check is not performed, and the system cannot obtain all the database connections and enters an "endless loop" that always uses invalid database connections. At this time, considering the stability and performance, we finally decided to discard some of the performance to ensure stability. In the end, the application server achieved a good performance and stability, which was agreed by the customer.