Become a Java GC expert (4) A detailed description of the maxclients parameters of-apache and their impact on Tomcat execution FULLGC

Let's take a look at MaxClients how Apache's parameters affect the system when the full GC occurs.

Most developers are aware of the "stop-world phenomenon (STW)" (see Understanding Java Garbage Collection) caused by GC occurrences. In particular, NHN Java developers often encounter tomcat errors due to GC reasons. Because the Java Virtual Machine (JVM) manages memory, Java-based programs cannot get rid of the STW phenomenon caused by GC. If at a certain time, when you are working on the application you are developing, the GC starts executing. Even if the TTS error does not occur, your service will show the customer an unexpected 503 error.

Service Execution Environment

Due to the nature of the architecture itself, the Web service is more suitable for scale-out, compared with vertical scaling (translator Note: Increase the number of servers rather than the configuration of the components). Therefore, in general, the physical device is configured to 1 Apache+n Tomcat according to the performance requirements. But this article assumes that our environment is 1 apache+ a tomcat is installed on a single host row as shown in.

Figure 1 : The service execution environment of this article fake shot

For reference only, the parameters described in this document are based on Apache 2.2.21 (prefork MPM), Tomcat 6.0.35,centos 4.72 (32-bit), JDK 1.6.0_24.

System available memory 2GB, garbage collector using PARALLELOLDGC,AdaptiveSizePolicy采用默认的设置true，堆内存空间600M

STW and HTTP 503

Let's assume that the request to access Apache is req/s and that there are 10 httpd processes running, and we don't consider the response time of each request for the moment. In this context, we assume that the pause time due to the full GC is 1 seconds. when Full GC when it happens. Tomcat What will happen?

The first thing that comes to your mind is that Tomcat will stop responding to any requests because of the full GC. In this case, what happens to Apache when Tomcat pauses the corresponding request ?

When Tomcat pauses, the request is constantly flooding into Apache at a rate of req/s. In general, the response of a request can be quickly processed by 10 or more httpd processes before the full GC occurs. However, because Tomcat is paused, the httpd process is constantly created to respond to the new incoming request. Until the definition in the httpd.conf file is exceeded MaxClients . Because the default value of 256,apache does not care about the influx of requests at a speed of req/s.

What happens to the newly created httpd thread?

The HTTPD process forwards the request to Tomcat through an idle AJP connection managed by the mod_jk module. If there is no idle connection, he will apply to create a new connection. However, because Tomcat is paused, the request to create a new connection is rejected. So these requests are stored in the backlog queue, depending on how much of the AJP connector is set up in server.xml . Once the number of requests exceeds the space limit of the backlog queue. Apache will return a reject connection error. and returns an HTTP 503 error to the user.

In this hypothetical condition, the default backlog queue space is 100, and the request arrival speed is req/s. Therefore, a one-second pause caused by the full GC causes more than 100 requests to return 503 errors.

This way, when the full GC ends, the content stored in the backlog queue is accepted by Tomcat and processed by the worker thread, depending on MaxThreads the value of the maximum number of threads (default 200).

MaxClients and backlog

In this case, which parameter can be set to avoid returning to the user 503 error?

First, we should know that the backlog is worth so much that it can accommodate all requests that have been flooded during a pause due to the full GC. In other words too should be no less than 200.

So, will this set up a new problem?

Let's assume that the backlog is set to 200 and then repeat the process above. The results are more severe than before. System memory usage is typically 50%, however, a rapid increase to 100% in the event of full GC, which results in a rapid increase in swap memory space and, more significantly, the pause time from 1 seconds to 4 seconds or more, which results in complete GC downtime and no response to any requests.

In the first case, only 100 or more requests return a 503 error. However, when we adjust the backlog to 200, more than 500 requests will hang for 3 seconds or more to be unable to get an answer.

The above example is a good illustration of the consequences of blindly modifying a system when you do not fully understand the intrinsic relationships between the settings (for example, the impact on the system).

So why does this happen?

The root of the problem is MaxClients the characteristics of the parameters.

MaxClientsThere is no problem with setting a large value itself, but most importantly MaxClients when setting parameters, you want to make sure that even if the equivalent MaxClients number of httpd processes are created at the same time, memory usage will not exceed 80%.

The memory exchange parameters of the system are typically set to 60 (default). Therefore, when memory usage exceeds 80%, memory swapping occurs.

Let's take a look at why this feature causes the serious problem above. Tomcat was paused due to full GC when the request rushed to Tomcat at a speed of req/s. The backlog is now set to 200. Apache creates approximately 100 httpd processes. In this case, once the memory usage exceeds 80%, the operating system activates the swap memory area and moves them to the swap area because the system considers the objects in the old age of the JVM to be unused for a long period of time.

The end result is that the GC uses memory swap space and pauses time spikes. The number of httpd processes is therefore further increased. This results in a situation where the memory usage described above is up to 100%.

The only difference between these two occasions is the value of the backlog: vs.200. why only in $ in the case of a situation happening?

The difference is due to the number of httpd processes created. When the backlog is set to 100 and a full GC occurs, 100 requested connections are created and saved in the backlog queue. Other requests get a reject connection error message and play 503 error. As a result, the total number of httpd processes is just slightly above 100. When the backlog is set to 200, 200 requests Create a connection, so. The total number of httpd processes will be more than 200. This exceeds the threshold, which causes memory exchange to occur. Then, regardless of the memory usage of the set MaxClients parameters, the full GC causes the number of httpd processes to burst, triggering memory exchange and reducing system performance.

Calculation formula of maxclients parameters

If the system's memory makes 2GB, MaxClients the value in any case should not exceed the memory of 80% (1.6GB) to avoid performance degradation due to memory exchange. Other words. 1.6GB of memory should be shared and assigned to APACHE,TOMCAT and the agents that are installed by default.

Let's assume that the agent is installed by default on the system and consumes 200m of memory and -Xmx is set to 600m for Tomcat heap memory. As a result of the top command, Tomcat will always occupy 725m (Perm Gen + Native Heap area). Eventually Apache can use 700m of memory space. as shown below.

Figure 2 : The test system Top Screen Cutting

As mentioned above, what should we do when MaxClients We set the memory to 700m ?

This depends on the number of modules loaded, for NHN Web services. Apache is just a simple proxy forwarding, each httpd thread 4m of memory (as the result of the top command) is sufficient (see Figure 2). So. 700m memory corresponds to MaxClients 175.

Summarize

A robust service configuration should at least reduce the time it takes to go down when the service is overloaded and successfully respond to requests within a reasonable range. For Java-based Web services. You must check that your service is able to respond to requests for stability in the STW time caused by the full GC.

In order to respond to more user requests and respond to DDoS attacks, without taking into account such factors as system memory, it will be MaxClients set to a very large value, then it loses its function as a threshold, causing more serious problems in the system.

The situation mentioned in this article lasts only 3-5 seconds, so most of the traditional monitoring tools cannot be discovered in time.

Become a Java GC expert (4) A detailed description of the maxclients parameters of-apache and their impact on Tomcat execution FULLGC