Java GC Expert series 4:apache maxclients settings and their impact on Tomcat full GC

This article is the fourth in the GC expert series. In the first understanding of Java garbage Collection, we learned several different GC algorithm processing processes, the way GC works, the difference between the new generation and the old age. So, you should have known about the 5 GC types in JDK 7, and the performance impact of each GC.

The second monitoring of Java garbage collection describes how the JVM runs the GC in real-world scenarios, how to monitor GC data, and what tools are available to facilitate GC monitoring.

The best options for GC tuning are described in the third GC tuning based on a real-world case. It also describes how to shorten the time spent on full GC by reducing the number of objects moving to the old age, and how to set the GC type and memory size.

This article describes the MaxClients importance of the Apache parameters and the significant impact on the overall performance of the system when GC occurs. With a few examples, you will have a clearer understanding of the MaxClients problems raised by values. Finally, we will show you how to set a reasonable value based on the available memory of the system MaxClients .

The effect of maxclients on the system

The NHN service operating environment has a large number of flow control (throttle valve) options that play an important role in the stable operation of the system. Let's look at the Apache option's impact on the MaxClients system when Tomcat happens to full GC.

Most developers know that GC will be accompanied by the "Stop the World" (STW) (see Understanding Java Garbage Collection for details). In particular, Java developers in NHN may have experienced a system crash in Tomcat caused by GC-related issues. Because the JVM manages memory, Java applications inevitably encounter STW phenomena caused by GC.

In the online system you are developing, the GC takes place many times a day. When a GC occurs, even if the TTS does not occur, it may still give the user a 503 error response.

System operating Environment

Based on the structural features, Web services are more suitable for scale-out rather than simply improving the performance of a single machine. As a result, the server deployment structure of the Web service is typically made up of one Apache server and multiple Tomcat servers, depending on the performance requirements. In this article, suppose that an Apache service and a Tomcat service are deployed on the same physical host as shown in:

Figure 1: The service runtime environment that this article assumes

As a reference, the parameters described herein are based on the Apache 2.2.21 (prefork MPM), Tomcat 6.0.35,jdk 1.6.0_24, and run on the CentOS 4.7.2 (32-bit) operating system.

System memory is 2GB, and with PARALLELOLDGC garbage collection, the option is turned on by default AdaptiveSizePolicy and the heap size is set to 600MB.

STW and HTTP 503

Suppose Apache has a traffic of 200QPS and turns on 10 httpd processing processes (although the actual scenario depends on the response time of the request). In this premise, assuming that the pause caused by the full GC takes 1 seconds, what happens if Tomcat has a full GC?

The first thing you can think of is that the full GC causes Tomcat to pause and the request in process will not respond. If so, what happens to Apache when Tomcat pauses and requests are not processed?

Even though Tomcat pauses processing due to full GC, the request still reaches Apache at a speed of req/s. Before the full GC occurs, only 10 or a little more httpd processes can respond quickly to service requests. But now Tomcat is paused, in order to process the new request Apache will continue to create a new httpd process until the threshold defined in the httpd.conf file MaxClients . Because the MaxClients default value is 256, the request for REQ/S does not cause much problem.

What happens to the newly created httpd process at this time?

The HTTPD process sends the request to Tomcat using an idle connection in the AJP connection pool managed by the MOD_JK module. If there is no idle connection, a new connection is required. However, because Tomcat handles a paused state, the request for a new connection is rejected. So these requests will be placed in the backlog queue, and the queue length is set in Server.xml AJP connector.

If the request data exceeds the length of the stacked queue, Apache will receive a connection rejection error and return the error to the user in HTTP 503.

In this case, the length of the stacked queue is set to 100 by default, and the request speed is req/s, so that in the 1 seconds that the full GC causes Tomcat to pause, more than 100 of the requests will receive a 503 error.

After the full GC ends, the socket connection in the stacked queue is received by Tomcat and assigned to the worker thread (the maximum number of worker threads is MaxThreads determined by the default value of 200) to process the request.

MaxClients and stacking queues

In the above scenario, how do I set up to avoid returning a 503 error to the user?

First we need to know that enough stack queue lengths should be set to accommodate incoming requests during a pause caused by the Tomcat full GC. So the minimum length of the stacked queue is at least 200 (QPS is 200 above).

After this configuration, are there any other issues?

After setting the stack queue length to 200, we repeat the above scenario again. As a result, the problem is more serious than before.

Normally, the system memory usage is maintained at 50%, while in the case of full GC, memory usage rapidly rises to 100%, resulting in a dramatic increase in memory swap usage. What is more serious is that the response from the full GC has been increased to 4 seconds from the original 1 seconds, and the direct result is that the system is not responding to any requests as it hangs.

In the previous scenario, only about 100 of the requests received a 503 error, and increasing the stacking queue to 200 resulted in 500 or more requests being suspended for at least 3 seconds without receiving any response.

This example is a good proof that failure to prepare the causal relationship between configuration information can have a very serious impact on the system.

Why is that?

The principle is to understand MaxClients the characteristics of the option.

MaxClientsThe value is not easy to set too large, the key to the setup is that MaxClients even if the MaxClients number of httpd processes are created, it is necessary to maintain the application's memory so that the usage should not exceed 80%.

The system swap zone defaults to 60, so if memory usage exceeds 80%, the system will have frequent memory swapping.

Let's take a look at why this feature causes the serious consequences described above.

When the requested QPS is 200, Tomcat pauses the response by the full GC, and then sets the build queue capacity to 200. At first, about 100 additional httpd processes were created by Apache, followed by more than 80% of the memory usage, causing the operating system to actively use the swap area's memory space, whereas the object that the GC survived in the old JVM was mistaken for a prolonged period of time by the operating system. This causes these objects to be moved to the swap area.

Finally, when the GC process involves swap areas, the time-consuming increases rapidly. The number of httpd processes continues to increase, resulting in memory usage reaching 100%, resulting in serious consequences.

The difference between the above cases is only the length of the stacked queue: 100 and 200. But why is there a more serious situation at 200?

The reason is that the different lengths of the stacked queues result in a different number of httpd processes. When the value is 100, the connection required to create 100 requests is placed in a stacked queue when the full GC occurs. A new request is rejected and a 503 error is returned, so the total number of httpd processes in the system exceeds 100.

However, when the queue length is set to 200, 200 requests are received and placed in a queue. This causes the number of httpd processes to exceed 200 and triggers the operating system to perform memory exchange thresholds.

Therefore, ignoring the memory usage and blindly increase MaxClients the number, will lead to the full GC when the HTTPD process increases rapidly, the introduction of memory exchange and ultimately reduce the overall performance of the system.

So how to set maxclients, how to find the current system threshold?

How to calculate the value of maxclients

If the total system memory is 2GB, set MaxClient the value to ensure that the memory usage is no more than 80% or 1.6GB at any time, thus avoiding performance degradation caused by memory exchange. That means there is only 1.6GB space for Apache, Tomcat and other default installed agents to share and allocate memory.

If the default installed agent consumes 200M of memory, and Tomcat's heap space -Xmx is set to 600M, as shown, the total tomcat consumption will be 725M (persistent + local heap space). Apache can use the space for the remaining 700M.

Figure 2:top the command

For Apache 700M memory, how do I set a reasonable maxclients value?

Of course this also depends on the type and number of modules that Apache loads. As an example of NHN Web services, Apache is used as a simple proxy, and 4M space is sufficient for each httpd process, according to Res. So the 700M space can be set MaxClients to 175.

Summarize

A reliable service configuration reduces system downtime and maximizes the ability to successfully respond to user requests at full load. For Java applications, it is important to verify that the configuration of the system provides enough reliable service in case of SWT caused by full GC.

If the setting is too large to deal with simple requests to increase and prevent DDoS attacks without regard to memory usage, it will MaxClients MaxClients not only lose its use as a flow control, but will have more serious consequences.

In this case, the best way to solve the problem is to increase the system's memory, or set it to MaxClients 175 (the above calculation) to ensure that only a QPS exceeds 175 when the 503 error occurs.

Java GC Expert series 4:apache maxclients settings and their impact on Tomcat full GC