Discussion on selection of large Java Web Systems

A jackson1225 user asked about the architecture and deployment selection of a large Web System in javaeye, hoping to improve the service capabilities of existing Java-based Web applications. Since the architecture mode and deployment are optimized, JavaCommunityHot Topic, which caused a lot of enthusiastic netizens to discuss, some of which have great guiding significance for other large web projects. At the beginning of the discussion, jackson1225 described the current application architecture and deployment scheme as follows:

The current system architecture is as follows:

The web layer is implemented using STRUTS + Tomcat. The entire system uses more than 20 Web servers, and its load balancing uses hardware F5;
The middle layer is implemented using the stateless Session Bean + Dao + helper class. Three WebLogic servers are deployed with multiple ejbs, and its load balancing is also implemented using F5;
The operations at the database layer are implemented by a general class written by the user. Two Oracle database servers store user information and business data respectively. One SQL Server database is the third-party business data information;
The web layer calls the EJB remote interface to access the middleware layer. The web layer first calls the corresponding EJB remote interface through the EJB interface information configured in an xml configuration file;

One operation in the system involves the access and operation of two Oracle databases and one SQL Server database, that is, three database connections, which are completed in one transaction.

In fact, many companies are using this architecture, because struts and tomcat are the most popular Java Web MVC frameworks and servlet containers, f5 Server Load balancer is a common solution for horizontal scaling (for example, session sticky configuration ). Because there are cross-Data Source transactions in this system, you can use the WebLogic Server EJB container and the database driver that supports two-phase commit to ensure transaction integrity across data sources (of course, distributed transactions managed by containers are not the only and optimal solution ).

However, with the popularity of rod Johnson's heavyweight book J2EE development without EJB and the Spring framework, the concept of lightweight frameworks and lightweight containers has become deeply rooted in the hearts of the people. As a result, most netizens have raised doubts about the scenario proposed by jackson1225, believing that this system has abused technology and is a waste of money. Most netizens think that slsb (stateless session bean) is completely unnecessary in this scenario. They think that slsb accessing local resources through a remote interface will have a great performance overhead, this is also the idea that rod Johnson criticized EJB 2 in without EJB. A large anti-pattern in X.

As javaee is a model-based solution that plays an important role in javaee, many industry experts are also wary of "anti-patterns). Jackson1225 immediately defended whether the above solution is against the pattern:

Our project is to use EJB as a facade, which is only provided to the remote interface called by the web layer, and only uses stateless session beans, so the performance is acceptable.

This explanation was quickly recognized by some netizens, but they soon realized that the architecture was determined by whether it could meet users' needs. davexin (probably a colleague from jackson1225) describes the users and concurrency of the system:

Now there are 40 million users, and we are about to merge them with the membership system of another company. This adds up to 90 million users. A single table contains more than 0.1 billion data records. This is the basic situation. In fact, I think the current architecture is still acceptable. Currently, about 5000 concurrent users are supported, and the system will be transformed to support 10 thousand concurrent users.

After the release of the specific concurrency, some netizens questioned the data and thought that the servlet container of the system supported a small number of concurrency, and wondered whether the configuration was not optimized enough. Davexin supplements the project's Server Configuration:

The front-end Tomcat of the system uses a blade, Which is configured with 2 GB memory and the CPU is about 2.0 GB. Each machine also supports-concurrent connections. If there are more than concurrent connections, the corresponding time will be very frequent, it took more than 20 seconds to reach a conclusion.

A user with ID cauherk made some pertinent comments. He did not propose an improvement solution based on the pure concurrent support capability of Web containers, however, some general improvement tips for similar applications are presented. The following is a summary:

Database pressure problems
You can configure databases based on business, region, and other features. You can consider database sharding, use RAC, partition, table sharding, and other policies to ensure normal transactions.

Transaction Problems
To operate in two databases, distributed transactions must be taken into account. You should carefully design your system to avoid using distributed transactions, so as to avoid more database pressure and other problems caused by distributed transactions. We recommend that you adopt a delayed commit policy (which does not guarantee data integrity) to avoid distributed transactions. After all, the probability of a commit failure is low.

Web optimization
The static, image independent use of different servers, for the normal static files, using E-TAG or client cache, many Google is doing this. For hotspot functions, fully loaded to the memory is considered to ensure absolute response speed. For hotspot data that requires frequent access, centralized cache is used (multiple Server Load balancer instances can be used ), reduce the pressure on the database.

Hardware-based compression schemes should be configured on L4 for almost all binary files to reduce network traffic. Improve user perception.

Network Problems
You can consider using images, multi-channel network access, and DNS-based load balancing. If you have enough investment, you can use CDN to reduce the pressure on your servers.

This analysis of cauherk is in place. The etags solution is a recent hot spot. infoq gave a detailed introduction to this solution in "using etags to reduce the bandwidth and load of Web Applications. Generally, the performance bottlenecks of database-centered web applications are all in the database. Therefore, cauherk puts the database and transaction issues to the first two to discuss. But davexin explains that in the project discussed, the database is not a bottleneck:

Our pressure is not on the database layer, but on the web layer and F5. At the peak, F5 was also killed, that is, more than 0.3 million clicks per second. The dynamic part of the Web cannot afford it. According to ourProgramRecord: 20 Web servers can withstand a maximum of 5000 concurrent requests. If there are more than one concurrent requests, Tomcat will not respond. It's like dead.

This reply will allow the following discussions to focus on the Performance Optimization of Web containers. However, Robbin, the webmaster of javaeye, expressed his opinion and introduced the topic back to the architecture of this project:

The most important thing about performance tuning is to locate the bottleneck and how it is generated.

I guess the bottleneck lies in the remote EJB method call!

The Java application on Tomcat needs to use the remote method call of EJB to access the stateless sessionbean on WebLogic. Such remote method calls are generally within ms ~ Ms or more. Without a remote method call, even if spring's dynamic reflection is adopted in large quantities, the completion time of a complete Web request processing in the local JVM is generally less than 20 ms. A Web request takes too long to execute, which will cause the servlet thread to take up more time and thus fail to respond to more subsequent requests in a timely manner.

If this assumption is true, I suggest that you remove ejbs since you have not used distributed transactions. WebLogic can also be removed. The service layer replaces EJB with spring. Instead of using a distributed architecture, deploy a complete layered structure on each Tomcat instance.

In addition, Apache also consumes memory and CPU to process static resources at high concurrency. you can replace it with lightweight Web servers such as Lighttpd, litespeed, and nginx.

The inference of Robbin is supported by netizens. davexin agrees with Robbin, But he explained that the company believes that giving up slsb is risky, therefore, the company tends to improve the user support capability of the system by replacing Tomcat with WebLogic Server 10. Robbin immediately criticized this practice:

Frankly speaking, I have never heard of a precedent for using EJB for large-scale Internet applications. The reason why large-scale Internet applications cannot use ejbs is that the performance of ejbs is too poor and there is almost a performance obstacle when using ejbs.

In the end, the performance of Web containers is nothing more than the servlet Thread Scheduling capability. Tomcat does not append n multiple management functions as WebLogic does, and runs fast and normal. Compare and test the performance of Weblogic database connection pool and c3p0 connection pool, and you will find that c3p0 is several times faster than WebLogic connection pool. This is not to say that WebLogic has poor performance, but WebLogic has to implement more features, so it will sacrifice a lot in terms of single speed.

Based on my experience, we can use tomcat5.5 or above to configure APR support and perform necessary tuning. If you use Bea jrockit JVM, on your current blade, supporting 500 concurrent jobs is completely feasible. With the current 20 blade hardware, it is okay to reach 10 thousand concurrency. Of course, the premise for doing so is to discard the EJB and put the web layer and the business layer inside the same JVM.

Next, Robbin also analyzes the test data of applications in the topic on Tomcat and WebLogic respectively for davexin:

Reference:
2. One weblogic10 Express (equivalent to one Tomcat for publishing JSP applications) and one weblogic10 (publishing EJB applications), supporting 1000 concurrent users ......
......
4. One tomcat4.1 and one weblogic8 can only support 350 concurrent users. Tomcat connection times out, indicating that this structure bottleneck lies in Tomcat.

This indicates that the bottleneck is not in the remote call of EJB, but the problem is gradually clear. Why does WebLogic support 1000 concurrent requests when acting as a Web container to initiate remote EJB calls, but Tomcat can only support 350 concurrent requests? There are only two possible reasons:

Your Tomcat is not properly configured, seriously affecting performance
The interface between Tomcat and WebLogic is faulty.
Then Jiangnan Baiyi, the initiator of the springside project, also proposed an overall optimization guide:

1. Basic Configuration Optimization

Tomcat 6? Tomcat parameter optimization?
Jrockit JVM? JVM parameter optimization?
How does Apache + squid process static content?

2. Business Layer Optimization

Some functions are localized without the remote session bean?
Asynchronous submit operation, JMS?
Cache hotspot data?

3. Presentation Layer Optimization

Dynamic Pages are released as static pages?
Some dynamic page content in cache?

Davexin adjusted the tomcat configuration and met the question about tomcat configuration by Robbin. davexin described the test results after configuration optimization:

After testing, the number of concurrent users can reach about 600, as Robbin said. If the number of concurrent users reaches 700, there will be about 15% failures, although after adjusting the above parameters, the number of concurrent users has increased, but the number of transactions completed in the same time has dropped by about 10%, and the response time has been delayed by about one second. However, in general, it is worthwhile to sacrifice a little transaction throughput and response time to increase the number of concurrent users by 500.

So far, this topic has a good result. This topic is not entirely meaningful for a specific project. More importantly, it refers to the ideas of netizens who solve the problem while analyzing and discussing the problem, in particular, the comments raised by several netizens, such as cauherk, Robbin, and Jiangnan Baiyi, can help Java Web project developers understand the key considerations for the architecture and deployment of medium and large projects. problem, it also eliminates some misunderstandings about the performance of lightweight servlet containers and EJB containers.