HTTP server-side-responsive programming for Java

Why respond to programming?

The traditional servlet model came to an end.

Traditional Java server programming follows the servlet specification of the EE, a threading-based model: Each HTTP request is handled by a single thread.

The disadvantage of the threading model is that each thread has to handle the read and write operations of the socket itself. For most requests, local processing of requests is fast and the request reads and returns are the most time consuming. This means that a large number of threads are wasted on remote connections and do not have the computing power to perform. However, it is important to note that threads are created with overhead, and each thread requires a separate memory resource. The-XSS parameter in the JVM is used to adjust the size of the thread stack. The total size of the JVM heap is limited to the-XMX parameter, so the number of threads that a running JVM server can run concurrently is fixed.

Even by adjusting the JVM parameters, it can run more multithreaded. However, the thread of the JVM is mapped to the user thread of the operating system, and the operating system can still only dispatch a limited number of threads. For example, the Linux system can refer to the discussion here: Maximum number of threads per process in Linux?.

In addition, when a large number of threads switch, it will also incur the overhead of context loading and unloading, which will also degrade the performance of the system.

Scalable IO

Doug Lea The Great God has a very classic pptscalable IO in Java that tells a better server model.

A scalable Network service system should meet the following conditions:

1. can increase load capacity as computing resources (CPU, memory, disk capacity, network bandwidth, and so on) increase.
2. Gracefully degrade when the network load increases beyond capacity to avoid direct crashes. For example, a denial of service to a request that exceeds the scope of competence, but a service is still available for requests within the scope of competence. When traffic peaks past, they still work.
3. Of course, high availability, high performance remains a must: for example, low response latency, requests for load changes, or release of compute resources.

The solution given by the author is reactor mode.

The reactor mode encapsulates the time-consuming IO resource as a handle object. The handle object is registered in the kernel of the operating system, and the handle object is processed when the object satisfies certain conditions (readable or writable). In reactor mode, the synchronous multiplexer is responsible for handling state changes to the handle object, and when the condition is met, the callback function provided when the handle object is registered is called.

The synchronous multiplexer handles IO links exclusively in a separate thread. After the request has been read, the task is submitted to the worker thread pool to complete the decoding, processing, encoding, and so on, and finally the multiplexer will be responsible for returning the result to the client, while the pool thread continues to process the next task. Compared to the way JDK1.5 has created a new thread for each request, the thread pool can be reused, reducing the overhead of creating recycled threads, and performing better when dealing with dense computational loads. At the same time, deploying a synchronous multiplexer on multiple threads can also make better use of the processing power of multicore CPUs.

In this way, the task division of the threads is clear, specifically dealing with IO-intensive tasks and specifically dealing with CPU-intensive tasks.

NiO is difficult to popularize

From the earliest select to later Linux Epoll and BSD Kqueue, the multiplexing performance of the operating system has been continuously enhanced.

JDK 1.4 Introduces the NIO module, which masks the details of the operating system layer and makes the same package for each system's Multiplexed API. The JDK NiO has the following core components:

• Buffer, a data container that is fixed when the capacity is created
• Charsets, responsible for the data codec work
• Channels, abstraction of remote connections
• Selector, multiplexed Selector

In a world outside the JVM, multiplexing through Nginx, the V8 engine-based node. JS, has long been a reality. But Java NiO is slow to develop in the production environment. For example, Tomcat did not completely remove the Bio connector until the release of version 8.5 in 2016, fully embracing nio.

The main problems with JDK NiO are that they are more troublesome:

1. First, NiO. To improve data transmission and delivery performance, you can create Directbuffer objects. The memory of the object is outside the JVM heap and cannot be reclaimed by a normal GC collector, but only when the full GC is triggered in the old age of the JVM. The full-volume GC often causes the system to lag and reduce the response efficiency. If you passively wait for the old age zone to trigger the full GC itself, it is possible to cause out-of-heap memory overflow. The contradiction between the two needs to be carefully balanced at the time of development.
2. Next is, JDK1.8 still exists epoll BUG: If selector poll result is empty, there is no wakeup or new message processing, there is a null polling, CPU utilization 100%.

Netty is the level of NIO.

As a third-party framework, Netty did what the JDK was supposed to do.

Netty's data container bytebuf is even more excellent .

BYTEBUF maintains two indexes simultaneously: Read index and write index. This ensures that the container object can be simultaneously adapted to both read and write scenarios. The NIO buffer, however, needs to perform a flip operation to accommodate the switching of the read-write scene. While bytebuf containers are managed manually using reference counts, you can reclaim memory by calling Jdk.internal.ref.Cleaner with reflection when the reference count is zero, to avoid leaks. When the GC is inefficient, it's no problem to choose to use the manual way to manage memory.

Netty has a higher API footprint .

Observe the Netty official website tutorial The demo, as long as dozens of lines of code to complete a reactor mode of the server. The Serverbootstrap group method defines how the main socket and child sockets are handled, using the Nioeventloopgroup class as the Java NIO + multithreading implementation. The solution for NiO's Epoll Bug,nioeventloopgroup is the Rebuildselectors object method. This method allows the new selector to be rebuilt when the selector fails, releasing the old one. In addition, Netty has implemented its own epolleventloopgroup through JNI, bypassing the NIO version of the bug.

Netty uses the responsibility chain mode to realize the processing of the server inbound and outbound messages, so that the server code can be better extended and maintained.

Netty is widely used in the production field, and the communication components of Hadoop Avro, Dubbo, ROCKETMQ, Undertow and other products used in the production field have chosen Netty as the basis and have withstood the test.

Netty is an excellent asynchronous communication framework, but is primarily used in the underlying components. Because Netty exposes a large amount of detail to the developer, the development of the business system is still troubled, so it cannot be further popularized.

Let me give you an example. Netty uses Channelfuture to receive incoming requests. Compared to its inherited java.util.concurrent.Future, Channelfuture can add a set of Genericfuturelistener to manage object state, avoiding repeated queries about the state of the future object. It's a progress. However, these listener have brought another problem--callback hell. Nested callback codes are often difficult to maintain.

For callback hell, what can we do

Netty is good at making a good base component. Business-level issues let us address the business-level API.

Poor adaptability of the Java API JDK7 previous asynchronous code is difficult to organize

In JDK7 and before, Java Multi-threading programming tools are mainly thread, Executorservice, future and related synchronization tools, the implementation of the code is cumbersome, and performance is not high.

Thread

For example A, consider a scenario with X, P, Q three logic to execute, where X's execution needs to be completed together with P, Q to start execution.

If you use thread, the code will look like this:

/ * Create thread * /Thread A =NewThread (NewRunnable () {@Override     Public void Run() {/ * p logic * /}}); Thread B =NewThread (NewRunnable () {@Override     Public void Run() {/ * Q logic * /}});/ * Start thread * /A.Start(); B.Start();/* Wait for a, b thread execution to end */Try{A.Join(); B.Join();}Catch(Interruptedexception e) {e.Printstacktrace();}/ * Start execution of x logic * /Thread C =NewThread (NewRunnable () {@Override     Public void Run() {/ * x Logic * /}}); C.Start();...

The above code, regardless of the cost of thread creation, from the formal perspective, the execution logic inside the thread, the scheduling logic of the thread itself, and the exception handling logic that must be captured by the interruptedexception are mixed together and the whole is chaotic. Imagine that the code is more difficult to maintain when the business logic fills it.

Threadpoolexecutor, future

Threadpoolexecutor and future help with thread reuse, but the specification of code logic doesn't help.

Executorservice pool = executors.Newcachedthreadpool(); Future<?> a = pool.Submit(NewRunnable () {@Override     Public void Run() {/ * p logic * /}}); Future<?> B = Pool.Submit(NewRunnable () {@Override     Public void Run() {/ * Q logic * /}});/ * Get thread execution Results* Still catching exceptions, handling logic */Try{A.Get(); B.Get(); Future<?> C = Pool.Submit(NewRunnable () {@Override         Public void Run() {/ * x Logic * /}    });}Catch(Interruptedexception e) {e.Printstacktrace();}Catch(Executionexception e) {e.Printstacktrace();}

JDK8 code readability has been significantly improved

JDK8 draws on a considerable number of functional programming features, providing a few very hand-called tools.

Completeablefuture and Forkjoinpool

If you want to use Completeablefuture to implement the previous example, you can write this.

CompletableFuture<?> a = CompletableFuture.runAsync(() -> {    /* P逻辑 */}).exceptionally(ex -> {    /* 异常处理逻辑 */    return ...;});CompletableFuture<?> b = CompletableFuture.runAsync(() -> {    /* Q逻辑 */});CompletableFuture<?> c = CompletableFuture.allOf(a, b).thenRun(() -> {    /* X逻辑 */});

With the addition of lambda expressions, the code in the example is mainly based on thread internal logic, and the dispatch logic is visually displayed by means of AllOf (), Thenrun (), and so on. In particular, the optional exception capture logic, which makes the code readability is greatly improved.

It is important to note that completablefuture can be executed using the specified executorservice. If the Executorservice object is not specified as in the example above, it is executed by default using the static object Commonpool in Forkjoinpool. Forkjoinpool.commonpool, as the only object in a JVM instance, is also the executor of stream concurrency, so try to ensure that the logic in the Completablefuture does not block the thread. If this is not possible, you can use Managedblocker to reduce the impact.

Forkjoinpool is a JDK1.7-provided concurrency thread pool that can handle computationally intensive concurrency tasks well, especially for tasks that can be "split-and-conquer". Traditional threadpoolexecutor need to specify the number of threads constructor, and Forkjoinpool uses a similar but more resilient concept-"concurrency." Concurrency refers to the number of active threads in the pool. For possible blocking tasks, Forkjoinpool designed a Managedblocker interface. When the pool thread executes to the ForkJoinPool.managedBlock(ForkJoinPool.ManagedBlocker blocker) method, the thread pool creates new threads to perform other tasks in the queue and polls the object's Isreleasable method to determine whether the recovery thread continues to run. JDK1.7 in the Phaser class source used to this method.

About the use of completeablefuture, recommend to see this blog: Understand Completablefuture, summed up very well.
For Forkjoinpool, take a look at this blog: Java concurrency Programming Note: How to use Forkjoinpool and how it works.

Stream

Stream flow is also a good programming tool introduced by JDK8.

Stream objects are usually constructed by iterator, collection. You can also use the Streamsupport stream static method to create an instance of a custom behavior.

The stream stream object uses a chain programming style that can develop a series of custom behaviors for convection, such as filtering, sequencing, transforming, iterating, and finally producing results. Look at an example.

List<Integer> intList = List.of(123);List<String> strList = intList.stream()        .filter(k -> k>1)        .map(String::valueOf)        .collect(Collectors.toList());

In the above code, Intlist gets the stream object through the stream method, then filters out elements greater than 1, generates a string object through the valueof static method of string, and finally collects each string object as a list strlist. Just like the Completablefuture method name, the stream's method names are self-describing, making the code very readable.

In addition, the stream stream is computationally inert. The stream stream object is broadly divided into two ways:

• Intermediate methods, such as filter, map, and other convection changes
• Finalization methods such as Collect, foreach, and so on can end the stream

The calculation of the stream will only actually execute when the finalization method is executed. The previous intermediate method is recorded as a step, but no modification is performed in real time.

If the stream method in the example is modified to Parallelstream, then the resulting stream object is a concurrent stream and is always executed in Forkjoinpool.commonpool.

About stream, we highly recommend Brian Goetz's series of articles Java Streams.

Just a little bit more.

Forkjoinpool is a powerful thread pool component that, when used properly, always maintains a reasonable level of concurrency and maximizes compute resources.

But, completeablefuture or stream, they all have the same problem: Unable to adjust the front-end call pressure by changing the load on the backend pool . For example, When the back-end Forkjoinpool.commonpool is in full operation and there are a lot of tasks queued in the queue, the newly submitted task is likely to have a high response delay, but the front-end completeablefuture or stream has no way to get such a state to delay the task's submission. This is a violation of the "responsiveness" of the "responsive system" requirement.

Gospel reactive from third-party APIs Streams

Reactive streams is a set of criteria that defines the line that should be implemented by a responsive programming framework that runs on the JVM platform
For.

The reactive streams specification derives from the "observer pattern". The logical flow that is dependent before and after is disassembled into events and subscribers. Only when an event is changed does the observer of interest follow the subsequent logic. So the idea of reactive stream and the stream of the JDK is a little closer, both of which focus on controlling the flow of data. Splitting a tightly coupled logical flow into a "subscribe-to-publish" approach is actually a big step forward. The code becomes more maintainable, and it's easy to use message-driven mode disassembly as your business needs.

The reactive streams specification defines four interfaces:

• Publisher, which is responsible for producing the data stream, each Subscriber calls the Subscribe method to subscribe to the message.
• Subscriber is the Subscriber.
• Subscription, in fact, is an order option, which is equivalent to a menu in a restaurant that is passed to subscribers by the Publisher.
• Processor, in the middle of the data stream, is the subscriber and the producer of the new data stream.

When Subscriber calls the Publisher.subscribe method to subscribe to a message, Publisher invokes subscriber's Onsubscribe method and callbacks a subscription menu.

The subscription menu consists of two choices:

1. One is the request method, the data flow requests, the parameter is the number of data flows requested, the maximum is long.max_value;
2. Another is the Cancel method, which cancels the data flow subscription, and it is important to note that the data stream may continue to be sent for a period of time to satisfy the previous request invocation.

A subscription object can only be called by the same subscriber, so there is no problem with object sharing. Therefore, even if the subscription object is stateful, it does not compromise the thread safety of the logical link.

Subscriber Subscriber also needs to define three kinds of behaviors:

1. OnNext, accepts the execution logic after the data stream;
2. OnError, how to respond when the issue is wrong;
3. OnComplete the behavior of a subscribed data stream after it has been sent.

Comparing business logic and exception handling logic with the future, thread, subscriber defines it in three different ways, and the code is clearer. Java.util.Observer (start discarding in JDK9) only defines the Update method, which is equivalent to the OnNext method here, the lack of management of the overall convection and the handling of exceptions. In particular, if the exception is passed along with the call chain, debugging positioning can be cumbersome. Therefore, we should pay attention to the OnError method and handle this exception within the subscriber as much as possible.

Although the reactive streams specification and stream are concerned with data flow, there is a significant difference between the two. That is, stream is based on the production side, how much the producer has, and how much data stream the stream creates. and the reactive streams specification is based on the consumer. Downstream of the logic chain, you can notify the upstream of the speed of the production data flow by changing the request method parameters. Thus, the "sensitivity" requirement of "responsive system" is realized. This is described in responsive programming, using the term "back pressure".

The reactive streams specification is only a standard, and its realization relies on the results of other organizations. The meaning is that the realization of each family can be called through such a unified interface, which will help the benign development of the response framework ecology. Reactive STREAMS specifications, although the Netflix, Pivatol, Redhat and other third-party manufacturers launched, but with the release of JDK9 incorporated into the official API, located in Java.util.concurrent.Flow. JDK8 can also integrate the corresponding module calls directly into the project.

By the way, JDK9 the official documents given by the demo in the data stream actually produced from the subscription, frighten me repeatedly confirmed reactive streams official norms.

RxJava2

Rxjava is maintained by NETFILX and implements the Reactivex API specification. The specification is implemented in many languages and is rich in ecology.

The RX paradigm was first implemented by Microsoft on the. NET platform. In November 2014, NETFILX ported RX to the JVM platform and released a stable version of 1.0. The reactive streams specification was first released in 2015, only to form a stable version 2017 years later. So the Rxjava 1.x and reactive streams specifications have a lot of discrepancy. The 1.x version iterates through the 1.3.8 version of March 2018, announcing the discontinuation of maintenance.

Netflix released the 2.0.1 stable version in November 2016, enabling compatibility with the reactive streams specification. 2.x is now the official recommended version.

These concepts are mainly found in the Rxjava framework:

• Observable and Observer. Rxjava directly re-uses the concept of the "observer pattern", which helps to be accepted by the development community more quickly. Observeble and publisher are a little different: the former has a "hot and cold" distinction, "cold" means that only the time of the subscription to publish the message flow, "hot" indicates that the message flow is published with the time of the object is irrelevant. Publisher is more like a "cold" observeble.
• Operators, the operator. Rxjava and JDK stream are similar, but more self-describing function methods are designed, and chained programming is also implemented. These methods include, but are not limited to, conversion, filtering, binding, and so on.
• Single, is a special kind of observable. A general observable can generate data streams, and single produces only one data. So single does not need a OnNext, OnComplete method, but a onsuccess instead.
• Subject, note that this is not an event, but an intermediary object between observable and observer, similar to streams in the reactive processor specification.
• Scheduler, a type of thread pool that handles concurrent tasks. Rxjava default execution on the main thread, you can invoke the blocking task asynchronously by means of the Observeon/subscribeon method.

RxJava 2.x in the Zuul 2, Hystrix, Jersey and other projects have been used in the production field has been applied.

Reactor3

Reactor3 has pivotal to develop maintenance, that is, Spring's brother.

On the whole, the concepts and Rxjava in the REACTOR3 framework are similar. Mono and flux are identical to Rxjava's single and observable. Reactor3 also uses self-describing operator functions for chained programming.

The RxJava 2.x supports the JVM 6+ platform and is friendly to older projects, while REACTOR3 requirements must be jvm8+. So, if it's a new project, it's better to use REACTOR3 because it uses a lot of new APIs, supports a lot of functional interfaces, and makes code readability more maintainable.

Backed by the spring tree, Reactor3 's design target is the server-side Java project. Reactor community for the server side, and constantly introduce new products, such as Reactor Netty, Reactor Kafka and so on. But if it's an Android project, RXJAVA2 is more appropriate (suggestions from the REACTOR3 official documentation).

To be honest, Reactor3 's documentation is more informative.

What is a responsive system

It is clear in the response declaration that a responsive system should be:

• Responsive: able to respond in a timely manner
• Resilient: can recover itself and generate a response even if a fault is encountered
• Scalable: Ability to invoke or release computing resources as workloads change, as well as to adjust workload capabilities as computing resources change
• Message-driven: Explicit message delivery enables the decoupling of various components of the system, and various components manage resource scheduling on their own.

Building a responsive web system vert.x

Vert.x is currently maintained by the Eclipse Foundation, creating a comprehensive set of responsive web-system development environments, including database management, Message Queuing, microservices, authorization authentication, Cluster Manager, DevOps, and more.

Based on Netty Development, the Vert.x core framework is an event-driven framework that invokes its corresponding handler whenever an event is feasible. In Vert.x, there is a dedicated thread responsible for calling handler, called EventLoop. Each of the Vert.x instances maintains multiple eventloop.

The Vert.x core framework has two important concepts: Verticle and Event Bus.

Verticle

Verticle is similar to the actor role of the Actor model.

What is actor?

Let's talk about it in general.

The actor model is mainly aimed at distributed computing systems. Actor is one of the most basic computational units. Each actor has a private message queue. The communication between actors relies on sending messages. Each actor can do this in parallel:

1. Send a message to another actor
2. Create a new actor
3. Specifies the behavior when the next message is received

The message passing between the verticle depends on the event Bus that follows.

Vert.x provides high-availability features for verticle deployments: in vert.x clusters, if the Veticle instances running on one node fail, the other nodes redeploy a new verticle instance.

Verticle is just a scheme provided by Vert.x and is not mandatory to use.

Event Bus

Event Bus is the central system of vert.x framework, which can realize the message passing and handler registration and logoff of each component in the system. Its messaging supports both "subscription-release" mode and "request-response" mode.

When multiple Vert.x instances form a cluster, the event bus of each system can form a unified distributed event bus. Each event bus node communicates with each other through the TCP protocol and does not rely on cluster Manager. This is a component that enables node discovery and provides distributed infrastructure components (locks, counters, maps), and so on.

Spring Webflux

One of the highlights of SPRING5 is the responsive framework spring Webflux, developed using its own Reactor3, but also supports Rxjava.

The default server-side container for Spring Webflux is reactor Netty, and the version of the Servlet 3.1 non-blocking API interface can be implemented using Undertow or Tomcat, jetty. Spring Webflux implements an adapter (Adapter) that interacts with the reactive streams spec implementation framework, respectively, for these containers, without exposing the servlet API to the user layer.

Spring Webflux's annotations are similar to spring MVC. This helps the development team adapt quickly to the new framework. And the spring Webflux is compatible with Tomcat, Jetty, and helps the stability of the project operation.

But if it's a new project, a new team, give it to me, I'll probably choose Vert.x because the event bus is really appealing.

Resources

• Reactive programming with Reactor
• Netty High Performance and NiO epoll empty polling bug
• Problems with JAVA NiO
• Reactor mode
• Netty Combat
• Guide to the Fork/join Framework in Java
• Java ' s Fork/join vs Executorservice-when to use Which?

Extended Reading

• Five ways to maximize Java NIO and nio.2
• Forkjoinpool configuration of Commonpool related parameters
• Is there anything wrong with using I/O + managedblocker in Java8 parallelstream ()?
• Can I Use the work-stealing behaviour of Forkjoinpool to avoid a thread starvation deadlock?

HTTP server-side-responsive programming for Java