Netty Source-One, server startup (1)

Netty as a network component in the Java ecosystem has a pivotal position, all kinds of open source middleware use Netty for network communication, such as Dubbo, ROCKETMQ. It can be said that Netty is the encapsulation of Java NIO, such as the encapsulation of BYTEBUF, channel, etc. to make network programming easier.

There are two things to know before you start the introduction of the Netty server:

1. Reactor Threading Model
2. IO multiplexing in Linux

Reactor Threading Model

For reactor threading Model please refer to this article, the corresponding three kinds of reactor threading model can be implemented by different configuration Netty

• Reactor single threading model
• Reactor multithreaded Model
• Reactor master-Slave multithreading model

 // reactor单线程模型，accept、connect、read、write都在一个线程中执行EventLoopGroup group = new NioEventLoopGroup(1);bootStrap.group(group);// reactor多线程，accept在bossGroup中的一个线程执行，IO操作在workerGroup中的线程执行EventLoopGroup bossGroup = new NioEventLoopGroup(1);EventLoopGroup workerGroup = new NioEventLoopGroup();bootStrap.group(bossGroup , workerGroup);// reactor主从多线程，用来accept连接的是在一个线程池中执行，这个时候需要bind多个port，因为Netty一个bind的port会启动一个线程来acceptEventLoopGroup bossGroup = new NioEventLoopGroup(2);EventLoopGroup workerGroup = new NioEventLoopGroup();bootStrap.group(bossGroup , workerGroup);

注意：本文后面的介绍如无特别说明都是基于reactor多线程模型

IO multiplexing in Linux

The network programming model in Linux is also evolving, following the sequential sequence of evolution (refer to Chapter 4th, 6 of the third edition of the UNIX Network programming Volume 1: Socket networking API)

Accept

Block waits for a connection, receives a new connection after the newly started thread to process the received connection, and then blocks in the new thread waiting for new data to arrive

Select

There are three different cases depending on the entry parameter.

1. Wait forever until the listener descriptor has an arbitrary IO event to return
2. Wait for a fixed period of time, if the time before the IO event is returned in advance, otherwise wait timeout after return
3. Do not wait, check the descriptor immediately after returning, called polling

Select returns the number of ready file descriptors, polls all sockets, determines the status of each socket to determine if there is an event, what event

Poll

Compared to the selectpoll is blocking wait, only read and write events will be returned, return the number of sockets that have read and write events, and the corresponding socket event set, you find a specific socket from all sockets

Epoll

Compared to Poll,epoll, you can return only those descriptors that do have an IO event, with only a small number of active connections under large concurrency.

The advantage of more poll

1. No need for developers to re-prepare a set of file descriptors (more poll than simple)
2. No need to traverse the descriptor of all listeners, as long as the descriptor collection of the ready queue is traversed by the asynchronous wake of the kernel IO event

The implementation of Java NiO in Linux is based on Epoll. Programming model for Epoll:

1. To create a Socket,socket method
2. Bind Server Ip,port,bind method
3. The listener binds the Ip:port file descriptor, and the Listen method
4. Create a Epoll handle (file descriptor), configure the maximum number of listener file descriptors, Epoll_create method
5. Configure Epoll Listener event for file descriptor: Register, modify, delete event corresponding to a file descriptor
6. Listen for all configured descriptors, epoll_wait
7. When there are new events, iterate over the returned descriptor and handle the corresponding event
8. If it is a connection from the client, register the file descriptor to Epoll
9. If a read-write event is processed separately

注意：Netty封装的Java NIO是跨平台的，后面还是以linux平台为例来介绍

Next, let's take a look at what the Netty server startup process does. Netty as a network framework, and common network programming to do things basically the same, corresponding to the above Epoll programming model, Netty the boot process for

1. Initialize thread pool, initialize selector
2. Initialize Nioserversocketchannel
3. Binding Server Ip:port
4. Registering the Nioserversocketchannel with the selector
5. Configuring Nioserversocketchannel-Monitored Events
6. Use Selector.select to wait for a new IO event
7. If the connection is from the client, register the Niosocketchannel with the selector (if the new thread is the new selector)
8. If it is a normal IO event, it is handled in the worker thread

Thread pool Initialization

Take a look at Nioeventloop before introducing Nioeventloopgroup.

You can see that Nioeventloop inherits from Singlethreadeventexecutor, is a single-threaded executor that loops through the thread to listen for IO events. The main methods are

// 初始化selectorio.netty.channel.nio.NioEventLoop#openSelector// 将channel注册到selectorio.netty.channel.nio.NioEventLoop#register// 监听selector上的事件io.netty.channel.nio.NioEventLoop#select

A nioeventloop Initializes a selector to handle the channel registered on the selector.

Nioeventloopgroup from the name can be seen to be composed of multiple Nioeventloop, class diagram as follows

The important properties of the Nioeventloopgroup are:

// 包含的EventExecutor数组private final EventExecutor[] children;// 选择哪一个EventExecutor执行task的选择器，不同的选择器有不同的策略private final EventExecutorChooserFactory.EventExecutorChooser chooser;

Important methods are:

// 选择下一个执行任务的线程io.netty.util.concurrent.MultithreadEventExecutorGroup#next// 创建EventLoopio.netty.channel.nio.NioEventLoopGroup#newChild// 在线程池中执行注册channel的任务io.netty.channel.MultithreadEventLoopGroup#register(io.netty.channel.Channel)// 创建默认的threadFactoryio.netty.channel.MultithreadEventLoopGroup#newDefaultThreadFactory

The thread pool initializes the code for

EventLoopGroup workerGroup = new NioEventLoopGroup();

If you use an argument-free construction method, you'll end up with a constructor that does the following:

• If executor is not initialized, use the default executor initialization
• Initialize each eventloop in the thread pool
• If an exception is thrown during one of the initialization processes, close all Nioeventloop

protected Multithreadeventexecutorgroup (int nthreads, Executor Executor, EVENTEXECU Torchooserfactory chooserfactory, Object ... args) {if (nthreads <= 0) {throw new IllegalArgumentException (S    Tring.format ("Nthreads:%d (expected: > 0)", nthreads));    } if (executor = = null) {executor = new Threadpertaskexecutor (Newdefaultthreadfactory ());    } children = new Eventexecutor[nthreads];        for (int i = 0; i < nthreads; i + +) {Boolean success = false;            try {//create EventLoop children[i] = newChild (executor, args);        Success = true; } catch (Exception e) {///todo:think about if the is a good Exception type throw new Illegalstate        Exception ("Failed to create a child event loop", E); } finally {if (!success) {for (int j = 0; J < i; J + +) {Children[j].shu                Tdowngracefully ();     }           for (int j = 0; J < i; J + +) {Eventexecutor e = children[j]; try {while (!e.isterminated ()) {e.awaittermination (Integer.max_value,                        Timeunit.seconds); }} catch (Interruptedexception interrupted) {//Let the caller handle the inte                        Rruption.                        Thread.CurrentThread (). interrupt ();                    Break }}}}}//Initialize chooser, decide to select the next thread's policy chooser = Chooserfactory.newchooser (childre    n); Final futurelistener<object> Terminationlistener = new futurelistener<object> () {@Override Publ IC void Operationcomplete (future<object> future) throws Exception {if (Terminatedchildren.incrementandge            T () = = Children.length) {terminationfuture.setsuccess (null);    }        }    }; For (Eventexecutor e:children) {e.terminationfuture (). AddListener (Terminationlistener);    } set<eventexecutor> Childrenset = new linkedhashset<eventexecutor> (children.length);    Collections.addall (Childrenset, children); Readonlychildren = Collections.unmodifiableset (Childrenset);}

Using the default parameters to construct the arguments, the values of the arguments to the above constructor are

Nthreads

// 默认的线程池大小private static final int DEFAULT_EVENT_LOOP_THREADS;static {    // 如果配置了io.netty.eventLoopThreads参数的话，先取该参数的值    // 如果没有配置上面的参数，则取机器处理器个数的2倍    // 如果上面算出的结果小于1则取1    DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt(        "io.netty.eventLoopThreads", Runtime.getRuntime().availableProcessors() * 2));    if (logger.isDebugEnabled()) {        logger.debug("-Dio.netty.eventLoopThreads: {}", DEFAULT_EVENT_LOOP_THREADS);    }}// 默认没有指定线程池大小，取DEFAULT_EVENT_LOOP_THREADSprotected MultithreadEventLoopGroup(int nThreads, Executor executor, Object... args) {    super(nThreads == 0 ? DEFAULT_EVENT_LOOP_THREADS : nThreads, executor, args);}

Executor

Executor is not specified by default, null

Chooserfactory

protected MultithreadEventExecutorGroup(int nThreads, Executor executor, Object... args) {    this(nThreads, executor, DefaultEventExecutorChooserFactory.INSTANCE, args);}// io.netty.util.concurrent.DefaultEventExecutorChooserFactory

Using the default Chooser, the primary function of the class is to provide a policy for selecting the next thread

Public final class Defaulteventexecutorchooserfactory implements Eventexecutorchooserfactory {//singleton public static F    Inal defaulteventexecutorchooserfactory INSTANCE = new Defaulteventexecutorchooserfactory (); Private Defaulteventexecutorchooserfactory () {} @SuppressWarnings ("Unchecked") @Override public Eventexecutorcho Oser Newchooser (eventexecutor[] executors) {if (Ispoweroftwo (executors.length)) {//If it is a power of 2, use this choose        R return new Poweroftoweventexecutorchooser (executors);        } else {return new genericeventexecutorchooser (executors);     }} private static Boolean ispoweroftwo (int val) {//Determines whether a number is a power of 2, the method is cleverly return (val &-val) = = val; } private static Final class Poweroftoweventexecutorchooser implements Eventexecutorchooser {private final        Atomicinteger idx = new Atomicinteger ();        Private final eventexecutor[] executors; Poweroftoweventexecutorchooser (eventexecutor[] Executors) {this.executors = executors; } @Override public Eventexecutor next () {///if it is a power of 2 threads, you can use the bitwise operation to calculate the next selected thread's index retur        n executors[idx.getandincrement () & Executors.length-1]; }} private static Final class Genericeventexecutorchooser implements Eventexecutorchooser {private final at        Omicinteger idx = new Atomicinteger ();        Private final eventexecutor[] executors;        Genericeventexecutorchooser (eventexecutor[] executors) {this.executors = executors; } @Override public Eventexecutor next () {//uses the redundancy method to calculate the index return of the next thread Executors[ma        Th.abs (idx.getandincrement ()% executors.length)]; }    }}

It can be seen that the final results calculated by the above two chooser are consistent, but using bit arithmetic is a bit faster, so if the thread pool size is just 2 power, then the chooser of the bitwise operation is used.

Args

// args[0]，下面方法返回的provider，在linux平台上默认是EPollSelectorProviderjava.nio.channels.spi.SelectorProvider#provider// args[1]，决定eventLoop每次执行select还是执行队列中的任务io.netty.channel.DefaultSelectStrategyFactory// args[2]，等待队列满以后的拒绝策略io.netty.util.concurrent.RejectedExecutionHandlers#REJECT

The initialization of the nioeventloopgroup process is primarily to initialize each nioeventloop in the thread pool, and each nioeventloop contains a selector.

Initialize Selector

And then the above-mentioned initialization nioeventloop, call the Newchild method to initialize the

io.netty.channel.nio.nioeventloopgroup#newchildprotected eventloop newChild (Executor Executor, Object ... args)                            Throws Exception {///The following parameters have been described above for return new Nioeventloop (this, executor, (Selectorprovider) args[0], ((selectstrategyfactory) args[1]). Newselectstrategy (), (Rejectedexecutionhandler) args[2]);} Nioeventloop (nioeventloopgroup parent, Executor Executor, Selectorprovider Selectorprovider, Selectstrategy St Rategy, Rejectedexecutionhandler Rejectedexecutionhandler) {//Call the parent class constructor method to initialize the Taskqueue,taskqueue size to take Math.max (16,    Maxpendingtasks) Super (parent, executor, false, Default_max_pending_tasks, Rejectedexecutionhandler);    Check Selectorprovider if (Selectorprovider = = null) {throw new NullPointerException ("Selectorprovider");    }//Verifies whether the select policy executed by EventLoop is empty if (strategy = = null) {throw new NullPointerException ("Selectstrategy");    } Provider = Selectorprovider; Initialize Selector selector = OpeNselector (); Selectstrategy = strategy;}    Private Selector Openselector () {final Selector Selector; try {//call is Sun.nio.ch.epollselectorprovider#openselector//return is Sun.nio.ch.EPollSelectorImpl Select    or = Provider.openselector ();    } catch (IOException e) {throw new Channelexception ("Failed to open a new selector", e);    }//whether to use Selectedselectionkeyset optimization, the default is to use False if (disable_keyset_optimization) {return selector;    }//Netty-Optimized final selectedselectionkeyset Selectedkeyset = new Selectedselectionkeyset (); An attempt was taken to get the Selectorimpl object, followed by the properties of this class using reflection operations Object Maybeselectorimplclass = accesscontroller.doprivileged (new Privilegedaction<object> () {@Override public Object run () {try {return Cl Ass.forname ("Sun.nio.ch.SelectorImpl", false, Platformdependent.            Getsystemclassloader ()); } catch (ClassNotFoundException E) {return e;            } catch (SecurityException e) {return e;    }        }    }); Ensure that you have access to the class if (! (        Maybeselectorimplclass instanceof Class) | |        Ensure the current selector implementation are what we can instrument. ! ((class<?>) maybeselectorimplclass). IsAssignableFrom (Selector.getclass ())) {if (Maybeselectorimplclass Insta            Nceof Exception) {Exception e = (Exception) maybeselectorimplclass;        Logger.trace ("Failed to instrument a special java.util.Set into: {}", selector, E);    } return selector;    } final class<?> Selectorimplclass = (class<?>) maybeselectorimplclass; Object maybeexception = accesscontroller.doprivileged (new privilegedaction<object> () {@Override publi C Object Run () {try {//Get fields Selectedkeys field Selectedkeysfield = Selectorimpl           Class.getdeclaredfield ("Selectedkeys");     Get fields publicselectedkeys Field Publicselectedkeysfield = Selectorimplclass.getdeclaredfield ("PublicSe                Lectedkeys ");                Selectedkeysfield.setaccessible (TRUE);                Publicselectedkeysfield.setaccessible (TRUE); Set Selectedkeys, Publicselectedkeys to Netty custom Selectedselectionkeyset selectedkeysfield.set (selector, sel                Ectedkeyset);                Publicselectedkeysfield.set (selector, selectedkeyset);            return null;            } catch (Nosuchfieldexception e) {return e;            } catch (Illegalaccessexception e) {return e; } catch (RuntimeException e) {//JDK 9 can throw an inaccessible object exception here; since Netty Compil ES//against JDK 7 and this exception is only added in JDK 9, we had to weakly//check T The He type if ("Java.lang.reflect.InaccessibleObjectException". Equals (E.getclass (). GetName())) {return e;                } else {throw e;    }            }        }    });        if (maybeexception instanceof Exception) {selectedkeys = null;        Exception e = (Exception) maybeexception;    Logger.trace ("Failed to instrument a special java.util.Set into: {}", selector, E);        } else {selectedkeys = Selectedkeyset;    Logger.trace ("instrumented a special Java.util.Set into: {}", selector); } return selector;}

Several things were done during the initialization of selector:

• Using the platform-related provider to initialize the corresponding Selectorimpl, the Java SPI is used to load the platform-related provider, and each provider corresponds to a Selectorimpl
• If Selectedkey optimizations are not disabled, Netty replaces Selectorimpl Publicselectedkeys, Selectedkeys with custom Selectedselectionkeyset

Description of the Selectorimpl.selectedkey optimization

1. Using reflection to replace Selectorimpl.selectedkey with Selectedselectionkeyset,selectedselectionkeyset using an array to implement elements to store
2. When the Select method is called, Selectedselectionkeyset#add is called when an event comes in, adding the selectkey of the IO event to the keyset
3. Using array traversal (processselectedkeysoptimized) is faster than using SET traversal, the first reference article after the reference text
4. After Java9, this optimization was invalidated because JAVA9 introduced the jigsaw

Next look at the selector creation process, which calls Epollselectorprovider#openselector to start initializing selector

Public Abstractselector Openselector () throws IOException {//Direct new one Epollselectorimpl return new Epollselectorimp L (this);} The construction method can only be used within the package for provider to call Epollselectorimpl (Selectorprovider sp) throws IOException {// Call the constructor of the parent class Selectorimpl initialize Selectedkeys, Publickeys, Publicselectedkeys//above already said, if using Netty optimization, Publickeys,    Publicselectedkey will be replaced by super (SP);    Call the Linux pipe method to create a pipeline that is configured as a non-blocking long Pipefds = Ioutil.makepipe (false);    High 32 is read file descriptor fd0 = (int) (Pipefds >>> 32);    Low 32 bits are write file descriptors fd1 = (int) Pipefds;    Epollarraywrapper contains a series of native methods to invoke Epollarraywrapper.c Local method pollwrapper = new Epollarraywrapper ();    Pollwrapper.initinterrupt (fd0, FD1); Fdtokey is used to save the mapping of file descriptors and Selectionkeyimpl Fdtokey = new hashmap<> ();} Epollarraywrapper () throws IOException {//creates the Epoll file descriptor//create Epoll filename Descriptor EPFD = epollcreate    (); The epoll_event array passed to epoll_wait int allocationsize = num_epollevents * Size_epollevenT    Pollarray = new Allocatednativeobject (allocationsize, true);    Pollarrayaddress = Pollarray.address (); Eventhigh needed when using file descriptors > 64k if (Open_max > max_update_array_size) eventshigh = n EW hashmap<> ();}

Finally see the creation of the Epoll file descriptor related code, the above is still not see what the local methods are called, we look at the relevant C code

Jdk/src/solaris/native/sun/nio/ch/ioutil.cjniexport jlong jnicalljava_sun_nio_ch_ioutil_makepipe (JNIEnv *env,    Jobject this, Jboolean blocking) {int fd[2];        Open Pipe if (pipe (FD) < 0) {Jnu_throwioexceptionwithlasterror (env, "Pipe failed");    return 0;  } if (blocking = = Jni_false) {//config pipeline is non-blocking if ((Configureblocking (fd[0], Jni_false) < 0) | | (Configureblocking (fd[1], Jni_false) < 0))            {Jnu_throwioexceptionwithlasterror (env, "Configure blocking failed");            Close (fd[0]);            Close (fd[1]);        return 0; }}//writes read-write file descriptor into a long type and returns return ((Jlong) fd[0] << 32) | (Jlong) fd[1];} Jdk/src/solaris/native/sun/nio/ch/epollarraywrapper.cjniexport Jint Jnicalljava_sun_nio_ch_epollarraywrapper_  Epollcreate (jnienv *env, jobject this) {/* * Epoll_create expects a size as a hint to the kernel what to * Dimension internal structures. We can ' t predict the size in AdvanCe.    /////Here Call the Linux function epoll_create create epoll file descriptor int EPFD = Epoll_create (256);    if (EPFD < 0) {Jnu_throwioexceptionwithlasterror (env, "Epoll_create failed"); } return EPFD;}

Summarize

As explained above, now for the Netty startup process of the thread pool initialization process and selector initialization process is clear, for the native method analysis let us compare epoll programming in Linux, the principle is more clear.

Next is the need to listen to the descriptor register to Epoll, corresponding to Netty is talking channel registration to selector, the next article continue to write Netty source-two, server startup (2)

Reference

Netty Source Analysis--reactor Processselectedkeys

Discussion on optimization of Selectedselectionkeyset

https://github.com/netty/netty/issues/2363

https://github.com/netty/netty/commit/cd579f75d2b5f236f35bc47f454cc07e50ae8037

Netty Source-One, server startup (1)