Java NIO File Transfer

Java NIO File Transfer

 

First, make sure you have learned the basics of Java NIO, including Buffer, Channel file Channel, Socket Channel, and Selector. The differences between NIO and I/O are not mentioned here. For more information, see the Reference Links.

This blog uses an example of using NIO to transfer files to learn the basic operations of the network in NIO.

Problems with traditional socket monitoring methods

The traditional Java solution for Monitoring Multiple sockets is to create a thread for each socket so that the thread can be blocked during the read call until the data is available. In fact, this solution has a major drawback: when many links need to be created with many threads, the creation and management of these threads will consume a lot of resources, this connection may only send a small amount of data, but CPU switching also consumes a lot of resources. Therefore, to reduce the overhead of system threads, the thread pool method is used to reduce the cost of thread creation and recovery. However, some use cases still cannot be solved. If persistent connections are established, in fact, they are not transmitting data all the time, so it is impossible to create so many connections.

Using NIO can effectively solve this problem. Using the Selector provided by NIO, you can use one thread to manage multiple connections, which is actually an I/O multiplexing.

NIO File Transfer example

The following example uses NIO to upload files to the server

Server code

Public class Server {private ByteBuffer buffer = ByteBuffer. allocate (1024*1024); // use Map to save each connection. When OP_READ is ready, find the corresponding file based on the key to write it. If it is encapsulated into a class and saved as a value, the progress can be displayed during the upload process.
 
  
FileMap = new HashMap
  
   
(); Public static void main (String [] args) throws IOException {Server server Server = new server (); Server. startServer ();} public void startServer () throws IOException {Selector selector = Selector. open (); ServerSocketChannel serverChannel = ServerSocketChannel. open (); serverChannel. configureBlocking (false); serverChannel. bind (new InetSocketAddress (8888); serverChannel. register (selector, SelectionKey. OP_ACCEPT); System. out. println (the server has enabled ...); while (true) {int num = selector. select (); if (num = 0) continue; Iterator
   
    
It = selector. selectedKeys (). iterator (); while (it. hasNext () {SelectionKey key = it. next (); if (key. isAcceptable () {ServerSocketChannel serverChannel1 = (ServerSocketChannel) key. channel (); SocketChannel socketChannel = serverChannel1.accept (); if (socketChannel = null) continue; socketChannel. configureBlocking (false); SelectionKey key1 = socketChannel. register (selector, SelectionKey. OP_READ); InetSocketAddress remoteAddress = (InetSocketAddress) socketChannel. getRemoteAddress (); File file = new File (remoteAddress. getHostName () + _ + remoteAddress. getPort () +. txt); FileChannel fileChannel = new FileOutputStream (file ). getChannel (); fileMap. put (key1, fileChannel); System. out. println (socketChannel. getRemoteAddress () + connection successful ...); writeToClient (socketChannel);} else if (key. isReadable () {readData (key);} // the features of NIO are accumulated only, and the set of selected keys is not deleted, the ready set will be cleared. // only the selected key set will be temporarily deleted. When the selected key represents a channel that is once again interested in the Set, it will be selected by the select function. remove () ;}} private void writeToClient (SocketChannel socketChannel) throws IOException {buffer. clear (); buffer. put (socketChannel. getRemoteAddress () + connection successful ). getBytes (); buffer. flip (); socketChannel. write (buffer); buffer. clear ();} private void readData (SelectionKey key) throws IOException {FileChannel fileChannel = fileMap. get (key); buffer. clear (); SocketChannel socketChannel = (SocketChannel) key. channel (); int num = 0; try {while (num = socketChannel. read (buffer)> 0) {buffer. flip (); // write the file fileChannel. write (buffer); buffer. clear () ;}} catch (IOException e) {key. cancel (); e. printStackTrace (); return;} // call close to-1 to reach the end if (num =-1) {fileChannel. close (); System. out. println (uploaded); buffer. put (socketChannel. getRemoteAddress () + upload successful ). getBytes (); buffer. clear (); socketChannel. write (buffer); key. cancel ();}}}
   
  
 

The client simulates three clients to send files to the server at the same time.

Public class Client {public static void main (String [] args) {for (int I = 0; I <3; I ++) {// simulate the three initiators new Thread () {public void run () {try {SocketChannel socketChannel = SocketChannel. open (); socketChannel. socket (). connect (new InetSocketAddress (127.0.0.1, 8888); File file = new File (E: \ + 11 +. txt); FileChannel fileChannel = new FileInputStream (file ). getChannel (); ByteBuffer buffer = ByteBuffer. allocate (100); socketChannel. read (buffer); buffer. flip (); System. out. println (new String (buffer. array (), 0, buffer. limit (), Charset. forName (UTF-8); buffer. clear (); int num = 0; while (num = fileChannel. read (buffer)> 0) {buffer. flip (); socketChannel. write (buffer); buffer. clear ();} if (num =-1) {fileChannel. close (); socketChannel. shutdownOutput ();} // receives the socketChannel from the server. read (buffer); buffer. flip (); System. out. println (new String (buffer. array (), 0, buffer. limit (), Charset. forName (UTF-8); buffer. clear (); socketChannel. close ();} catch (IOException e) {e. printStackTrace ();}};}. start ();} Thread. yield ();}}

It can be seen that we only use one thread to manage the three connections. Compared to the previously blocked Socket, We need to enable the thread to manage the connection after the accept function returns, when NIO is used, we only register it on the selector after the accept is returned. The read operation will process it the next time it detects a set of readable keys.

NIO + thread pool Improvement

Of course, it is not always a good idea to use a single thread. The advantage of using a single thread is that only one thread can run under any circumstances. By eliminating the additional overhead of context switching between threads, the total throughput can be improved. However, in other cases, especially for multi-core CPUs, using a single thread is a waste of CPU resources. A better strategy is to use a thread to manage the selector and monitor the readiness of the channel, you can use a thread pool to process data.

The above example is changed

 

Public class ThreadPoolServer extends Server {private ExecutorService exec = Executors. newFixedThreadPool (10); public static void main (String [] args) throws IOException {ThreadPoolServer server = new ThreadPoolServer (); server. startServer () ;}@ Overrideprotected void readData (final SelectionKey key) throws IOException {// remove the readable event of this key, which has been processed in the thread pool. interestOps (key. interestOps ()&(~ Selectionkey.op_read000000000000exec.exe cute (new Runnable () {@ Overridepublic void run () {ByteBuffer buffer = ByteBuffer. allocate (1024*1024); FileChannel fileChannel = fileMap. get (key); buffer. clear (); SocketChannel socketChannel = (SocketChannel) key. channel (); int num = 0; try {while (num = socketChannel. read (buffer)> 0) {buffer. flip (); // write the file fileChannel. write (buffer); buffer. clear () ;}} catch (IOException e) {Key. cancel (); e. printStackTrace (); return;} // call close to-1 to reach the end if (num =-1) {try {fileChannel. close (); System. out. println (uploaded); buffer. put (socketChannel. getRemoteAddress () + upload successful ). getBytes (); buffer. clear (); socketChannel. write (buffer);} catch (IOException e) {e. printStackTrace ();} // The set of selected keys will be removed only when cancel is called. Otherwise, the end of the selected key will be closed again, the peer end is still readable. The EOF is obtained and-1key is returned. cancel (); return;} // re of the Channel The ad method may return 0, and the return 0 does not necessarily mean that the read is complete. // When the worker thread ends reading the channel, it needs to update the ready Set of the key again and put the set of interest in the key again. interestOps (key. interestOps () | SelectionKey. OP_READ); // call wakeup so that the first selection operation on the selector has not yet returned returns immediately and then returns the selectkey again. selector (). wakeup ();}});}}

Notes

1. register the channel to a selection, and a key is returned. This key represents the registration relationship between the channel and the selector.

2. The SelectionKey key contains two sets.

Interest set: indicates the operations that each channel cares about. You can also use the key-based interestOps method to modify the operations that are determined during channel registration.

Ready Set: indicates that the channel has been prepared for this operation. It is a subset of the interest set and indicates the operations that have been completed since the previous select call in the interest set. This set cannot be modified, as the operating system tells us.

Selector contains three sets

Registered key set: Call the register Method of the Channel.

Selected key set)

Canceled key set: the Canceled key set is used to call the cancel Method of the key, and the select cancel method puts the key in the set of Canceled keys. When the channel is closed, the related keys are also automatically canceled.

3. When the select method is called, the following operations are performed, which is the key to understanding each operation called in NIO.

1) Check the set of canceled keys. If it is not empty, remove the key from the other two sets. After this step, the set of canceled keys is empty.

2 ). check the interest set of keys in the set of registered keys. For channels that indicate that at least one operation in the interest set is ready, either of the following operations is performed:

A. this key does not exist in the selected key set: the ready Set of this key is cleared, and the currently ready operation will be added to the ready set, then place the key in the selected key set.

B. otherwise, the key exists in the selector's selected key set, and its ready set will be cumulative. Only the bit determined by the operating system will be set, the previous settings will not change.

3). Repeat 1

4). The value returned by the select operation is the number of Keys Modified by the ready Set in step 2, rather than the number of all keys prepared.

From the above selection process, you can know that once a selector adds a key to its selected key set, it will not remove this key, and, once a key is in a selected key set, the ready Set of this key will only be set and will not be cleared. As a result, the tasks that manage keys to ensure that they have correct status information are done by programmers.

The remove Method in the program deletes a key from the selected set. Otherwise, the key is not automatically deleted, indicating that the channel represented by the key has been processed.

When the operation on a channel is completed and you want to delete the channel, the cancel Method of the key is called. It will be added to the cancellation set, and the registered relationship will not be canceled immediately, the key will expire immediately. The next time you call the select method, the key will be deleted from the other two sets, and the channel will be canceled.

When using the NIO + thread pool, when there is a readable event in the above program, it is handled by the thread pool. During this period, the read operation of this channel should be ignored to delete it from the interest set. When the thread ends the operation for this channel, it needs to update the ready Set of the key and re-place the set of interest in it. The selector's wakeup method is called to prevent the selector from returning immediately if it is blocked on the select statement.

4. in tcp, you still need to tell the end flag of the Peer end. The shutDownOutput method is called here (the end flag is available only after nio api 1.7 ), in this way, the read method on the other end will read EOF and return-1.