Buffer data structure and memory-mapped files for new IO

First, the Buffer class

Java.nio.Buffer what is this class for? What kind of structure is there?

"Core Java" is defined as "A buffer is array of values of the same type". So, we can sense the perceptual:buffer is like an array, storing the same type of data . Another important thing is thatbuffer is a data structure of a random storage type , just like a normal array (with subscript) to be able to position the index number at any location in the buffer.

The buffer class is an abstract class with subclasses (note: The StringBuffer class and the buffer here do not have any connection):

the properties and methods in the buffer class are used to manage and control the state of the buffer, and its subclasses have a get or put method for "read out" or "write" different types of data; We should also note:Bytebuffer, Charbuffer such as flammable is an abstract class. Therefore, we are not able to get a bytebuffer or charbuffer by new way. Now, what if we want a charbuffer (Bytebuffer can get it in the same way)?

Answer: You can take advantage of the Charbuffer allocate (int capacity) method, or the Wrap method that wraps the existing char[] array , as follows:

The most used in our actual programming are Bytebuffer and Charbuffer, as shown in buffer with the following 4 properties:

①, capacity, a buffer after the establishment, it will be fixed unchanged;

②, position, which indicates the next location to read or write data;

③, limit, the data beyond the limit position is meaningless;

④, Mark, which marks an important location for re-reading or writing back to that position;

Their four relationships are:

0≤mark≤position≤limit≤capacity

One of the most important functions of buffer is: The loop is used for "write first, then read". Here is a first-write post-read process:

①, at the beginning, position=0 && limit=capacity;

②, call the Put method to write data to buffer, when the data is finished or position reached the location of the capacity, the following starts to read from the buffer data;

③, from the write state to the read state, you need to call the Flip method. The function is: first let limit equals the current position, then set the position to 0;

④, when reading the data, as long as the remaining method (Limit-position) Returns a positive number, then we can continue to call the Get method to read the data from buffer;

⑤, when the read data process is complete, we can call the clear method, the buffer from the "read" state to "write" state, into the next "read First, write" loop;

⑥, Clear method set position=0 && limit=capacity;

⑦, if you want to re-read buffer, you can call its rewind or Mark/reset method, the API is described in detail;

API:java.nio.Buffer

①, Buffer Clear ()

Set buffer to enter write state, set position=0 && limit=capacity;

②, Buffer Flip ()

Set buffer to enter the read state, set limit equal to the current position && position=0;

③, Buffer Rewind ()//rewind can be translated as "rewind", "rewind", etc.

Ready to re-read the same data in buffer, set position=0 && limit to remain unchanged;

④, Buffer Mark ()

Sets the current position to mark, which can be used in conjunction with the Reset () method to reread/write buffer;

⑤, Buffer Reset ()

Set the Position=mark so that you can start reading or writing from the mark position again;

⑥, int remaining ()

Returns the number of "readable" data in buffer, or the number of new data that can be "written", returning a limit-position;

⑦, int position () returns the value of the current position;

⑧, int capacity () returns the value of the capacity in buffer;

Ii. file mapping for NiO

Let's look at the new enhancements to the Java.nio package with respect to the old IO:

NiO mainly supports four enhanced features:

Encoding and decoding of ① and character sets;

②, non-blocking IO (nonblocking I/O);

③, memmory-mapped files;

④, file lock;

The encoding and decoding of the character set can be taken separately. Non-blocking IO is mainly used in network communication. File lock is a complex but not reliable thing (depends on the specific operating system to lock the support), in the case of concurrency, usually can be aided by the database lock mechanism, the file is stored in the database to achieve file synchronization.

Here is the main summary of memmory-mapped files.

Most operating systems can take advantage of virtual memory to map an entire file or part of a file into memory. Then, we can access the mapping file like a memory array (primarily its random access feature), which is much faster than the traditional file operation (randomaccessfile).

"Discussion: There are roughly three ways to manipulate files, file streams, randomaccessfile, memmory-mapped files, but each has its own features: ① file streams and buffered streams combine quickly, but do not have random access features; ②, Randomaccessfile has a random access feature, but it is inefficient; ③, memmory-mapped files have random access features that are even more efficient than buffered streams, and are primarily used for "large files" operations.

The map operation of the file is simple, according to the following steps:

①, gets a channel from the file. The channel is an abstraction of the disk file, which can acquire some features of the operating system, such as: memory mapping, file lock, fast data transfer between files. The FileInputStream, FileOutputStream, and Randomaccessfile classes have been rewritten in jak1.4, and the Getchannel method has been added to them . So, we can get the channel to the disk file by calling the Getchannel method. As follows:

New= In.getchannel ();

②, obtained from the channel to Mappedbytebuffer. We can map the file by calling the Channel class's Map method, which returns a Mappedbytebuffer object. In the map method we can specify the scope of the file map (all or part), and you can specify the mode of the mapping, which supports three modes altogether:

----FileChannel.MapMode.READ_ONLY: Data can only be read from buffer and cannot be written as buffer. A Readonlybufferexception exception is thrown when the Write method is called.

----FileChannel.MapMode.READ_WRITE:buffer is writable, and the changed data in buffer is written back to the file at some point. Note that other programs that also map the file do not immediately perceive this change (so there is a file synchronization lock mechanism).

----FileChannel.MapMode.PRIVATE:buffer is writable, but any changes will not be written back to the file.

③, once we get a buffer, we can call buffer or its subclass to access the data in buffer. Note that buffer supports both sequential and random access methods. For example, the following two examples:

// use sequential access to buffer  while (Buffer.hasremaining ()) {     byte b = buffer.get ();     ...} // use random access buffer  for 　　(int i = 0; i < buffer.limit (); i++) {     byte b = buffer.get (i); //buffer This abstract class does not have a get and put method, it is only responsible for managing and controlling the state     of the buffer ...}

Buffer.order (Byteorder.little_endian); Specify small-End storage

Byteorder B = Buffer.order (); Identify the mode in which byte is stored in the current buffer

In the following example, Fileinputstream,bufferedinputstream,randomaccessfile and mappedfile are used respectively to read the Rt.jar file (59.8MB), and the calculator CRC32 the value. We can compare the efficiency of four kinds of treatment, get a perceptual understanding:

 PackageNiO;ImportJava.io.*;ImportJava.nio.*;ImportJava.nio.channels.FileChannel;ImportJava.util.Scanner;Importjava.util.zip.CRC32; Public classNiotest { Public Static LongChecksuminputstream (String filename)throwsexception{CRC32 CRC32; InputStream in=NULL; Try{CRC32=NewCRC32 (); Inch=Newfileinputstream (filename); intC;  while((c = In.read ())! = 1) {crc32.update (c); }        } finally{            if(In! =NULL) In.close (); }                returnCrc32.getvalue (); }         Public Static LongChecksumbufferedinputstream (String filename)throwsexception{CRC32 CRC32=NewCRC32 (); InputStream in=NULL; Try{ in=NewBufferedinputstream (Newfileinputstream (filename)); intb;  while((b = In.read ())! =-1) {crc32.update (b); }        } finally{            if(In! =NULL) In.close (); }                returnCrc32.getvalue (); }         Public Static LongChecksumrandomaccessfile (String filename)throwsexception{randomaccessfile File=NULL; CRC32 CRC=NewCRC32 (); Try{file=NewRandomaccessfile (filename, "R");//read-only mode            LongLength =file.length ();  for(inti = 0; i < length; i++) {File.seek (i); intb =File.read ();            Crc.update (b); }        } finally{            if(File! =NULL) File.close (); }                returnCrc.getvalue (); }         Public Static LongChecksummappedfile (String filename)throwsexception{CRC32 CRC=NewCRC32 (); FileInputStream in=NULL; FileChannel Channel=NULL; Try{ in=Newfileinputstream (filename); Channel=In.getchannel (); intSize = (int) channel.size (); Mappedbytebuffer Buffer= Channel.map (FileChannel.MapMode.READ_ONLY, 0, size);  for(intp = 0; p < size; p++){                intb =Buffer.get (P);            Crc.update (b); }        } finally{            if(In! =NULL) In.close (); }                returnCrc.getvalue (); }         Public Static voidMain (string[] args)throwsException {System.out.println ("Enter test file path:"); InputStream in=system.in; Scanner Scanner=NewScanner (in); String filename=Scanner.nextline (); System.out.println ("Start calculation ...");                System.out.println (); LongStart =System.currenttimemillis (); LongCrcvalue =checksuminputstream (filename); LongEnd =System.currenttimemillis (); System.out.println ("---inputstream---->" + (End-start) + "milliseconds. CRC32 Value: "+long.tohexstring (Crcvalue)); Start=System.currenttimemillis (); Crcvalue=checksumbufferedinputstream (filename); End=System.currenttimemillis (); System.out.println ("---bufferedinputstream---->" + (End-start) + "milliseconds. CRC32 Value: "+long.tohexstring (Crcvalue)); Start=System.currenttimemillis (); Crcvalue=checksumrandomaccessfile (filename); End=System.currenttimemillis (); System.out.println ("---randomaccessfile---->" + (End-start) + "milliseconds. CRC32 Value: "+long.tohexstring (Crcvalue)); Start=System.currenttimemillis (); Crcvalue=checksummappedfile (filename); End=System.currenttimemillis (); System.out.println ("---mappedfile---->" + (End-start) + "milliseconds. CRC32 Value: "+long.tohexstring (Crcvalue)); if(Scanner! =NULL) Scanner.close (); }}

Execution Result:

enter the path to the test file: E:\Java\jdk1. 8. 0_25\jre\lib\rt.jar Start Calculation ... ---inputstream----> 222721 milliseconds. CRC32 value: 2A57AC2---bufferedinputstream----> 3020 milliseconds. CRC32 value: 2A57AC2---randomaccessfile----> 377263 milliseconds. CRC32 value: 2A57AC2---mappedfile----> 4323 ms. CRC32 Value: 2A57AC2

Buffer data structure and memory-mapped files for new IO