Use RandomAccessFile in java to read large files

For ultra-large files, we use common file reading methods very slowly and very slow. in java, we provide the RandomAccessFile function for me to quickly read ultra-large files without getting stuck, see one of my demo instances below.

The log files on the server are usually more than 400 mb. Reading simple files is too slow and occupies machine resources too much.

In particular, if you need to scan the log file once every five minutes to collect some real-time data. For example, visit customers in the past 10 minutes (large Website user statistics systems such as 51.la will be frequently used .) It is very important to scan a portion of big data files in real time.
This article describes how to use the java RandomAccessFile method to read Partial bytes from a large file.

The total size of the test file is 46085 bytes.

Reads the last 85 bytes of a file.

Size: 46085
85

Test Results

? Show function-> youku video
 
Other operations such as Qiyi and Tudou are in the same order. Of course, we can also read 20 bytes from 46000, which is only used as an example.

The Code is as follows:

Copy code

Package com. javaer. examples. file;   Import java. io. IOException; Import java. io. RandomAccessFile;   Public class ReadBigFile { Public static void readBigFile () throws IOException { String fileName = "/Users/mc2/Desktop/youku.txt "; RandomAccessFile randomFile = null; RandomFile = new RandomAccessFile (fileName, "r "); Long fileLength = randomFile. length (); System. out. println ("file size:" + fileLength ); Int start = 46000; RandomFile. seek (start ); Byte [] bytes = new byte [91]; Int byteread = 0; // Read 10 bytes at a time. If the file content is less than 10 bytes, read the remaining bytes. // Assign the number of bytes read at a time to byteread While (byteread = randomFile. read (bytes ))! =-1 ){ // System. out. write (bytes, 0, byteread ); } System. out. println (bytes. length ); System. out. println (new String (bytes, "UTF-8 ")); If (randomFile! = Null ){ RandomFile. close (); }   } /** * @ Param args * @ Throws IOException */ Public static void main (String [] args) throws IOException { ReadBigFile. readBigFile (); }   }

Even if a large file reads a little data from it, the speed is fast. It also occupies a small amount of memory.

Core tips: randomFile. seek (start );

Skip read. read from here. The pointer directly points to the start position to start reading files.

Bytes retrieval can be replaced as follows.

The Code is as follows:

Copy code

Byte [] bytes = new byte [91]; Int byteread = 0; // Read 10 bytes at a time. If the file content is less than 10 bytes, read the remaining bytes. // Assign the number of bytes read at a time to byteread While (byteread = randomFile. read (bytes ))! =-1 ){ // System. out. write (bytes, 0, byteread ); } System. out. println (bytes. length); byte [] bytes; Int byteread = 0;   ByteArrayOutputStream byteout = new ByteArrayOutputStream (); Byte tmp [] = new byte [1024]; Byte context []; Int I = 0; Int has = 0; While (I = randomFile. read (tmp ))! =-1 ){ Byteout. write (tmp, 0, I ); Has + = I; If (has> 10240) Break; } Bytes = byteout. toByteArray ();