Use memory ing in Java to process large files

Source: Internet
Author: User

When processing large files, frequent read/write operations using normal FileInputStream, FileOutputStream, or RandomAccessFile will lead to slow down the process due to frequent read/write. The following is a comparative experiment.
[Java]
Package test;
 
Import java. io. BufferedInputStream;
Import java. io. FileInputStream;
Import java. io. FileNotFoundException;
Import java. io. IOException;
Import java. io. RandomAccessFile;
Import java. nio. MappedByteBuffer;
Import java. nio. channels. FileChannel;
 
Public class Test {
 

Public static void main (String [] args ){
Try {
FileInputStream FCM = new FileInputStream ("/home/tobacco/test/res.txt ");
Int sum = 0;
Int n;
Long t1 = System. currentTimeMillis ();
Try {
While (n = FCM. read ()> = 0 ){
Sum + = n;
}
} Catch (IOException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}
Long t = System. currentTimeMillis ()-t1;
System. out. println ("sum:" + sum + "time:" + t );
} Catch (FileNotFoundException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}

Try {
FileInputStream FCM = new FileInputStream ("/home/tobacco/test/res.txt ");
BufferedInputStream bis = new BufferedInputStream (FCM );
Int sum = 0;
Int n;
Long t1 = System. currentTimeMillis ();
Try {
While (n = bis. read ()> = 0 ){
Sum + = n;
}
} Catch (IOException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}
Long t = System. currentTimeMillis ()-t1;
System. out. println ("sum:" + sum + "time:" + t );
} Catch (FileNotFoundException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}

MappedByteBuffer buffer = null;
Try {
Buffer = new RandomAccessFile ("/home/tobacco/test/res.txt", "rw"). getChannel (). map (FileChannel. MapMode. READ_WRITE, 0, 1253244 );
Int sum = 0;
Int n;
Long t1 = System. currentTimeMillis ();
For (int I = 0; I <1253244; I ++ ){
N = 0x000000ff & buffer. get (I );
Sum + = n;
}
Long t = System. currentTimeMillis ()-t1;
System. out. println ("sum:" + sum + "time:" + t );
} Catch (FileNotFoundException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
} Catch (IOException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}
 
}
 
}

The test file is a 1253244-byte file. Test results:
Sum: 220152087 time: 1464
Sum: 220152087 time: 72
Sum: 220152087 time: 25
The read data is correct. Delete the Data Processing Section.
[Java]
Package test;
 
Import java. io. BufferedInputStream;
Import java. io. FileInputStream;
Import java. io. FileNotFoundException;
Import java. io. IOException;
Import java. io. RandomAccessFile;
Import java. nio. MappedByteBuffer;
Import java. nio. channels. FileChannel;
 
Public class Test {
 

Public static void main (String [] args ){
Try {
FileInputStream FCM = new FileInputStream ("/home/tobacco/test/res.txt ");
Int sum = 0;
Int n;
Long t1 = System. currentTimeMillis ();
Try {
While (n = FCM. read ()> = 0 ){
// Sum + = n;
}
} Catch (IOException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}
Long t = System. currentTimeMillis ()-t1;
System. out. println ("sum:" + sum + "time:" + t );
} Catch (FileNotFoundException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}

Try {
FileInputStream FCM = new FileInputStream ("/home/tobacco/test/res.txt ");
BufferedInputStream bis = new BufferedInputStream (FCM );
Int sum = 0;
Int n;
Long t1 = System. currentTimeMillis ();
Try {
While (n = bis. read ()> = 0 ){
// Sum + = n;
}
} Catch (IOException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}
Long t = System. currentTimeMillis ()-t1;
System. out. println ("sum:" + sum + "time:" + t );
} Catch (FileNotFoundException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}

MappedByteBuffer buffer = null;
Try {
Buffer = new RandomAccessFile ("/home/tobacco/test/res.txt", "rw"). getChannel (). map (FileChannel. MapMode. READ_WRITE, 0, 1253244 );
Int sum = 0;
Int n;
Long t1 = System. currentTimeMillis ();
For (int I = 0; I <1253244; I ++ ){
// N = 0x000000ff & buffer. get (I );
// Sum + = n;
}
Long t = System. currentTimeMillis ()-t1;
System. out. println ("sum:" + sum + "time:" + t );
} Catch (FileNotFoundException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
} Catch (IOException e ){
// TODO Auto-generated catch block
E. printStackTrace ();
}
 
}
 
}

Test results:
Sum: 0 time: 1458
Sum: 0 time: 67
Sum: 0 time: 8
It can be seen that the speed will be improved by ing some or all files to the memory for read/write operations.
This is because the memory ing file first maps the external files to a contiguous area in the memory and is processed as a byte array. read/write operations are performed directly on the memory, then, the memory area is re-mapped to the external storage file, which saves the time for frequent external storage read/write and greatly reduces the read/write time.
Author: tobacco5648

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.