Java read large file operation "Go"

Transfer from http://aronlulu.iteye.com/blog/1018370

Read File size: 1.45G
The first kind, Oldio:

Java code

2. Public static void Oldioreadfile () throws ioexception{
4. BufferedReader br = new BufferedReader (new FileReader ("G://lily_947.txt"));
6. PrintWriter pw = new PrintWriter ("g://oldio.tmp");
8. char[] C = new char[100*1024*1024];
10. for (;;) {  
12. if (Br.read (c)!=-1) {
14. Pw.print (c);
16. }else{
18. Break ;
20. }
22. }
24. Pw.close ();
26. Br.close ();
28. }

Time Consuming 70.79s


The second kind, Newio:

Java code

2. Public static void Newioreadfile () throws ioexception{
4. FileChannel read = new Randomaccessfile ("G://lily_947.txt","R"). Getchannel ();
6. FileChannel writer = new Randomaccessfile ("g://newio.tmp","RW"). Getchannel ();
8. Bytebuffer BB = bytebuffer.allocate (200*1024*1024);
10. While (Read.read (BB)!=-1) {
12. Bb.flip ();
14. Writer.write (BB);
16. Bb.clear ();
18. }
20. Read.close ();
22. Writer.close ();
26. }

Time Consuming 47.24s


The Third Kind, randomaccessfile:

Java code

2. Public static void Randomreadfile () throws ioexception{
4. Randomaccessfile read = new Randomaccessfile ("G://lily_947.txt","R");
6. Randomaccessfile writer = new Randomaccessfile ("g://random.tmp","RW");
8. byte[] B = new byte[200*1024*1024];
10. While (Read.read (b)!=-1) {
12. Writer.write (b);
14. }
16. Writer.close ();
18. Read.close ();
20. }

Time 46.65

The fourth kind, Mappedbytebuffer:

Java code

2. Public static void Mappedbuffer () throws ioexception{
4. FileChannel read = new FileInputStream ("G://lily_947.txt"). Getchannel ();
6. FileChannel writer = new Randomaccessfile ("g://buffer.tmp","RW"). Getchannel ();
8. long i = 0;
10. Long size = Read.size ()/30;
12. Bytebuffer BB,CC = null;
14. While (I<read.size () && (Read.size ()-i) >size) {
16. bb = Read.map (FileChannel.MapMode.READ_ONLY, I, size);
18. CC = Writer.map (FileChannel.MapMode.READ_WRITE, I, size);
20. Cc.put (BB);
22. I+=size;
24. Bb.clear ();
26. Cc.clear ();
28. }
30. bb = Read.map (FileChannel.MapMode.READ_ONLY, I, read.size ()-i);
32. Cc.put (BB);
34. Bb.clear ();
36. Cc.clear ();
38. Read.close ();
40. Writer.close ();
44. }

Time: 36

The resource occupancy graphs for the first three readings are as follows:
Compared to the last memory direct mapping method before the test actually meaningless, basic second kill ....
There is not enough memory for a large file block map directly, because Mappedbytebuffer is not released, Sun does not provide a way to reclaim the Mappedbytebuffer area directly, and this time there are two ways to solve it, the first of which is rather stupid:

Java code

1. System.GC ();
2. System.runfinalization ();
3. try {
4. Thread.Sleep (3000);
5. } catch (Interruptedexception e) {
7. E.printstacktrace ();
8. }

The second kind of online search, using reflection to call the Clean method:

Java code

2. Public static void Unmap (final mappedbytebuffer buffer) {
4. if (buffer = = null) {
6. return;
8. }
10. Accesscontroller.doprivileged (new privilegedaction<object> () {
12. Public Object Run () {
14. try {
16. Method Getcleanermethod = Buffer.getclass (). GetMethod ("cleaner", new class[0]);
18. if (getcleanermethod! = null) {
20. Getcleanermethod.setaccessible (true);
22. Object cleaner = getcleanermethod.invoke (buffer, new object[0]);
24. Method Cleanmethod = Cleaner.getclass (). GetMethod ("clean", new class[0]);
26. if (cleanmethod! = null) {
28. Cleanmethod.invoke (cleaner, new object[0]);
30. }
32. }
34. } catch (Exception e) {
36. E.printstacktrace ();
38. }
40. return null;
42. }
46. });
48. }

The above two methods feel awkward, there is the ability to separate into physical files to recycle calls, this is not too beautiful.
The speed will also slow down a lot.

Java read large file operation "Go"