Java one-time query hundreds of thousands of, millions of data solutions.

Java query one-time query hundreds of thousands of, millions of data solutions

A way to write tools at an early age.

It was used to package millions of data into a RAR file.

So I used a stupid way. Hope the Master Guide, what a good way not ah

1, the first batch to find all the data, examples are 10,000 batch.
2. After the data is detected, each data is deposited into local documents according to certain rules.
3, obtain the data, through the batch reads, obtains the massive quantity data. This method is see: http://yijianfengvip.blog.163.com/blog/static/175273432201191354043148/

The following is a query database. Query by lot
public static void Getmonthdatalist () {
ResultSet rs = null;
Statement stat = null;
Connection conn = null;
list<databean> list = new arraylist<databean> ();
try {
conn = CreateConnection ();
if (conn!=null) {
SimpleDateFormat SDF = new SimpleDateFormat ("Yyyy-mm-dd");
SimpleDateFormat timesdf = new SimpleDateFormat ("Yyyy-mm-dd HH:mm:ss");
String nowdate = Sdf.format (New Date ());
Config.lasttimetext = Timesdf.format (New Date ());
String lastdate = Sdf.format (Createdata.adddaysfordate (New Date (), 30));
Stat = conn.createstatement (resultset.type_scroll_sensitive,resultset.concur_updatable);
int lastrow = 0;
int datanum = 0;
String countsql = "SELECT count (a.id) from Trip_special_flight a" +
"Where A.dpt_date >= to_date (' +nowdate+ ', ' yyyy-mm-dd ')" +
"and A.dpt_date <= to_date (' +lastdate+" ', ' Yyyy-mm-dd ') and rownum> "+lastrow+" ORDER by a.get_time Desc ";
rs = Stat.executequery (countsql);
while (Rs.next ()) {
Datanum = Rs.getint (1);
}
int onerun = 10000;
int runnum = datanum%onerun==0? (datanum/onerun):(datanum/onerun) +1;
for (int r =0;r<runnum;r++) {
System.out.println ("getmonthdatalist--" +datanum+ "Start Inquiry" + (r+1) + "batch data");
String sql = "SELECT * from" (select RowNum rn, A.dpt_code, a.arr_code,a.dpt_date,a.airways,a.flight, "+
"A.cabin,a.price from Trip_special_flight a" +
"Where A.dpt_date >= to_date (' +nowdate+ ', ' yyyy-mm-dd ')" +
"and A.dpt_date <= to_date (' +lastdate+" ', ' yyyy-mm-dd ') Order by rownum ASC) WHERE rn > "+lastrow;
Stat.setmaxrows (Onerun);
Stat.setfetchsize (1000);
rs = stat.executequery (SQL);
String text = "";
int i = 1;
while (Rs.next ()) {
Text + rs.getstring (2) + "|" +rs.getstring (3) + "|" +rs.getdate (4) + "|" +rs.getstring (5) + "|" +rs.getstring (6) + "|" +rs.getstring (7) + "|" +rs.getstring (8) + "| |";
if (i%1000==0) {
Fileutil.appendtofile (config.tempdatafile, text);
Text = "";
}
i++;
}
if (Text.length () >10) {
Fileutil.appendtofile (config.tempdatafile, text);
}
Lastrow+=onerun;
}
}
catch (Exception e) {
E.printstacktrace ();
finally {
CloseAll (RS, stat, conn);
}

}

-----Java one-time query hundreds of thousands of, millions of data solutions

After you save the temporary file, read a large number of data file methods.

Set Cache size Buffer_size, Config.tempdatafile is a file address

SOURCE Blog http://yijianfengvip.blog.163.com/blog/static/175273432201191354043148/

Package com.yjf.util;
Import Java.io.File;
Import Java.io.RandomAccessFile;
Import Java.nio.MappedByteBuffer;
Import Java.nio.channels.FileChannel;

public class Test {
public static void Main (string[] args) throws Exception {
Final int buffer_size = 0x300000; Buffer is 3M
File F = new file (config.tempdatafile);
SOURCE Blog http://yijianfengvip.blog.163.com/blog/static/175273432201191354043148/
int len = 0;
Long start = System.currenttimemillis ();
for (int z = 8; z >0; z--) {
Mappedbytebuffer InputBuffer = new Randomaccessfile (f, "R")
. Getchannel (). Map (FileChannel.MapMode.READ_ONLY,
F.length () * (z-1)/8, F.length () * 1/8);
byte[] DST = new byte[buffer_size];//reads 3M of content each time
for (int offset = 0; offset < inputbuffer.capacity (); offset + = buffer_size) {
if (inputbuffer.capacity ()-offset >= buffer_size) {
for (int i = 0; i < buffer_size; i++)
Dst[i] = inputbuffer.get (offset + i);
} else {
for (int i = 0; i < inputbuffer.capacity ()-offset; i++)
Dst[i] = inputbuffer.get (offset + i);
}
int length = (inputbuffer.capacity ()% buffer_size = 0)? Buffer_size
: inputbuffer.capacity ()% Buffer_size;
Len + + new String (DST, 0, length). Length ();
System.out.println (New String (DST, 0, length). Length () + "-" + (Z-1) + "-" + (8-z+1));
}
}
System.out.println (len);
Long end = System.currenttimemillis ();
SYSTEM.OUT.PRINTLN ("Read File file Cost:" + (End-start) + "millisecond");
}

Read a large number of data file methods.

Reproduced from: http://blog.csdn.net/yjflinchong/article/details/7287648