The most recent automated test for the Hadoop project was to read the data from the Hadoop Mart and write it into a. csv file. A query result set contains 100,000 + records, and each record contains 30+ columns, so JdbcTemplate's gorgeous memory overflows. In fact, the amount of data, this is not really big ...
Look at the code carefully, probably know the reason, the result set is saved in list<map<string, object>> this structure, and map is more memory-consuming, 100,000 + map, each map30+ to key-value ... Memory overflow is understandable.
Public list<map<string, Object>> querysql (final String query) {return
jdbctemplate.queryforlist ( query);
}
Instead, use the original statement to execute query, set the fetch Szie, and then save the result set in list<list<string>>. Every 2000 lines, write a csv and empty the list<list<string>>, 666, run fast and will not overflow.
public void Readwriteresultsettocsv (string query, String csvpath) {Try (Connection Co
n = jdbcTempalte.getDataSource.getConnection (); Statement stmt = con.createstatement (resultset.type_forward_only, resultset.concur_read_only);) {Stmt.setfetch
Size (5000);
ResultSet rs = stmt.executequery (query);
list<list<string>> rows = new arraylist<> ();
List<string> Header = Getcolheaderfromrs (RS);
while (Rs.next ()) {list<string> row = new arraylist<> ();
for (String H:header) {Row.add (rs.getstring (h));
} rows.add (row);
if (Rows.size () >) {//write to CSV rows.clear ();
} if (Rows.isempty ()) {//write to CSV} rs.close (); }