JdbcTemplate Read Data memory overflow

Source: Internet
Author: User
Tags stmt

The most recent automated test for the Hadoop project was to read the data from the Hadoop Mart and write it into a. csv file. A query result set contains 100,000 + records, and each record contains 30+ columns, so JdbcTemplate's gorgeous memory overflows. In fact, the amount of data, this is not really big ...
Look at the code carefully, probably know the reason, the result set is saved in list<map<string, object>> this structure, and map is more memory-consuming, 100,000 + map, each map30+ to key-value ... Memory overflow is understandable.

Public list<map<string, Object>> querysql (final String query) {return
    jdbctemplate.queryforlist ( query);
}

Instead, use the original statement to execute query, set the fetch Szie, and then save the result set in list<list<string>>. Every 2000 lines, write a csv and empty the list<list<string>>, 666, run fast and will not overflow.

 public void Readwriteresultsettocsv (string query, String csvpath) {Try (Connection Co
        n = jdbcTempalte.getDataSource.getConnection (); Statement stmt = con.createstatement (resultset.type_forward_only, resultset.concur_read_only);) {Stmt.setfetch
            Size (5000);
            ResultSet rs = stmt.executequery (query);
            list<list<string>> rows = new arraylist<> ();
            List<string> Header = Getcolheaderfromrs (RS);
                while (Rs.next ()) {list<string> row = new arraylist<> ();
                for (String H:header) {Row.add (rs.getstring (h));
                } rows.add (row);
                if (Rows.size () >) {//write to CSV rows.clear ();
        } if (Rows.isempty ()) {//write to CSV} rs.close (); }

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.