How to increase the speed of inserting 0.1 million records

Source: Internet
Author: User
Tags oracleconnection sql loader
A test is being conducted recently to see how long it takes to insert 10 records into the database!
System: RedHat + Oracle 9.2.0.4, + RAID 5 + 10 thousand rpm scsi hard drive

Condition: commit is used for each insert.
Create an insert Statement on the client and send it to the database.

After using all the methods, I found that only about 600 entries can be inserted per second. However, under the same conditions, SQL Server can insert more than 1000 entries per second, which makes me a good Oracle DBA, so I decided to optimize it.

So I tried my best to optimize it:
Create a data file and an online log file on a bare device.
Insert/* + append */.....

But it does not improve. The insertion speed is increased to 65 per second!

In the V $ system_event view, a large number:
Log File parallel write 27088 27078 11
Log File sync 26935 0 784
Wait.

Who can help me? I also want to view all comments in two sentences.Related Comments

  • Zhanghuajie (09:39:33) cannot make these 0.1 million records into files. Can I use sqlload to import them ?? You can also create an excle file !!!!
  • Sunnyxu (09:55:27) does not work, because through the program, we have a lot of concurrent access!
  • Tim_wong (11:12:40) 2000 commit once.
  • Tim_wong (2004-7-20 11:15:32) Sorry,

    The condition is missing.

    Maybe you have too many table constraints.

  • Sunnyxu (12:07:50) does not have any indexes or constraints on this table.

    Under the same conditions, the SQL Server database inserts more than 1200 rows per second!

  • Jametong (12:15:04) Why do I have to insert a single commit without inserting a single commit? Can I insert it in batches? As a result, rollback segments are allocated too frequently and log files are written too frequently,
  • Zhanhu_hou (12:56:42) has not been optimized, but I personally feel that the data processing speed of Oracle may not be able to catch up with SQL Server, and Oracle may be more reflected in stability and cross-platform. Basically, I don't have any expectation on its data processing speed.
  • Jametong (12:59:44) does not agree with the upstairs opinion that the performance of Oracle can be adjusted,
    DBA needs to adjust and choose specific situations (stability, high user performance, flexibility, and manageability)
  • Zhanhu_hou (13:23:30) has aroused my interest. Currently, Oracle has been properly optimized (SQL Server has not been optimized ), there are 650 records per second (see the author). We all want to optimize it. The method I came up with is to modify the program first without having to insert a commit for every record. Therefore, optimize the SQL statement first.
  • Jametong (13:29:28) in the current situation, if it is changed to 10 thousand commit once, I believe there is no problem in doubling efficiency. The specific test has no time to do it today.
    I used to perform an SQL loader import test.
    0.2 million data entries. By default, the Database Import time is 1 minute or 50 seconds,
    When I modified readsize = 21971520, the loading time is about 40 seconds,
    When the direcy = y parameter is added, the loading time is about 10 seconds.

    The following article is a test conducted by a dbaspecialist engineer.

    Load your data faster.htm
    (, Size: 29.5 kb, downloads: 138)

  • Zhanhu_hou (13:49:10) it seems that Oracle is very scalable. Different data loading methods differ greatly.
    From this point of view, it is not easy to manage databases.
  • Jametong (14:31:51)
  • Sunnyxu (15:58:52) I had to make sure that no one was inserted before commit. This is the application requirement.
    I did not load some data for testing. This is a formal application.
  • D. C. B. A (20:30:25) I use a text import tool written in Java. There are no problems with 10 thousand entries per second in Oracle.
    This must be a program problem. The program runs on Windows.
  • Jametong (20:39:32) Can you consider modifying the rollback segment parameters?
  • Chao_ping (23:04:25) inserts a commit entry, which is on design and limits the scale.

    In this regard, the implementation mechanism of Oracle determines that it will not be very fast. I think it is understandable than SQL Server, but the difference should not be so wide.

    Another possibility is that your connection is transmitted through SQL * Net every time, which also limits your speed.

    I did a simple test, PC server, RAID5 cx300 disk array.
    The speed can reach 3000.
    Of course, if batch commit is used, the efficiency should be improved a lot.

    1 begin
    2 For X in 1 .. 100000 Loop
    3 insert into test values (X );
    4 commit;
    5 end loop;
    6 * end;
    SQL>/

    PL/SQL procedure successfully completed.

    Elapsed: 00:00:35. 93
    SQL> select 100000/35 from dual;

    100000/35
    ----------
    2857.14286

    Elapsed: 00:00:00. 0

  • D. C. B. A (08:05:39) should also post the index, the primary key of ms SQL is generally equivalent to Oracle's Iot.
  • Rollingpig (08:51:29) studies how Oracle inserts a piece of data and commit it.

    Next, let's take a look at how SQL Server inserts a piece of data and commit it.

    You should find that Oracle does more work than SQL-server.
    These tasks are required to ensure Oracle stability/High concurrency/Etc ..

    The above is purely based on common sense speculation and has not been carefully verified.

  • Sunnyxu (10:14:24) Thank you for your participation!

    Let me make a summary:

    Test prerequisites:
    Because it is a website, the number of visits per person is counted, the business requirements require that a must be inserted into a commit;

    Test with Test Procedure 1:
    Insert 100000 records, with an average of 600 records per second.

    Test with Test Procedure 2:
    Insert 100000 entries, with an average of 1000 entries per second

    Test with test procedure 3:
    You can use the stored procedure to implement the insert operation, and call the stored procedure in the program to implement the insert operation:
    Insert 100000 entries, with an average of 500 entries per second

    Test with Test Procedure 4:
    The test program used is the same as the test program 1, except that the cursor_sharing parameter is set to similar.
    Insert 100000 entries, with an average of 800 entries per second

    Cause Analysis
    In Oracle 9.2.0.1, The cursor_sharing parameter is set to exact by default. Therefore, since each insert statement only has different values, Oracle still causes parse again, which takes a considerable amount of time. Therefore, the insert statement is implemented using the preparedstatement method, effectively avoiding re-parse, so the performance is improved quickly. Similarly, setting the cursor_sharing parameter to similar can achieve similar effects.

    As for the cause of poor insertion performance using a stored procedure, we can analyze it as follows: there are not many tasks implemented using this stored procedure, but a record is simply inserted. At this time, it does not reflect the advantages of stored procedures. Instead, each insert operation calls a stored procedure, which increases the overhead. That is, if you use a stored procedure to access a large amount of data or perform complex queries, it will reflect the benefits of using a stored procedure.

    As for why SQL Server inserts 100000 entries faster than Oracle, I think it is related to the optimization method used by the optimizer. For example, sqlserver uses the Oracle cursor_sharing parameter force mode by default to execute SQL statements.

    I also use stored procedures for testing on Oracle and SQL server. Under the same conditions, Oracle can insert at least 50 thousand entries per second, which is far greater than SQL Server (about 10 thousand ). This also shows that the processing capability of Oracle Data is strong.

    Test procedure 1:
    Import java. SQL .*;
    Import java.net .*;
    Import java. Io .*;
    Import java. util .*;
    // Import oracle. JDBC. oracleconnection;

    Public class failover {

    Static final string user = "test ";
    Static final string Password = "test ";
    Static final string driver_class = "racle. JDBC. Driver. oracledriver ";
    Static final string url = "JDBC: oracle: thin192.168.11.180: 1521: sensky ";


    // Static final string user = "sa ";
    // Static final string Password = "sa ";
    // Static final string driver_class = "com. Microsoft. JDBC. sqlserver. sqlserverdriver ";
    // Static final string url = "JDBC: Microsoft: sqlserver: // 192.168.11.172: 1433; databasename = WAP ";

    Public static void main (string [] ARGs) throws exception {

    Connection conn = NULL;
    String MSG = NULL;
    Statement stmt = NULL;
    Resultset rset = NULL;

    // Load JDBC driver
    Try {
    Class. forname (driver_class );
    }
    Catch (exception e ){
    System. Out. println (E );
    }

    // Connect to the database
    Conn = drivermanager. getconnection (URL, user, password );
    // Conn. setautocommit (false );

    // Create a statement
    Stmt = conn. createstatement ();
    Thread. Sleep (100000 );
    Long start = system. currenttimemillis ();
    // Conn. begintrans ();
    For (long I = 800000; I <900000; I ++ ){
    // Select the names column from the employees table
    Stmt.exe cute ("insert into tbl_test (ID, name, age, ADDR) values (" + I + ", 'asdfasdfsa ', 22, 'aaaaaaaaaaaaaaa ')");
    // Stmt.exe cute ("insert into tbl_test (ID, name, age, ADDR) values (100, 'asdfasdfsa ', 22, 'aaaaaaaaaaaaaaaaa ')");
    // If (I %2000 = 1)
    //{
    // Conn. Commit ();
    //}
    } // End
    // Conn. Commit ();
    Long end = system. currenttimemillis ();
    System. Out. println ("waste time:" + (end-Start ));
    // Close the statement
    Stmt. Close ();
    // Close the connection
    Conn. Close ();
    } // End main ()
    }

    Test Procedure 2:
    Import java. SQL .*;
    Import java.net .*;
    Import java. Io .*;
    Import java. util .*;
    Import oracle. JDBC. oracleconnection;

    Public class prestat {

    Static final string user = "test ";
    Static final string Password = "test ";
    Static final string driver_class = "racle. JDBC. Driver. oracledriver ";
    Static final string url = "JDBC: oracle: thin192.168.11.180: 1521: sensky ";

    Public static void main (string [] ARGs) throws exception {

    Connection conn = NULL;
    String MSG = NULL;
    Statement stmt = NULL;
    Resultset rset = NULL;

    // Load JDBC driver
    Try {
    Class. forname (driver_class );
    }
    Catch (exception e ){
    System. Out. println (E );
    }

    // Connect to the database
    Conn = drivermanager. getconnection (URL, user, password );
    // Conn. setautocommit (false );

    // Create a statement
    Preparedstatement pstat = conn. preparestatement ("insert into tbl_test values (?, ? ,? ,?) ");

    Long start = system. currenttimemillis ();
    // Conn. begintrans ();
    For (INT I = 0; I <100000; I ++ ){
    // Select the names column from the employees table
    Pstat. setint (1, I );
    Pstat. setstring (2, "aaaaaaaa ");
    Pstat. setint (3, I );
    Pstat. setstring (4, "bbbbbbbbb ");
    Pstat.exe cuteupdate (); // JDBC queues this for later execution
    // Conn. Commit ();
    // Sleep one second to make it possible to shutdown the DB.
    } // End
    // Conn. Commit ();
    Long end = system. currenttimemillis ();
    System. Out. println ("waste time:" + (end-Start ));
    // Close the statement
    Pstat. Close ();

    // Close the connection
    Conn. Close ();

    } // End main ()

    } // End class jdemofo

    Test procedure 3:
    Import java. SQL .*;
    Import java.net .*;
    Import java. Io .*;
    Import java. util .*;
    // Import oracle. JDBC. oracleconnection;

    Public class prcejava {

    Static final string user = "test ";
    Static final string Password = "test ";
    Static final string driver_class = "racle. JDBC. Driver. oracledriver ";
    Static final string url = "JDBC: oracle: thin192.168.11.180: 1521: sensky ";


    // Static final string user = "sa ";
    // Static final string Password = "sa ";
    // Static final string driver_class = "com. Microsoft. JDBC. sqlserver. sqlserverdriver ";
    // Static final string url = "JDBC: Microsoft: sqlserver: // 192.168.11.172: 1433; databasename = WAP ";

    Public static void main (string [] ARGs) throws exception {

    Connection conn = NULL;
    String MSG = NULL;
    // Statement stmt = NULL;
    Resultset rset = NULL;

    // Load JDBC driver
    Try {
    Class. forname (driver_class );
    }
    Catch (exception e ){
    System. Out. println (E );
    }

    // Connect to the database
    Conn = drivermanager. getconnection (URL, user, password );
    // Conn. setautocommit (false );

    // Create a statement
    Callablestatement stmt = conn. preparecall ("begin pro_ins (?,?,?,?,?); End ;");
    Long start = system. currenttimemillis ();
    // Conn. begintrans ();
    Int J;
    For (INT I = 0; I <100000; I ++ ){
    // Select the names column from the employees table
    // Stmt.exe cute ("insert into tbl_test (ID, name, age, ADDR) values (" + I + ", 'asdfasdfsa ', 22, 'aaaaaaaaaaaaaaa ')");
    // Stmt.exe cute ("insert into tbl_test (ID, name, age, ADDR) values (100, 'asdfasdfsa ', 22, 'aaaaaaaaaaaaaaaaa ')");

    Stmt. setint (1, I );
    Stmt. setstring (2, "aaaaaaaa ");
    Stmt. setint (3, I );
    Stmt. setstring (4, "bbbbbbbbb ");
    Stmt. registeroutparameter (5, types. tinyint );
    Stmt.exe cute ();
    // J = stmt. getint (1 );
    } // End
    // Conn. Commit ();
    Long end = system. currenttimemillis ();
    System. Out. println ("waste time:" + (end-Start ));
    // Close the statement
    Stmt. Close ();
    // Close the connection
    Conn. Close ();
    } // End main ()
    }

    // Create a stored procedure statement
    // Create or replace procedure pro_ins (p_id in number, p_name in varchar2, p_age in number, p_addr in varchar2, p_return out number)
    //
    // Begin
    // Insert into tbl_test (ID, name, age, ADDR) values (p_id, p_name, p_age, p_addr );
    // P_return: = 1;
    // End;
    ///

    // Create a table statement
    // Create tabel tbl_test (ID number (10, 0), name varchar2 (50), age Number (10, 0), ADDR varchar2 (50 ));

  • Tengrid (18:23:26) according to your test program, it seems that there are two problems, right?
    1. in SQL Server testing, each 600 requests are submitted, while in Oracle, one request is submitted.
    2. In addition, you are using the JDBC thin driver. We recommend you change it to the jdbc oci driver.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.