There is a table with millions of data, in the query, how to optimize? Answer from database side, Java side and query statement

Original: https://www.2cto.com/database/201612/580140.html

1) Database design aspects:

A. To optimize the query, avoid full-table scanning as far as possible, and first consider indexing on the columns involved in where and order by.

B. You should try to avoid null values in the WHERE clause, otherwise it will cause the engine to discard full table scans using the index, such as: Select ID from t where num is null you can set the default value of 0 on NUM to ensure that the NUM column in the table does not have a null value , and then query: Select ID from t where num=0

C. Not all indexes are valid for queries, and SQL is optimized for queries based on the data in the table, and when there is a large number of data duplication in the index column, the query may not take advantage of the index, as there are fields in the table Sex,male, female almost half, So even if you build an index on sex, it doesn't work for query efficiency.

D. The index is not the more the better, although the index can improve the efficiency of the corresponding select, but also reduce the efficiency of insert and UPDATE, because the INSERT or update when the index may be rebuilt, so how to build the index needs careful consideration, depending on the situation. The number of indexes on a table should not be more than 6, if too many you should consider whether some of the indexes that are not commonly used are necessary.

E. Avoid updating the index data columns as much as possible, because the order of the indexed data columns is the physical storage order of the table records, which can consume considerable resources once the column values change to make the order of the entire table record. If the application needs to update the index data columns frequently, you need to consider whether the index should be built as an index.

F. Use numeric fields as much as possible, and if fields with numeric information are not designed as character types, this can degrade query and connection performance and increase storage overhead. This is because the engine compares each character in a string one at a time while processing queries and joins, and it is sufficient for a numeric type to be compared only once.

G. Use Varchar/nvarchar instead of Char/nchar as much as possible, because the first variable-length field has a small storage space and can save storage space, and secondly for queries, the search efficiency in a relatively small field is obviously higher.

I. Avoid frequent creation and deletion of temporary tables to reduce the consumption of system table resources.

J. Temporary tables are not unusable, and they can be used appropriately to make certain routines more efficient, for example, when you need to repeatedly reference a dataset in a large table or a common table. However, for one-time events, it is best to use an export table.

K. When creating a temporary table, if you insert a large amount of data at one time, you can use SELECT INTO instead of CREATE table to avoid causing a large number of logs to increase speed, and if the amount of data is small, create table and insert to mitigate the resources of the system tables.

L. If a temporary table is used, be sure to explicitly delete all temporary tables at the end of the stored procedure, TRUNCATE table first, and then drop table, which avoids longer locking of the system tables.

2) SQL statement aspect:

A. You should try to avoid using the! = or <> operator in the WHERE clause, or discard the engine for a full table scan using the index.

B. You should try to avoid using or in the WHERE clause to join the condition, otherwise it will cause the engine to discard full table scans using the index, such as: Select ID from t where num=10 or num=20 can query: Select ID from t WH Ere num=10 UNION ALL select IDs from T where num=20

C. In and not is also used with caution, otherwise it will cause a full table scan, such as: Select ID from t where num in (three-to-three) for consecutive values, can be used between do not use in the: Select ID from t where Num between 1 and 3

D. The following query will also cause a full table scan: Select ID from t where name like '%abc% '

E. If you use a parameter in the WHERE clause, it also causes a full table scan. Because SQL resolves local variables only at run time, the optimizer cannot defer the selection of access plans to run time; it must be selected at compile time. However, if an access plan is established at compile time, the value of the variable is still unknown and therefore cannot be selected as an input for the index. The following statement will perform a full table scan: Select ID from t where [email protected] can be changed to force query using index: SELECT ID from T with (index name) where [email Protec Ted

F. Try to avoid expression of the field in the Where clause, which will cause the engine to discard the use of the index for a full table scan. For example: Select ID from t where num/2=100 should be changed to: Select ID from t where num=100*2

G. You should try to avoid function operations on the fields in the WHERE clause, which will cause the engine to discard the full table scan using the index. such as: Select ID from t where substring (name,1,3) = ' abc ' –name with ABC start ID select ID from t where DATEDIFF (day,createdate, ' 2005-11- 30′) =0– ' 2005-11-30 ' generated ID should be changed to: Select ID from t where name like ' abc% ' select ID from t where createdate>= ' 2005-11-30′a nd createdate< ' 2005-12-1′

H. Do not perform functions, arithmetic operations, or other expression operations on the left side of the "=" in the WHERE clause, or the index may not be used correctly by the system.

I. Do not write meaningless queries, such as the need to generate an empty table structure: Select Col1,col2 into #t from T where 1=0 such code will not return any result set, but will consume system resources, should be changed to this: Create TABLE #t (?)

J. Many times replacing in with exists is a good choice: Select num from a where num in (select num from B) is replaced with the following statement: Select Num from a where exists ( Select 1 from b where num=a.num)

K. Do not use SELECT * from t anywhere, use a specific field list instead of "*", and do not return any fields that are not available.

L. Avoid using cursors as much as possible because cursors are inefficient and should be considered for overwriting if the cursor is manipulating more than 10,000 rows of data.

M. Try to avoid returning large amounts of data to the client, and if the amount of data is too large, you should consider whether the corresponding requirements are reasonable.

N. Try to avoid large transaction operations and improve the system concurrency capability.

3) Java aspect:

A. Create as few objects as possible.

B. Reasonable positioning of the system design. A large number of data operations, and a small number of data operations must be separate. A lot of data manipulation, certainly not the ORM framework is fixed. ，

C. Manipulating data using a JDBC link database

D. Control the memory, let the data flow, not all read to the memory and processing, but the edge of reading edge processing;

E. Reasonable use of memory, some data to be cached

There is a table with millions of data, in the query, how to optimize? Answer from database side, Java side and query statement