Processing millions of data

Source: Internet
Author: User

How to Improve the query speed by processing millions of data:
1. Try to avoid using it in the where clause! = Or <> operator. Otherwise, the engine will discard the index for full table scanning.
2. To optimize the query, try to avoid full table scanning. First, consider creating an index on the columns involved in where and order.
3. Try to avoid null value determination on the field in the where clause. Otherwise, the engine will discard the index and perform full table scanning, for example:
Select id from t where num is null
You can set the default value 0 on num to make sure that the num column in the table does not have a null value, and then query it like this:
Select id from t where num = 0
4. Try to avoid using or in the where clause to connect to the condition. Otherwise, the engine will discard the index and perform full table scanning, for example:
Select id from t where num = 10 or num = 20
You can query it as follows:
Select id from t where num = 10
Union all
Select id from t where num = 20
5. The following query will also cause a full table scan: (the percentage sign cannot be prefixed)
Select id from t where name like '% abc %'
To improve efficiency, you can consider full-text search.
6. Use in and not in with caution. Otherwise, a full table scan may occur, for example:
Select id from t where num in (1, 2, 3)
For continuous values, you can use between instead of in:
Select id from t where num between 1 and 3
8. Avoid performing expression operations on fields in the where clause as much as possible. This will cause the engine to discard the use of indexes for full table scanning. For example:
Select id from t where num/2 = 100
Should be changed:
Select id from t where num = 100*2
9. Avoid performing function operations on fields in the where clause as much as possible, which will cause the engine to stop using the index for full table scanning. For example:
Select id from t where substring (name, 1, 3) = 'abc'-name id starting with abc
Select id from t where datediff (day, createdate, '2017-11-30 ') = 0-'2017-11-30' generated id
Should be changed:
Select id from t where name like 'abc %'
Select id from t where createdate> = '2014-11-30 'and createdate <'2014-12-1 ′
10. do not perform functions, arithmetic operations, or other expression operations on the left side of "=" in the where clause. Otherwise, the system may not be able to correctly use the index.
11. when using an index field as a condition, if the index is a composite index, you must use the first field in the index as the condition to ensure that the system uses the index, otherwise, the index will not be used, and the field order should be consistent with the index order as much as possible.
12. Do not write meaningless queries. If you need to generate an empty table structure:
Select col1, col2 into # t from t where 1 = 0
This type of code will not return any result set, but will consume system resources, should be changed to this:
Create table # t (...)
13. in many cases, replacing in with exists is a good choice:
Select num from a where num in (select num from B)
Replace the following statement:
Select num from a where exists (select 1 from B where num = a. num)
14. not all indexes are valid for queries. SQL queries are optimized based on the data in the table. When there is a large number of duplicate data in the index column, SQL queries may not use indexes, for example, if a table contains sex fields, male and female are almost half of each other, indexing sex does not play a role in query efficiency.
15. the more indexes, the better. Although the index can improve the efficiency of the select statement, it also reduces the efficiency of insert and update, because the insert or update statements may recreate the index, therefore, you need to carefully consider how to create an index, depending on the actual situation. It is recommended that the number of indexes in a table be no more than six. If there are too many indexes, consider whether the indexes on some columns that are not frequently used are necessary.
16. update the clustered index data column should be avoided as much as possible, because the order of the clustered index data column is the physical storage order of the table records. Once the column value changes, the order of the entire table record will be adjusted, it will consume a considerable amount of resources. If the application system needs to frequently update the clustered index data column, consider whether to create the index as a clustered index.
17. use numeric fields whenever possible. If fields containing only numerical information are not designed as numeric fields, this will reduce query and connection performance and increase storage overhead. This is because the engine compares each character in the string one by one during query and connection processing, and only one comparison is required for the number type.
18. try to use varchar/nvarchar instead of char/nchar, because the first step is to reduce the storage space of the variable-length field, which can save storage space. Secondly, for queries, searching in a relatively small field is obviously more efficient.
19. Do not use select * from t anywhere, replace "*" with a specific field list, and do not return any fields that are not used.
20. Try to use table variables instead of temporary tables. If the table variable contains a large amount of data, note that the index is very limited (only the primary key index ).
21. Avoid frequent creation and deletion of temporary tables to reduce the consumption of system table resources.
22. Temporary tables are not unavailable. Using them appropriately can make some routines more effective. For example, when you need to reference large tables or a data set in common tables repeatedly. However, for one-time events, it is best to use the export table.
23. when creating a temporary table, if a large amount of data is inserted at one time, you can use select into instead of create table to avoid creating a large number of logs to increase the speed. If the data volume is small, to ease system table resources, create table first and then insert.
24. if a temporary table is used, you must explicitly delete all temporary tables at the end of the stored procedure. First truncate the table and then drop the table, so that the system table can be locked for a long time.
25. Avoid using a cursor whenever possible, because the efficiency of the cursor is poor. If the cursor operation has more than 10 thousand rows of data, you should consider rewriting.
26. before using the cursor-based or temporary table method, you should first find a set-based solution to solve the problem. The set-based method is generally more effective.
27. Like a temporary table, the cursor is not unavailable. Using a FAST_FORWARD cursor for a small dataset is usually better than other row-by-row processing methods, especially when several tables must be referenced to obtain the required data. A routine that includes "sum" in the result set is usually faster than a cursor. If this is allowed during development, you can try both the cursor-based method and the set-based method to see which method works better.
28. set nocount on at the beginning of all stored procedures and triggers, and set nocount off at the end. You do not need to send the DONE_IN_PROC message to the client after executing each statement of the stored procedure and trigger.
29. Avoid returning large amounts of data to the client as much as possible. If the data volume is too large, consider whether the corresponding requirements are reasonable.
30. Avoid large transaction operations as much as possible to improve the system concurrency capability.
Reasons for slow query speed:
1. No index or no index is used (this is the most common problem of slow query and is a defect in programming)
2. Low I/O throughput, resulting in a bottleneck effect.
3. the query is not optimized because no computing column is created.
4. Insufficient memory
5. slow network speed
6. The queried data volume is too large (you can use multiple queries to reduce the data volume in other ways)
7. Lock or deadlock (this is also the most common problem of slow query and is a defect in programming)
8. sp_lock and sp_who are active users. The reason is that they read and write competing resources.
9. Unnecessary rows and columns are returned.
10. The query statement is not good and is not optimized.
You can optimize the query by using the following methods:
1. Place data, logs, and indexes on different I/O devices to increase the reading speed. In the past, Tempdb can be placed on RAID0, which is not supported by SQL2000. The larger the data size (size), the more important it is to increase I/O.
2. vertically and horizontally split the table to reduce the table size (sp_spaceuse)
3. upgrade hardware
4. Create an index based on the query conditions, optimize the index, optimize the access mode, and limit the data volume of the result set. Note that the fill factor should be appropriate (preferably the default value 0 ). The index should be as small as possible. Use a column with a small number of bytes to create an index (refer to the index creation). Do not create a single index for fields with a limited number of values, such as gender fields.
5. Improve network speed;
6. Expand the server memory. Windows 2000 and SQL server 2000 support 4-8 GB memory. Configure virtual memory: the virtual memory size should be configured based on services running concurrently on the computer. Run Microsoft SQL Server? 2000, you can consider setting the virtual memory size to 1.5 times the physical memory installed on your computer. If you have installed the full-text search feature and intend to run the Microsoft Search Service for full-text indexing and query, consider: set the virtual memory size to at least three times the physical memory installed on the computer. Configure the SQL Server max server memory Server configuration option to 1.5 times the physical memory (half the virtual memory size ).
7. Increase the number of server CPUs. However, you must understand that resources such as memory are more required for concurrent processing of serial processing. Whether to use parallelism or serial travel is automatically evaluated and selected by MsSQL. A single task is divided into multiple tasks and can be run on the processor. For example, if the sort, connection, scan, and group by statements of delayed queries are executed simultaneously, SQL SERVER determines the optimal parallel level based on the system load, complex queries that consume a large amount of CPU are most suitable for parallel processing. However, UPDATE, INSERT, and DELETE operations cannot be processed in parallel.
8. If you use like for query, you cannot simply use index, but the full-text index consumes space. Like 'a % 'when the index like' % a' is used and like '% a %' is not used for the query, the query time is proportional to the total length of the field value, so the CHAR type cannot be used, but VARCHAR. Create a full-text index for a long field value.
9. Separate DB Server and APPLication Server; Separate OLTP and OLAP
10. Distributed partition view can be used to implement Database Server consortium. A consortium is a group of separately managed servers, but they collaborate to share the processing load of the system. This mechanism of forming Database Server consortium through partition data can expand a group of servers to support the processing needs of large multi-layer Web sites. For more information, see designing a database federation server. (Refer to the SQL Help File 'partition view ')
A. before implementing the partition view, a horizontal partition table must be created.
B. After creating a member table, define a distributed partition view on each Member Server, and each view has the same name. In this way, queries that reference the view name of a distributed partition can run on any Member Server. System operations are the same as if each member server has a copy of the original table, but in fact each server has only one member table and a distributed partition view. The data location is transparent to the application.
11. Rebuild the index dbcc reindex, dbcc indexdefrag, shrink data and log dbcc shrinkdb, and dbcc shrinkfile. set automatic log shrinking. for large databases, do not set Automatic database growth, which will reduce the server performance. The writing of T-SQL is very important. The following lists common points: first, the process of DBMS processing the query plan is as follows:
1. query the statement lexical and syntax check
2. submit the statement to the query optimizer of the DBMS.
3. optimizer performs algebra optimization and access path optimization
4. A query plan is generated by the Pre-compilation module.
5. Then, submit it to the system for processing and execution at the appropriate time.
6. Finally, return the execution result to the user. Next, let's take a look at the data storage structure of SQL SERVER: the size of a page is 8 K (8060) bytes, eight pages are a disk area and are stored in the B-tree format.
12. Difference Between Commit and rollback Rollback: Roll back all things. Commit: Submit the current transaction. there is no need to write things in dynamic SQL. If you want to write things, write them out, such as begin tran exec (@ s) commit trans, or write dynamic SQL into functions or stored procedures.
13. Use the Where clause in the Select statement to limit the number of returned rows to avoid table scanning. If unnecessary data is returned, the server's I/O resources are wasted, this increases the burden on the network and reduces performance. If the table is large, the table is locked during the table scan and other connections are prohibited from accessing the table. The consequence is serious.
14. SQL statement comments have no impact on execution
15. Try not to use a cursor. It occupies a large amount of resources. If you need row-by-row execution, try to use non-cursor technology, such as loop on the client, using temporary tables, Table variables, subqueries, and Case statements. The cursor can be classified according to the extraction options it supports: only the rows must be extracted from the first row to the last row. Fetch next is the only allowed extraction operation and is also the default method. You can extract arbitrary rows randomly anywhere in the cursor. The cursor technology becomes very powerful in SQL2000, and its purpose is to support loops.
There are four concurrent options
READ_ONLY: Update cannot be located through the cursor, and there is no lock in the rows that make up the result set.
Optimistic with valueS: OPTIMISTIC Concurrency Control is a standard part of transaction control theory. Optimistic Concurrency control is used in this case. In the interval between opening the cursor and updating the row, there is only a small chance for the second user to update a row. When a cursor is opened with this option, there is no lock to control the rows, which will help maximize its processing capability. If you try to modify a row, the current value of the row is compared with the value obtained from the last row extraction. If any value changes, the server will know that the other person has updated the row and will return an error. If the value is the same, the server executes the modification. Select this concurrency option optimistic with row versioning: this OPTIMISTIC concurrency control option is based on ROW version control. Use row version control. The table must have a version identifier, which can be used by the server to determine whether the row is changed after the cursor is read.
In SQL Server, this performance is provided by the timestamp data type. It is a binary number that indicates the relative sequence of changes in the database. Each database has a global current timestamp value: @ DBTS. Every time you change a row with a timestamp column in any way, SQL Server first stores the current @ DBTS value in the timestamp column, and then adds the value of @ DBTS. If a table has a timestamp column, the timestamp is recorded as a row. The server can compare the current timestamp value of a row with the timestamp value stored during the last extraction to determine whether the row has been updated. The server does not need to compare the values of all columns. You only need to compare the timestamp column. If the application requires Optimistic Concurrency Based on Row Version Control for tables without a timestamp column, the cursor is optimistic concurrency control based on the value by default.
Scroll locks implements pessimistic concurrency control. In pessimistic concurrency control, when the row of the database is read into the cursor result set, the application attempts to lock the row of the database. When a server cursor is used, an update lock is placed on the row when it is read into the cursor. If the cursor is opened in the transaction, the update lock of the transaction will be kept until the transaction is committed or rolled back. When the next row is extracted, the cursor lock will be removed. If the cursor is opened outside the transaction, the lock is discarded when the next row is extracted. Therefore, whenever you need full pessimistic concurrency control, the cursor should be opened in the transaction. The update lock prevents any other task from obtaining the update lock or exclusive lock, thus preventing other tasks from updating the row.
However, the update lock does not prevent the shared lock, so it does not prevent other tasks from reading rows, unless the second task also requires reading with the update lock. Based on the lock prompts specified in the SELECT statement defined by the cursor, these cursor concurrency options can generate a scroll lock. The scroll lock is obtained on each row during extraction and is kept until the next extraction or cursor is closed. The first occurrence prevails. During the next extraction, the server obtains the scroll lock for the Newly Extracted row and releases the scroll lock of the last extracted row. The rolling lock is independent of the transaction lock and can be kept after a commit or rollback operation. If the option to close the cursor when submitting is off, the COMMIT statement does not close any opened cursor, and the scroll lock is retained until it is committed to maintain isolation of the extracted data. The type of the obtained scroll lock depends on the cursor concurrency option and the lock prompt in the SELECT statement of the cursor.
Lock prompt read-only optimistic value optimistic row version control lock No prompt not locked update NOLOCK not locked HOLDLOCK sharing update UPDLOCK error Update TABLOCKX Error unlocked unlocked update other unlocked update * The specified NOLOCK prompt will make the table specified with this prompt read-only in the cursor.
16. Use Profiler to track the query, obtain the time required for the query, and locate the SQL problem. Use the index optimizer to optimize the index.
17. Pay attention to the difference between UNion and UNion all. Good UNION all
18. Use DISTINCT unless necessary. Similar to UNION, it slows down the query. Duplicate records are no problem in the query.
19. Do not return unwanted rows or columns during query.
20. Use sp_configure 'query governor cost limit 'or SET QUERY_GOVERNOR_COST_LIMIT to limit the resources consumed by the query. When the resource consumed by the evaluation query exceeds the limit, the server automatically cancels the query and kills the query before the query. Set locktime: SET the lock time.
21. Use select top 100/10 Percent to limit the number of rows returned by the user or set rowcount to limit the rows to be operated.
22. Before SQL2000, do not use the following statements: "is null", "<> ","! = ","!> ","! <"," NOT "," not exists "," not in "," not like ", and" LIKE '% 100' ", because they do NOT leave the index and are all table scans.
Do not add a function to the column name in the WHere clause, such as Convert and substring. If a function is required, create a computed column and then create an index. you can also change the syntax of where substring (firstname,) = 'M' to WHERE firstname like'm % '(index scan). You must separate the function from the column name. In addition, the index cannot be too large or too large.
Not in scans the table multiple times and uses EXISTS, not exists, IN, left outer join instead, especially the left join. Exists is faster than IN, and the slowest operation is NOT. if the column value is null, its index does not work in the past. Now the 2000 optimizer can process it. The same is null, "NOT", "not exists", "not in" can optimize her, but "<>" cannot be optimized, and no index IS used.
23. Use Query Analyzer to check the SQL statement Query plan and evaluate and analyze whether the SQL statement is optimized. Generally, 20% of the Code occupies 80% of the resources, and our optimization focuses on these slow points.
24. If the in or query is not indexed, use the display statement to specify the INDEX: SELECT * FROM PersonMember (INDEX = IX_Title) WHERE processid IN ('male ', female ')
25. Pre-calculate the results to be queried and place them in the table. SELECT the results when querying. This was the most important method before SQL7.0. For example, hospital hospitalization fee calculation.
26. Appropriate indexes can be used for MIN () and MAX ().
27. There is a principle in the database that the code is closer to the data, the better. Therefore, the Default is the preferred one, which is Rules, Triggers, and Constraint (constraints such as the external key CheckUNIQUE ......, The maximum length of the data type, etc. are constraints), Procedure. This not only requires low maintenance work, high programming quality, and fast execution speed.
28. If you want to INsert a large binary value to the Image column, use the stored procedure. Do not INsert the value using an embedded INsert Statement (whether JAVA is used or not ). In this way, the application first converts the binary value to a string (twice the size of the string), and then converts it to a binary value after the server receives the character. the stored procedure does not have these actions: Method: Create procedure p_insert as insert into table (Fimage) values (@ image), and calls this stored procedure on the foreground to pass in binary parameters, this significantly improves the processing speed.
29. Between is faster IN some cases than IN, and Between can locate the range based on the index faster. Use the query optimizer to see the difference. Select * from chineseresume where title in ('male', 'female ') Select * from chineseresume where between 'male' and 'female' are the same. Because in may be more than once, it may be slower sometimes.
30. If it is necessary to create an index for a global or local temporary table, it may increase the speed, but not necessarily because the index also consumes a lot of resources. Its creation is the same as that of the actual table.
31. Do not create useless things, such as wasting resources when generating reports. Use it only when necessary.
32. The OR clause can be divided into multiple queries and connected to multiple queries through UNION. Their speed is only related to whether an index is used. If a query requires a UNION Index, UNION all is more efficient. no index is used for multiple OR statements, and it is rewritten to the form of UNION to try to match the index. Whether or not indexes are used in a key issue.
33. Use a view as little as possible, which is less efficient. Operations on a view are slower than operations on a table. You can replace it with stored procedure. In particular, do not use nested views. nested views increase the difficulty of searching for original data. Let's look at the essence of the View: it is an optimized SQL statement stored on the server that has produced a query plan. When retrieving data from a single table, do not use a view pointing to multiple tables. Read data directly from the view that only contains the table. Otherwise, unnecessary overhead is added, the query is disturbed. to speed up View query, MsSQL adds the View index function.
34. Do not use DISTINCT or order by unless necessary. These actions can be executed on the client. They increase additional overhead. This is the same as UNION and union all. SELECT top 20 ad. companyname, comid, position, ad. referenceid, worklocation, convert (varchar (10), ad. postDate, 120) as postDate1, workyear, degreedescription FROM your ad where referenceID in ('jcnad00329667 ', 'jcnad132168', 'hangzhou', 'jcnad00338345', 'hangzhou ', 'cnad00303569 ', 'jcnad00303568', 'jcnad00306698 ', 'jcnad00231935', 'hangzhou', 'hangzhou', 'cnad00254585 ', 'hangzhou', 'hangzhou ', 'jcnad00258524', 'jcnad00332379', 'jcnad00268618 ', 'jcnad00279196', 'jcnad00268613 ') order by postdate desc
35. IN the post-IN nominal value list, place the most frequent values at the beginning and the least at the end to reduce the number of judgments.
36. When select into is used, it locks the system table (sysobjects, sysindexes, etc.) and blocks access from other connections. When creating a temporary table, use the show statement instead of select. drop table t_lxh begin tran select * into t_lxh from chineseresume where name = 'xyz' -- commit in another connection SELECT * from sysobjects You Can See That select into locks the system table, create table also locks the system table (whether it is a temporary table or a system table ). So never use it in things !!! In this case, use real tables or temporary table variables for temporary tables that are frequently used.
37. Generally, redundant rows can be removed before group by having clauses, so try not to use them for row removal. Their execution sequence should be optimal as follows: select Where clause Selects all appropriate rows, Group By is used to Group statistical rows, and Having clause is used to remove redundant groups. In this way, the consumption of Group By Having is small, and queries are fast. Grouping and Having large data rows consumes a lot of resources. If the purpose of Group BY is not to include computing, but to Group, it is faster to use Distinct.
38. Updating multiple records at a time is faster than updating multiple records at a time, that is, batch processing is good.
39. Use less temporary tables and replace them with result sets and Table variables. Table variables are better than temporary tables.
40. In SQL2000, calculated fields can be indexed. The following conditions must be met:
A. The expression of calculated fields is definite.
B. Data Types of TEXT, Ntext, and Image cannot be used.
C. The following options must be prepared: ANSI_NULLS = ON, ANSI_PADDINGS = ON ,.......
41. Try to put data processing on the server to reduce network overhead, such as using stored procedures. Stored procedures are compiled, optimized, organized into an execution plan, and stored in the database as SQL statements. They are a collection of control flow languages and are fast. You can use a temporary stored procedure to execute dynamic SQL statements repeatedly. This process (temporary table) is stored in Tempdb. In the past, because SQL SERVER did not support complex mathematical computing, it had to put this job on another layer to increase network overhead. SQL2000 supports UDFs and now supports complex mathematical computing. the return value of a function is not too large, which is costly. User-Defined Functions consume a large amount of resources like the cursor. If a large result is returned, the stored procedure is used.
42. Do not use the same function repeatedly in one sentence, waste resources, and put the result in a variable before calling it faster.
43. The efficiency of select count (*) is low. Try to change his writing style as much as possible, while EXISTS is fast. note the difference: the return values of select count (Field of null) from Table and select count (Field of NOT null) from Table are different.
44. When the server has enough memory, the number of preparation threads = the maximum number of connections + 5, which can maximize the efficiency; otherwise, use the number of prepared threads <maximum number of connections to enable the SQL SERVER thread pool. If the number is equal to the maximum number of connections + 5, the performance of the SERVER is seriously damaged.
45. Access your table in a certain order. If you lock table A and table B first, lock them in this order in all stored procedures. If you first lock table B in A stored procedure and then lock Table A, this may lead to A deadlock. If the lock sequence is not designed in detail in advance, it is difficult to find deadlocks.
46. Monitor the load of the corresponding hardware through SQL Server Performance Monitor Memory: Page Faults/sec counters. If this value increases occasionally, it indicates that there were threads competing for Memory. If it continues high, memory may be the bottleneck. Process:
1.% DPC Time indicates the percentage of the processor used to receive and provide services during the sample interval in the deferred program call (DPC. (DPC is running at a lower priority interval than the standard interval ). Because DPC is executed in privileged mode, the percentage of DPC time is part of the privileged time percentage. These time values are calculated separately and are not part of the total number of interval values. This total number shows the average busy hours as the percentage of instance time.
2. If the value of % Processor Time counter exceeds 95%, the bottleneck is the CPU. You can consider adding a processor or changing a faster processor.
3.% Privileged Time indicates the percentage of idle processor Time used in Privileged mode. (Privileged mode is a processing mode designed for operating system components and operating hardware drivers. It allows direct access to hardware and all memory. Another mode is the user mode. It is a finite processing mode designed for applications, Environment subsystems, and integer subsystems. The operating system converts the application thread to the privileged mode to access the Operating System Service ). The privileged time % includes the time when the service is interrupted and the DPC is provided. The high privileged time ratio may be caused by a large number of failed device intervals. This counter displays the average busy hours as part of the sample time.
4.% User Time indicates CPU-consuming database operations, such as sorting and executing aggregate functions. If the value is very high, you can consider increasing the index and try to reduce the value by using simple table join and horizontal table segmentation methods. Physical Disk: Curretn Disk Queue Length counter this value should not exceed 1.5 ~ 2 times. To improve performance, you can add disks. SQLServer: Cache Hit Ratio counter. The higher the value, the better. If the duration is lower than 80%, consider increasing the memory. Note that the value of this parameter is accumulated after SQL Server is started. Therefore, after running for a period of time, this value cannot reflect the current value of the system.
47. Analyze select emp_name form employee where salary> 3000 in this statement, if salary is of the Float type, the optimizer optimizes it to Convert (float, 3000 ), because 3000 is an integer, we should use 3000.0 during programming, instead of waiting for the DBMS to convert at runtime. Conversion of the same character and integer data.

From the wanglei_smartfish Column

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.