ASP + sqlsever Big Data solution PK HADOOP

I saw a blog park half a month ago and someone said. NET not that article, I just want to say that you have the time to complain than to write more real things.

1. What are the advantages and disadvantages of SQL Server?

Pros: Support for indexing, transactions, security, and high fault tolerance

Disadvantage: The data volume of more than 1 million need to start optimization, generally we will be horizontal split table, sub-table, partition and job synchronization, which greatly improve the complexity of logic, difficult to maintain, only cluster fault-tolerant, there is no multi-Library load balancing parallel computing capabilities.

2. Does SQL Server really not handle big data?

Answer: Of course, an analogy: the operation of a single database is called a one-dimensional operation, if the same structure, distributed across multiple servers on multiple databases this can be called a two-dimensional operation. We just need a layer of encapsulation of this two-dimensional operation, let him support parallel operations, the server pressure spread, we do not need to write too much, SQL has encapsulated a lot of us, it is like a giant, and we just need to stand on his shoulder, we can easily achieve the big web data processing.

3. is Hadoop suitable for. NET, what are his weaknesses?

(1), data synchronization slow

(2), transaction processing difficult

(3), abnormal catch difficult

(4), it is difficult to combine with ASP, whether it is learning costs, or their own support aspects

(5), need to install, suitable for offline big data processing, but not necessarily suitable for the web

4. What is the sqlsugar framework?

The Sqlsugar is a lightweight, sqlsever-based, high-performance ORM framework that now supports multi-Library parallel computing In addition to the performance that is comparable to that of ADO.

Advantages:

(1), non-latency query for large data

(2), Support distributed transactions

(3), let join Fly up, say goodbye to Big data nojoin

(4), C #. NET home syntax and a lot of encapsulation functions

(5), random storage, that is, can be stored in any node database, to achieve a true load balancing, rather than the previous master-slave mode of Read and write separation.

Disadvantages: SQL Server Licensing fees are too expensive for a wealthy company or a small business that does not pay a license fee.

Sqlsugar Learning Catalogue

0. function Update

1. Sqlsugar Basic Application

2. Using Sqlsugar to process big data

3. Use Sqlsugar to implement join to be updated

4, using Sqlsugar to achieve paging + grouping + multi-column sorting to be updated

5, node fault how to master and slave exchange

"" 2, using Sqlsugar to deal with big data ""

1, the principle of Sqlsugar

Insert: Randomly store to a node database (the probability that each node can configure processing, if set to 0 means that the node will not have new data added)

Update, Delete: Asynchronously requests all database nodes to synchronize summary processing results

Search: The page before the X page, the next X page and pagecount<1000 (1000 This value can be set in the program) of the data is specially optimized, the other data of the asynchronous node algorithm synchronization of the results to sink, performance in the multi-server architecture can be perfectly reflected, In a single-server architecture, it is necessary to ensure sufficient IO to avoid full-table scan, otherwise it will not be optimized.

1, single-server, single-drive, multi-Library architecture:

Suitable for low concurrency, the data volume under 100 million, the response speed is higher requirements, the recommended data volume is best not to exceed 1000W, in the query to avoid full table scan, take full advantage of IO performance, so that the advantages of async reflect.

Fuzzy search for 10 identical libraries deployed on the same PC

Name has a full-text index, ID and NUM have composite indexes built

10 libraries add up to a total of 5.4 million data, ordinary mechanical hard disk only takes 0.3 seconds.

2. Single-server, multi-drive or array:

full table scan can be performed with like, with significant performance improvements

3, multi-server, multi-Library architecture

According to a single PC 10 libraries 5.4 million. 3 seconds to calculate, if there are 10 PC machine that can handle 50 million, time budget in 0.3 seconds-0.5 seconds.

How much can these 10 PCs be replaced with 10 servers?

Summary: The more nodes, the more processing power the server has.

Landa

2. Usage

1. Quoting SqlSugar.dll

2. Configure the connection string

where rate is the chance of storing to a node at INSERT, 0 means no new data is added to the node, and the following setting is 1 means I'm not eccentric at all.

3, add, delete, change usage

4. Enable Distributed transactions

Server needs to turn on services such as MSDTC

5, taskable is the bottom core of all distributed computing

Pages, groups, and other complex queries are expanded from here, supporting DataTable, T:class, value type three types, can easily docu the results of synchronization into a container.

The use of taskable needs to pay attention to the amount of data obtained by each node can not be very large, by taking less, memory operations, in the way of processing complex data in the operation of the query.

6, using taskable for group query

The query of the Statistical class report class, the query result set is not too large, can be processed completely using taskable, the merge method can merge the result set of all library query into a new collection

7, using taskable extension function, let you handle multi-Library operations more convenient.

8. Distributed paging

Considering the library mechanism, the primary key recommends using a GUID to ensure that only the primary key is unique to use the paging function

Pagination is the number of nodes per page display of the current page number, etc. to calculate a preliminary index, and then take out the index location of the data, in the calculation of the real index of the data and page begin comparison in the calculation of a new index until the precise location of the data read out. The principle is like this.

As for the principle I will not speak more, a post is not finished, the lucky friend can add group: 225982985 Discussion

Source Address: Https://github.com/sunkaixuan/SqlSugar

ASP + sqlsever Big Data solution PK HADOOP