ASP + sqlsever Big Data solution PK HADOOP

I saw a blog park half a month ago and someone said. NET not that article, I just want to say that you have the time to complain than to write more real things.

1. What are the advantages and disadvantages ofSQL Server ?

Pros: Support for indexing, transactions, security, and high fault tolerance

Disadvantage: The data volume of more than 1 million need to start optimization, generally we will be horizontal split table, sub-table, partition and job synchronization, which greatly improve the complexity of logic, difficult to maintain, only cluster fault-tolerant, there is no multi-Library load balancing parallel computing capabilities.

2. DoesSQL Server really not handle big data?

Answer: Of course, an analogy: the operation of a single database is called a one-dimensional operation, if the same structure, distributed across multiple servers on multiple databases this can be called a two-dimensional operation. We just need a layer of encapsulation of this two-dimensional operation, let him support parallel operations, the server pressure spread, we do not need to write too much, SQL has encapsulated a lot of us, it is like a giant, and we just need to stand on his shoulder, we can easily achieve the big web data processing.

3. IsHadoop suitable for. NET, what are his weaknesses?

(1), data synchronization slow

(2), transaction processing difficult

(3), abnormal catch difficult

(4), it is difficult to combine with ASP, whether it is learning costs, or their own support aspects

(5), need to install, suitable for offline big data processing, but not necessarily suitable for the web

4. What is the sqlsugar framework?

The Sqlsugar is a lightweight, sqlsever -based, high-performance ORM framework that now supports multi-Library parallel computing In addition to the performance that is comparable to that of ADO.

Advantages:

(1), non-latency query for large data

(2), Support distributed transactions

(3), let join Fly up, say goodbye to Big data nojoin

(4), C #. NET home syntax and a lot of encapsulation functions

(5), random storage, that is, can be stored in any node database, to achieve a true load balancing, rather than the previous master-slave mode of Read and write separation.

Disadvantages: SQL Server Licensing fees are too expensive for a wealthy company or a small business that does not pay a license fee.

Sqlsugar Learning Catalogue

1. Sqlsugar Basic Application

2. Using Sqlsugar to process big data

3. Use Sqlsugar to implement join to be updated

4, using Sqlsugar to achieve paging + grouping + multi-column sorting to be updated

5, node fault how to master and slave exchange

"" 2,using Sqlsugar to process big Data

1, the principle of Sqlsugar

Insert: Randomly store to a node database (the probability that each node can configure processing, if set to 0 means that the node will not have new data added)

Update, Delete: Asynchronously requests all database nodes to synchronize summary processing results

Search: The page before the X page, the next X page and pagecount<1000 (1000 This value can be set in the program) of the data is specially optimized, the other data of the asynchronous node algorithm synchronization of the results to sink, performance in the multi-server architecture can be perfectly reflected, In a single-server architecture, it is necessary to ensure sufficient IO to avoid full-table scan, otherwise it will not be optimized.

1, single-server, single-drive, multi-Library architecture:

Suitable for low concurrency, data volume under 100 million, high response data is best not more than 1000W, in the query to avoid full table scan, take full advantage of IO performance, so that the advantages of asynchrony.

Fuzzy search for 10 identical libraries deployed on the same PC

Name has a full-text index, ID and NUM have composite indexes built

10 libraries add up to a total of 5.4 million data, ordinary mechanical hard disk only takes 0.3 seconds.

2. Single-server, multi-drive or array:

full table scan can be performed with like, with significant performance improvements

3, multi-server, multi-Library architecture

Because the pressure on the allocation of the nodes of the server, so can easily be more than billion levels of data, the more the node server processing more data, the faster, even if the T-level data, the second check is not a matter, just n cheap pc.

2. Usage

1. Quoting SqlSugar.dll

2. Configure the connection string

where rate is the chance of storing to a node at INSERT, 0 means no new data is added to the node, and the following setting is 1 means I'm not eccentric at all.

2, add, delete, change usage

3. Enable distributed Transactions

Server needs to turn on services such as MSDTC

4, taskable is the bottom core of all distributed computing

Page grouping and other complex operations are expanded from here, supported DataTable, T:class, value type three types, can be easily docu the results of synchronization into a container.

The use of taskable needs to pay attention to the amount of data obtained by each node can not be very large, by taking more and more memory operations, in the way of processing complex data in the operation of the query.

5, using taskable for group query

The query of the Statistical class report class, the query result set is not too large, can be processed completely using taskable, the merge method can merge the result set of all library query into a new collection

6, using taskable extension function, let you handle multi-Library operations more convenient.

7. Distributed paging

Considering the library mechanism, the primary key recommends using a GUID to ensure that only the primary key is unique to use the paging function

As for the principle I will not speak more, a post is not finished, the lucky friend can add group: 225982985 Discussion

Source Address: Https://github.com/sunkaixuan/SqlSugar

Haha, I've tried my best, for good or bad, to give a good ha.

ASP + sqlsever Big Data solution PK HADOOP