Performance & distributed & newlife. xcode supports unlimited data

Last week, codetimer and xcode performance tests were released to demonstrate the performance of newlife. xcode. Actually newlife. xcode is a very common Orm. It only requires a little effort in paging and caching. Pay attention to every detail to ensure good performance in a large data volume and busy business environment.

Newlife. one of the busiest systems that xcode has experienced is a core system in the internet cafe industry. It provides services for five thousand internet cafes and 1 million clients. Tens of thousands of members log on to the Client Multiple times every day. Of course, this 1 million client cannot log on to all the clients at the same time. Because of service requirements, each client can ping the server every several seconds to refresh online records. Only one server runs the server, Windows2003, dual-core CPU, 4G memory, and self-assembly costs 7 krmb. Due to concerns about the limit on the number of TCP connections, WebService + IIS is directly used instead of TCP communication. When three or four IIS sites are enabled and the cache is fully enabled, each process occupies MB to MB of memory. The database is windows2008r2 + mssql2008, with quad-core CPU and 32 GB memory. Self-assembly costs 14 krmb in total, and MSSQL processes account for more than 10 Gb of memory.

This time, the main character is a system (Collection + sorting + website) made by a webmaster using newlife. xcode. Let's take a look at the status quo first.

Server configuration (Foreign, 64-bit platform, 2 GB memory less)

Website creation time: 20 days

Daily traffic: 14000ip 12000pv

Iis cpu: 0 (because the cache hit rate of the website is extremely high)

IIS Memory: 200,000 K * 3 (3 processes)

MSSQL

CPU: 0 (data sorting subsystem write, website read)

Memory: 500,000 KB

Number of important tables: 400 (the table structure is consistent, because the data volume is too large to be split into tables)

Important table data volume: 20,000,000 (20 m * 400 = 8B? 8 billion ?)

Data growth rate: 10 thousand entries per 2 hours

SQLite

Number of important tables: 7

Important table data volume: 20,000,000

Data growth rate: 20 thousand entries per hour

1. The collection subsystem writes the collected data to an SQLite database and needs to be queried during the collection process.

2. The data sorting subsystem analyzes and organizes SQLite data and archives the data to MSSQL.

3. The website reads data in MSSQL to display data based on user queries.

What is the scale of this system? Yesterday, the webmaster told me that as of four o'clock P.M., the advertisement received 173 million knives that day.

In less than a month, there is still little data. What the webmaster is worried about is: how to store the data that will increase infinitely?

The following are the methods currently used by webmasters:

1. Split the table. Xcode has a martial art (for details, see what can be done by the congestion model Orm? -- ORM Component xcode (), which can dynamically change the table name mapped to the object class. As a result, table sharding is performed based on the data category, the data operation method of the object class is reloaded, and the table name is calculated and modified according to the current data category before query and writing, an object class corresponds to multiple data tables with the same structure. In addition, if the data table with the name does not exist, xcode's reverse engineering is automatically created. Users do not need to worry about it at all.CodeJust like using a single table.

2. The collection is separated from the website database. At the beginning, the collection is also written to MSSQL. Obviously, this will make MSSQL very busy and cause the risk of abnormal website due to collection.

3. cache. Websites do not require high real-time data. The collected data can be reflected on the website after an hour or two. Therefore, the cache time can be set to 1 hour for a website to open a level-1 cache. The level-1 cache cannot be set to permanent. Otherwise, the collected new data will no longer be obtained unless the process is restarted. During this period, the cache often becomes invalid. It is found that it is an IIS application.ProgramBecause the pool is recycled, you can set it to be recycled at a fixed time.

  static void  testlog () { newlog  log =  New   newlog  (); log. action =  "test" ; log. category =  "systemlog" ; log. save (); log =  New   newlog  (); log. action =  "test" ; log. category =  "userlog" ; log. save () ;}< span style = "color: Blue"> class   newlog :  log   newlog  { Public override int  insert () { meta . tablename = category;  return base . insert () ;}

In this case, there should be no problem with thousands of tables and 20 million of data in each table.

Of course, there are still some problems.

1. SQLite is frequently written, and sometimes multi-thread conflicts occur. The SQLite provider in xcode adds a retry failure mechanism, reducing the chance of conflict, which is about one in ten

2. SQLite data grows too fast. Obviously, this problem is very serious, but it cannot be solved. In addition to dynamically changing the table name, xcode can also dynamically change the connection name, that is, similar to sharding tables, sharding databases can be easily implemented.

3. Splitting databases can cause Io problems. At this time, more database servers can be used.

4. What if the MSSQL used by the WebSite becomes a bottleneck? You can use multiple MSSQL servers. For example, if there are 10, configure 10 corresponding connection strings in the configuration file, reload the object query method, and modify the connection name dynamically before querying. Which connection name should be used depends on your own implementationAlgorithmThe simplest is round robin or random. In this way, a simple distributed architecture is easily implemented. The new version has built-in distributed providers, which can distribute queries randomly based on weights, and write data to multiple target databases at the same time without modifying the business implementation code.

Don't blame us for being bloated ?), Because we are a congestion model!

Http://xcode.codeplex.com/