Database architecture in Big Data Environment

With the rapid growth of data, sub-tables, sub-Libraries, memcache,redis,mongodb,hadoop,bigtable, and so on, a variety of solutions. After testing, in MySQL, regardless of index, data over hundred W, the query time is obvious.

So MySQL sub-table sub-Library +memcache+redis is also a perfect solution.

Because Redis does not support complex queries, the read performance of Redis is still lagging behind Mem's, and therefore requires some blocking.

Process

Look at the underlying architecture from CRUD.

1. Inserting data

In general, the user is not necessary for the table, the user's article or Weibo want $ table, after all, the user table and micro-bo table is incomparable, not a grade.

When inserting, any table, MySQL stores only indexed fields, and other data is stored on Redis. The article table can be a static sub-table according to a certain rule, for example, according to the user ID 100 tables, each user sent the article will be routed to the same table.

The total number of data requires a separate field to be stored in Redis, and each user has a key to store the total number of articles. Sometimes the total data will differ from the actual total number of bars, so MySQL count needs to be performed periodically to update the total number of Redis.

Here is the Sub-table route, the UID to 100 to take the remainder:

protected function Gettablename ($name, $id =null) {   $tableName = self::sys_db_prefix. $name;   if (Is_numeric ($id) && $id > 0)       

2. Querying data

Each time the first data query, according to the index in MySQL from Redis query, need Jakarta casino to store the query results in Memcache, the next query directly from the Memcache get. When you query someone's article, limit queries are made directly from a single table.

About dynamic page query, such as the homepage of the watercress after login, there will be some dynamic friends update information. This information is stored in a temporary table and only 10 days of updates are saved.

3. Delete data

The entire system does not have any actual operation, only the field state modification.

4. Modify the data

When you modify the data, you need to update the corresponding memcache.

Sub-Library

When the system application is more complex, a database server pressure is very large, according to the business flow of the system, such as a separate library, comment on a separate library and so on.

More complex searches

For example, in the recruitment site, by location, by industry, by size and other conditions to inquire, you need a certain search system to complete, here do not do more search details discussion.

Database architecture in Big Data Environment