Some insights about database consistency and tuning servers

Last night I discussed a question with a friend about how he asked me how to keep the database consistent, and some of the problems that derive from it, I think I should write a blog, at least I think I understand a lot.

Throw a question: What is database consistency?

A: Since the abstract, then give an example, easy to understand, for example, grab a single problem, when there is only 10 pieces of an item, and the number of people involved in the robbery has 100,000, it is clear that the number of items is far from enough, assuming that the 100,000 people at the same time send a request to come in, the first person robbed, the , but the number of those who came in at the same time is not modified, that is, 10, which is obviously illogical; If you think this is not serious enough, I suppose the current product has only 1 left, that is, the 100,000 people to Rob 1, originally the first person robbed, the data will become 0, The next person will not be able to rob, but because the data can not be updated, so those people may also rob, this will cause serious consequences. So that's the question of keeping the database consistent.

Since it is very serious, what can we do to solve this problem?

Here is my opinion, there is the wrong place, hope can point out, the better insight sent to my mailbox:[email protected].

can use the transaction to solve this problem, like the last problem, can be a single action to add a transaction, this action contains the two steps of select and update, the database transaction will keep the two operations consistent, that is, only one user is allowed to take orders at the same time, As long as the select and update the end of the two operations to allow the next user to continue to grab orders, so that the user will grab the data is the latest data, you can maintain the consistency of the data; in fact, I think the database transaction is also a lock mechanism, only the operation to obtain the lock can continue, otherwise it will need to wait.

While resolving this consistency, it leads to the derivation of another problem: assuming that 100,000 people are simultaneously making a request, And assuming each request will take 0.5 seconds, then, the first user spent 0.5 seconds to request to, no problem, the second user spent 1 seconds to request, also no problem, the 100,000th user, it takes 50,000 seconds to wait for the front of the person, this is obviously not practical, this is the problem of the request concurrency.

In fact, this is contradictory, since the database consistency is inevitable, then it will inevitably lead to such a request concurrency problem; Since it cannot be solved, it must be optimized so that it responds faster. This is the time to use the cache.

Here is a point: The database is on the hard disk, and the cache can be memory, memory should be able to respond faster than the hard disk 10,000 times times, so the use of caching to optimize it.

I used the cache has two: one is Memcache, one is a redis, two each has advantages and disadvantages, of which, Memcache is no transaction, it is in the form of a queue, let the user (request) FIFO, and Redis is a business;

About the difference between Redis and memcache:

1, Redis and memcache are stored in memory, are memory database, but memcache can also be used to cache other things, such as video, such as film;

2, Redis can not only store simple key/value types of data, but also provide the storage of data structures such as List,set,hash;

3, virtual memory--redis when the physical memory is low, you can exchange some long-useless value to disk;

4. Expiration policy--memcache specified at set, and Redis can be specified by expire for example

5, distributed-set memcache cluster, using Magent to do a master more from, Redis can do a master more from, can also do a master one from;

6, storage data security--memcache hangs, the data is gone, Redis can be saved to disk periodically (persistent),

7, after the disaster recovery--memcache hangs, the data is not recoverable, Redis can recover through aof;

8, Redis support data backup, that is, Master-slave mode of data backup;

Back to the previous question:

The principle of caching is that when the first request comes in, put this request to the data all into the cache (that is, memory), and then set an expiration time, in this time, if there is the same request come in, then no longer look up the database, directly from memory, when the expiration time has passed, the data in the cache is deleted, Then put the latest request data into the cache, and then set an expiration time, and so on, the memory of the data, certainly a lot faster.

Of course, there is a problem here: As mentioned above in the 100,000 requests, perhaps really effective request is not enough 1000, because there may be a hacker wrote a program, constantly execute this request, his purpose is to bring down your server, how to solve the problem?

In fact, the idea of this problem is very simple, is to detect his IP address, the general user in a period of time such as 10 seconds, the number of requests generally not more than 10 or 5 times, then you can detect the IP in a period of time such as 10 seconds of access, if a certain number such as 10 times, It is only a thought to mark a sensitive address and manipulate it accordingly.

Some insights about database consistency and tuning servers