Lightning MDB source Code Analysis (4)-mvcc/cow

This blog post will describe MVCC and cow technologies and how to use and implement both in Lmdb.

COW (Copy on Write):

The idea behind Cow technology is the procrastination technique, the basic approach is that if there are multiple callers who need access to the resource, it is indistinguishable at the time of initialization, that is, for multiple callers, the resource is the same. This will give each

The caller a pointer to the resource. This method persists until the caller needs to modify the resource to be accessed, at which point the caller is assigned a copy of a truly private resource, so that the caller can

Changes are not visible to other callers. All of the above operations are transparent to the caller, and the biggest benefit of this approach is that if the caller does not modify the resource, the private copy is not created. So this technique is more readable than

Written application scenarios, such as virtual memory and paging, databases, string objects, and so on, are used to improve system performance.

MVCC (multiversion concurrency control):

MVCC is a technique of implementing concurrency control in database system and implementing transactional memory control in general system, mainly for concurrency control, using MVCC to read without blocking write, writing not blocking read, only two threads writing the same row of data can cause conflicts.

This can provide the highest level of concurrency. MVCC for concurrent writes of data in the same region can cause conflicts that require transactions to be rolled back or wait for resources to cause deadlocks.

For a database system, if a user is reading the data while another user is writing the same data, the read user may read half of the data or inconsistent data, so the database system will use concurrency control. The simplest way is to

Blocking all reads when writing, this is blocking technology. MVCC is the way that every user sees data that is a snapshot of the database they are connected to, and all writes to the data are not visible to other database users unless the change has been completed (that is, the transaction has been committed).

The update of the data in MVCC is implemented using the Delete (tag) +insert, so there may be different versions of the same row of data in multiple places in the database, the old data and the new data are not in the same place (not the in-place update), but only the last version is up-to-date.

The previous version is valid for the previous transaction. The benefit of MVCC is that for reading data, even if the data is deleted by other users during the entire transaction, the data it reads is the data that was just beginning to use the database. The other is that it doesn't need to be deleted in time

Old data, which avoids the performance degradation caused by the system paging back and forth. For a document database, it can also be optimized for storing data in contiguous chunks, Delete+insert can not update the part of the data, so that the subsequent assembly data to provide the most convenience. MVCC

Provides immediate, consistent views, read-write isolation, and no blocking, so you can increase concurrency.

Diagram of MVCC:

Figure 1: Transaction T1 changes the data V1, changing it to data V2, in the heap, data such as

650) this.width=650; "Src=" http://images.cnitblog.com/blog/424894/201309/ 02153919-73140101454f4f3fad150b233253d80a.png "" 717 "height=" 385 "/>

Figure 2: Transaction T3 changed V2 to V3, in the heap, data such as: Current transaction T2 is still active, so V1 and V2 belong to recently dead state instead of true dead state.

650) this.width=650; "Src=" http://images.cnitblog.com/blog/424894/201309/02154204- A5a7b62d6e8a4c17b6dccef67e3de0a2.png "" 717 "height=" 408 "/>

Figure 3: In terms of visibility, transaction T0 can only see data V1. Because it starts before the transaction T1.

650) this.width=650; "Src=" http://images.cnitblog.com/blog/424894/201309/ 02160429-7f1abbdbe7e44e6b88c92cacfe4d6336.png "" 716 "height=" 415 "/>

Figure 4: After the transaction T1 commits, the transaction T2 started, and the transaction T3 has not been started, so T2 can see the T1 data V2 after the commit.

650) this.width=650; "Src=" http://images.cnitblog.com/blog/424894/201309/02160501- B90046b735bf4e41a132de55d16cb186.png "" 713 "height=" 417 "/>

Figure 5: After transaction T3 commits, the transaction T4 started, so T4 can only see the data V3.

650) this.width=650; "Src=" http://images.cnitblog.com/blog/424894/201309/02163730- C798f077e312459a93de1d721fd3ec10.png "" 716 "height=" 414 "/>

Figure 6: As mentioned earlier, when there are transactional activities accessing data V1 and V2,V1 and V2 states are recently dead.

When T0 and T2 are all over, there is no transaction to access data V1 and V2, when V1 and V2 are dead, so V1 and V2 are vacuum handlers.

650) this.width=650; "Src=" http://images.cnitblog.com/blog/424894/201309/02163904- F1cb7c5c437a4d6e8eb05bb1f8453fb3.png "" 711 "height=" 362 "/>

The above graphs take the MVCC implementation of PostgreSQL as an example to describe the process of reading and writing between different transactions and the data they access. For simultaneous updates, it's mostly more

Update conflict detection, if there is a read-write dependency conflict, the update fails, the transaction must be rolled back, and if there is interdependence, a transaction will be unlocked to avoid deadlocks.

The realization of Mvcc/cow in Lmdb

Lmdb adds a limit to MVCC, which is to allow only one write thread to exist, avoiding write conflicts at the root, and, of course, the concurrency degradation of writes. Because only

A write thread, so there is no need for a series of basic tools to control concurrency, transaction rollback, and data recovery, such as the Wal log, read-write dependency queue, lock queue, and so on. The foundation of MVCC

is cow, for different users, if they do not make any data changes during the entire operation, it will use the same data, if necessary to change, such as

Additions, deletions, modifications, etc., need to be made on the private version of the data, and the modifications are completed before they are visible to other transactions.

Lmdb, the basic unit of data manipulation is the page, so cow is also in the page, corresponding function is mdb_page_touch,mdb_page_copy,copy really implement page copy,

Touch call Copy to complete the copy and then modify Pgno to insert into the b+tree, so that for this transaction, subsequent operations access to the data page is the latest data page, rather than

The data page at which the transaction starts, and the association of this page with other pages is only visible in the list of this transaction page and is not visible to other transactions.

In fact, through the above two functions also realize the core of the MVCC, for the control of Read and write, through the Mdb_txn_begin control, in which the transaction starts to check the read-write lock situation,

If the transaction needs to update the data, it will be blocked, and if the data is read only, the read lock can be obtained regardless of whether a write transaction exists.

One side effect of MVCC is that for applications that have a large number of writes, there are many versions of the data, so the old data takes up a lot of space, and PostgreSQL solves the problem by vacuum,

In Lmdb, the old data page space, which is no longer used, is inserted into a b-tree by freedb, so that the old space can be lmdb after all transactions are no longer accessible.

Eliminate the need to perform cleanup operations on a regular basis. Of course its side effect is that the data can only be kept up to date and cannot be restored to any time, before the vacuum is performed, save all versions of the database

Can be restored to any moment.

This article refers to the following information, together with thanks.

Https://en.wikipedia.org/wiki/Multiversion_concurrency_control

Http://www.kuqin.com/system-analysis/20120319/319108.html

Http://blog.chinaunix.net/uid-20726500-id-4040024.html

Http://www.cnblogs.com/gaojian/p/3295951.html

Lightning MDB source Code Analysis (4)-mvcc/cow