A brief analysis of MongoDB and Redis design principles

Transferred from:http://blog.csdn.net/yangbutao/article/details/8309539

Redis

1, NiO communication because all in memory operation, so the logic of the operation is very fast, reduce the CPU switching overhead, so for single-threaded mode (logical processing thread and the main thread is one). Reactor mode, implement its own multiplexing NiO mechanism (epoll,select,kqueue, etc.) single-threaded multi-Task 2, data structure hash+bucket structure, when the length of the list is too long, will take the migration measures (expansion of the original twice times the hash table,           Migrate data past, Expand+rehash) 3, storage full-scale (traverse Redisdb, read key,value in bucket), save command block the main thread, Bgsave open child process for snapshot persistence operation, generate Rdb file. When the shutdown is called, the save operation data changes, in how many seconds to trigger the Bgsave Sync,master to accept slave the command increment persistence (AOF), first write to the log buffer, and then flush to the log text        (Flush policy can be configured, just a single, can also be batch), only flush to the file, only to really return to the client. To periodically merge the AOF and Rdb files (during the snapshot process, the changed data is written to the AoF buf, after the child process completes the snapshot < memory snapshot>, merges the aofbuf changes and the full mirror data). Mongodb:1, communication multi-threaded way, the main thread to monitor the new connection, after connection, start a new thread to do the data operation (IO switchover), dispatch and IO operations analysis The following is the related thread

–interruptthread---Handle only the semaphore.

–datafilesync::run (background)---Call the Memorymappedfile::flush method to brush the in-memory data (dirty pages) to disk. We know that MongoDB is called Mmap to map the data in the disk into memory, so there must be a mechanism to brush the data to the hard disk to ensure reliability, how often the brush is related to the Syncdelay parameter.

The –fileallocator::run---is used to assign a new file, which determines the size of the allocated file, for example, in double-fold mode.

–durthread--do bulk commit and rollback work.

–snapshotthread::run---will generate snapshot files to help you recover quickly

–clientcursormonitor::run---will manage the user's cursor, call the Idletimereport () method every 4 seconds, and call the Saymemorystatus () method every minute

–periodictask::runner::run---will get recurring task executions from dynamic array std::vector<periodictask* > _tasks

–ttlmonitor::run---ttl, check all DB by calling the Dottlfordb () method.

–replslavethread---is the synchronization thread at the current node as secondary

The –replmasterthread---is the synchronization thread for the current node as master.

–webserverthread

– The main thread that processes database requests

2. Data Structure Database-->collection-->record

Each database has its own separate files. If you turn on the DIRECTORYPERDB option, the files for each library will be placed in a separate folder.

The database file is internally sliced into a single block, with each block holding only one namespace of data. In MongoDB, namespaces are used to differentiate between different storage classes. For example, each collection has a separate namespace, and each index has its own namespace.

In one block, multiple records are saved, each record is in Bson format, and the record is connected to the record through a doubly linked list.

The index data also exists in the data file, but the index is organized into a B tree structure instead of a doubly linked list.

For each database, there is a namespace file that holds the metadata corresponding to each namespace. We can find the storage block location of the corresponding namespace by querying these meta-data.

If you turn on the jorunaling log, there will be files that store all of your operational records.

3, storage mmap way to map the file address to the memory address space, directly manipulate the memory address space can manipulate the file, no longer call Write,read operation.

–datafilesync::run (background)---Call the Memorymappedfile::flush method to brush the in-memory data (dirty pages) to disk. We know that MongoDB is called Mmap to map the data in the disk into memory, so there must be a mechanism to brush the data to the hard disk to ensure reliability, how often the brush is related to the Syncdelay parameter.

Journal (for recovery purposes) is the redo log in MongoDB, while Oplog is the binlog responsible for replication (corresponding to MySQL).

If you turn on journal, you will lose 100ms of data even if the power is lost, which is tolerable for most applications. From 1.9.2+

, MongoDB will turn on the journal feature by default to ensure data security. And the refresh time of journal can be changed, 2-300ms

Range, use the--journalcommitinterval command.

Oplog and data refresh to disk Time is 60s, for replication, do not wait until Oplog flush the disk, in memory can be directly re-

to the Sencondary node.

A brief analysis of MongoDB and Redis design principles