Memcache persistent Distributed Database MemcacheDB

Memcachedb is xinlang Based on Memcached (the source code is based on Memcached .) Developed a Distributed Key-Value storage persistent open source project. By adding a persistent storage mechanism and an asynchronous master-slave replication mechanism of BerkeleyDB for Memcached, Memcached has the transaction recovery, persistence, and distributed replication capabilities.

Memcachedb is xinlang Based on Memcached (the source code is based on Memcached .) Developed a Distributed Key-Value storage persistent open source project. By adding a persistent storage mechanism and an asynchronous master-slave replication mechanism of Berkeley DB for Memcached, Memcached has the transaction recovery, persistence, and distributed replication capabilities.

Memcachedb is xinlang Based on Memcached (the source code is based on Memcached .) Developed a Distributed Key-Value storage persistent open source project. By adding a persistent storage mechanism and an asynchronous master-slave replication mechanism for Memcached, Memcached has the transaction recovery, persistence, and distributed replication capabilities, it is ideal for applications that require ultra-high read/write speeds and persistent storage. For example, you can apply memcachedb to Sina Blog management. If you have persistence requirements for Memcached, consider using memcachedb.

Features:

1. High-performance key-value read/write

2. Highly Reliable persistent storage and transactions

3. High-performance data storage and Replication

4. Memcahce Protocol

Official website address: http://memcachedb.org/

Project address:

Http://code.google.com/p/memcachedb/

Https://github.com/mplatov/memcachedb

I. Application scenarios

From the perspective of basic data storage, we can see the differences and features of mysql, memcached, and memcachedb. Of course, they certainly do not have this difference.

Because memcached is non-persistent storage, it is positioned as a cache. Memcachedb is not used for caching. It is at the same level as mysql. The difference is that memcachedb has higher performance than mysql, while mysql retrieval is better than memcachedb. From the perspective of caching, memcached is to relieve mysql read pressure, while memcachedb is to relieve mysql write pressure.
Memcachedb is suitable for application scenarios:
1. browsing/clicking/and other statistics can also partially replace the count function of mysql
2. Flag, for example, JE's "What am I doing" Function
3. Frequent writes, including visitor lists and comments

Ii. Installation

Install Libevent and Berkeley Db before installing MemcacheDB.
1. Install libevent first
: Http://www.monkey.org /~ Provos/libevent/

Wget http://jaist.dl.sourceforge.net/project/levent/libevent/libevent-2.0/libevent-2.0.22-stable.tar.gz

Tar-zxvf libevent-2.0.22-stable.tar.gz

Cd libevent-2.0.22-stable

./Configure -- prefix =/home/slim/libevent

Make

Make install

2. Install Berkeley Db (using root)

MemcacheDB 1.2.0 is released, for BerkeleyDB 4.7, so you need to download BerkeleyDB 4.7
: Http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html

Wget http://download.oracle.com/berkeley-db/db-4.7.25.tar.gz

Tar-zxvf db-4.7.25.tar.gz

Cd db-4.7.25

Cd build_unix/

../Dist/configure

Make

Make install

After installation, edit/etc/ld. so. conf and add the path of the Berkeley + Db Runtime Library to the system configuration.

Echo "/usr/local/BerkeleyDB.4.7/lib/">/etc/ld. so. conf

# Reload the system Ld Runtime Library

Ldconfig

3. Install memcachedb
: Http://code.google.com/p/memcachedb/downloads/list
Tar-zxvf memcachedb-1.2.0.tar.gz
Cd memcachedb-1.2.0/
./Configure -- enable-threads -- prefix =/home/slim/memcachedb -- with-libevent-dir =/home/slim/libevent

Make
Make install

Check installation:

./Bin/memcachedb-hmemcachedb 1.2.0-p
 
  
TCP listening port; default value: 21201-U
  
   
UDP listening port. The default value is 0, which is disabled. -S
   
    
Use socket communication. The parameter is to set the unix socket path. This communication method can only be used locally-
    
     
Unix socket File Permission. The default value is 0700-l.
     
      
Listening address. The default value is INDRR_ANY-d. Run-r maximize core file limit-u in daemon mode.
      
        To
       
         Start: only root can be used, because memcachedb cannot start-c with root
        
          Max simultaneous connections, default is 1024-B
         
           Item size smaller
          
            Will use fast memory alloc, default is 512B-v print error and warning information-vv print the entire interaction process-h print this help and exit-I print license info-P
           
             Process PID save path, can only be used in daemon mode-t
            
              Number of threads. The default value is 4 -------------------- BerkeleyDB Options ------------------------------m.
             
               In-memmory cache size of BerkeleyDB in megabytes, default is 64 MB-
              
                Underlying page size in bytes, default is 4096, (512B ~ 64 KB, power-of-two)-f
               
                 Data File name. The default value is 'data. db'-H.
                
                  Data storage directory. The default value is '/data1/memcachedb'-B.
                 
                   Data structure type, which can be 'btree' or 'hash'. The default value is 'btree'-L.
                  
                    Log buffer size in kbytes, default is 32KB-C
                   
                     The number of seconds at which the checkpoint is set to 0 to write data to the hard disk in real time. The default value is 5 minutes-T.
                    
                      Do memp_trickle every
                     
                       Seconds, 0 for disable, default is 30 seconds-e
                      
                        Percent of the pages in the cache that shocould be clean, default is 60%-D
                       
                         Do deadlock detecting every
                        
                          Millisecond, 0 for disable, default is 100 ms-N sets the DB_TXN_NOSYNC flag to improve performance, but sacrifices the transaction persistence. By default, the address and port for data synchronization are disabled on the master server -------------------------- Replication Options ----------------------R, in Master/Slave Mode, you must set-O slave server synchronization source address and port-M/-S to start with master or slave server-n number of slave servers allowed. The default value is 2 -----------------------------------------------------------------------
                        
                       
                      
                     
                    
                   
                  
                 
                
               
              
             
            
           
          
         
        
       
      
     
    
   
  
 

The-C and-N options involve data integrity.

Note:

If the error "./bin/memcachedb: error while loading shared libraries: libevent-2.0.so.5: cannot open shared object file: No such file or directory" is reported.

Add the libevent library path to the System Configuration

Echo "/home/slim/libevent/lib">/etc/ld. so. conf

Ldconfig

4. Create a data storage point

Cd ~ /Memcachedb

Mkdir data/11211-p

Iv. Use

1. standalone Mode

. /Bin/memcachedb-p 11211-d-r-H/home/slim/memcachedb/data/11211-v>/home/slim/memcachedb/data/prepare 11.log 2> & 1

2. Master/Slave Mode

./Bin/memcachedb-p 11211-d-r-H/home/slim/memcachedb/data/11211-N-R 192.168.13.102: 11200-M

Master server, which uses 11200 for Data Synchronization

./Bin/memcachedb-p 11311-d-r-H/home/slim/memcachedb/data/11311-N-R 192.168.13.102: 11300-O 192.168.13.102: 11200-S

Slave server, Master/Slave Mode-R is required.

The slave server cannot store data.

3. Test Synchronization

# telnet 192.168.13.102 11211set yy 0 0 4  yyyy  STORED  get yy  VALUE yy 0 4  yyyy  END  #telnet 192.168.13.102 11311get yy  VALUE yy 0 4  yyyy  END

V. backup and recovery
1. Hot Backup
Because memcachedb is based on bdb, you can use the hot backup tool that comes with bdb to back up data.
[Root @ srv willko] #/usr/local/BerkeleyDB.4.7/bin/db_hotbackup-h/data/21201/-B./bak/
-H is the data directory.
-B is the backup directory.
If it is restored, start the data directory of memcacedb and set it to the backup directory.
2. standalone mode crash recovery
Similar to many transaction engines, bdb writes logs first and stores data in memory. Therefore, after the crash operation, you can use logs to restore it. When starting memcachedb, you can use the-C option to set the checkpoint interval. If it is set too long, the longer the restoration time is.
/Usr/local/BerkeleyDB.4.7/bin/db_recover-cf-h/data/21201/
-H is your data directory
3. Simulate crash

#. /Bin/memcachedb-p 11211-d-r-H/home/slim/memcachedb/data/11211-v>/home/slim/memcachedb/data/prepare 11.log 2> & 1-P/home/slim/memcachedb/data/11211. pid # telnet 192.168.13.102 11211 set t 0 0 4 hhhh # set the key to t and the STORED get t VALUE to hhhh t 0 4 hhhh END quit Connection closed by foreign host. # kill-9 'cat/home/slim/memcachedb/data/11211. pid '# Use-9 to simulate crash #/usr/local/BerkeleyDB.4.7/bin/db_recover-cf-h/home/slim/memcachedb/data/11211 recovery 100% complete #. /bin/memcachedb-p 11211-d-r-H/home/slim/memcachedb/data/11211-v>/home/slim/memcachedb/data/prepare 11.log 2> & 1-P/home/slim/memcachedb/data/11211. pid # telnet 192.168.13.102 11211 get t VALUE t 0 4 hhhh # You can also get the VALUE set before the crash END

4. Master-slave mode crash recovery

In master-slave mode, you can enable the-N option to improve performance. If-N is not used, logs are written to the hard disk in real time, which is why crash can be restored through logs in the future. However, if-N is used to improve the performance, there is a price. That is, if logs are not written to the hard disk in real time, crash cannot guarantee full recovery in the future, and data not written to the hard disk cannot be recovered through logs. But if you do the master-slave mode, you don't have to worry about it, because after the transaction is committed, it will be synchronized to slave immediately. That is to say, after the-N option is used, crash is restored through slave.

References:

Memcachedb User Manual

Introduction, compilation and installation of Memcachedb

Lightweight persistent storage system MemcacheDB

Memcachedb Baidu encyclopedia

Memcachedb cache solution (nginx small File Cache)