Memcached+magent implementing memcached Clusters

First of all, the following problems exist in memcached

There is no built-in distributed function in itself, it is not possible to use multiple Memcache servers to store different data, maximize the use of the same resources, and unable to synchronize data, which can cause single point of failure. (Memagent proxy Implementation cluster)

There is no limit to the amount of item data that can be saved in memcached, as long as the memory is sufficient.
memcached single process maximum memory is 2G, to use more memory, you can open multiple memcached processes on multiple ports
The maximum 30 days of data expiration, set to permanent will also expire at this time, constant Realtime_maxdelta
60*60*24*30 Control
The maximum key length is 250 bytes, greater than the length cannot be stored, constant key_max_length 250 control
Single item maximum data is 1MB, over 1MB data is not stored, constant Power_block 1048576 is controlled,
It is the default slab size
The maximum number of simultaneous connections is 200, controlled by the freetotal in Conn_init (), the maximum number of soft connections is 1024, by
settings.maxconns=1024 to control
Parameters related to space occupancy: settings.factor=1.25, settings.chunk_size=48, data occupancy and stepping methods that affect slab

Memcached is a non-blocking socket communication mode service, based on the Libevent library, because of non-blocking communication, memory read and write speed very fast.
Memcached sub-server and client, can be configured for multiple server-side and client, applied to distributed services is very extensive.
Memcached as a small-scale data distributed platform is very effective.

Memcached is the key value one by one corresponding, key default maximum can not exceed 128 bytes, value default size is 1M, that is, a slabs, if you want to save 2M value (continuous), can not use two slabs, because two slabs is not continuous, can not be stored in memory, It is necessary to modify the size of the slabs, when multiple keys and value are stored, even if the slabs is not exhausted, no other data will be stored.

Cluster configuration

Because there is no communication between the memcached server and the server, and no data replication backups are made, a single point of failure occurs when any server node fails, and if HA is required, it needs to be addressed in another way.

Through the magent cache proxy, to prevent the single point phenomenon, the cache proxy can also do backup, through the client connection to the cache proxy server, the cache proxy Server connection cache connection server, the cache proxy server can connect more than one memcached machine can be each memcached machine to synchronize data. If one of the cache servers is down, the system can continue to work, and if one of the memcached machines is down, the data is not lost and data integrity can be guaranteed. For details, refer to: http://code.google.com/p/memagent/

memcache implementation of the cluster

Memcached Although it is a "distributed" cache server, there is no distributed functionality on the server side. Each memcached does not communicate with each other to share information. So, how to distribute it? This depends entirely on the client's implementation.

memcached of Distributed

Memcached as a centralized cache, there is a centralized fatal problem: a single point of problem, memcached support multi-instance distributed on multiple machines, only to solve the problem of all data loss, but when one of the machine error, still will cause some data loss , a basket falls on the ground or breaks some of the eggs.

Therefore, it is necessary to implement a backup mechanism, to ensure that the memcached after the partial failure of the data can still be used, of course, we often use the cache to go to the data source to obtain the strategy, but in the SIP scenario, if some information can not find the database to find, So it is very easy to break up sip, so sip is considered credible for the data in memcached, so it is necessary to do cluster.

1. Apply the key that is required to operate, get the client configured in cluster through CacheManager.

2. When the cache client is obtained, perform the cache operation.

3. A. If it is a read operation, when it fails to hit the cluster other cache client gets the data, if it gets to the data, attempts to write to the cache client this time and returns the result. (To achieve the effect of data recovery)

B. In the case of an update operation, the asynchronous update thread that submits the update cluster other machine command to the client is returned immediately after the cache client performs an update operation, and executes asynchronously to the column. (Because if the cache is obtained from key, asynchronous execution does not affect the query operation of this primary key)

problem : If the data is set to timeout, then the process of being copied after the loss becomes permanently valid content.

More and more feel the db, in the face of Tens user Volume application, DB in the face of ordinary curd, especially when the query, already overwhelmed! How to solve the high concurrency of data query efficiency, in the application of more and more important, good nonsense not much to say, first introduce the next magent and memcached

First, Magent

Magent is an open source memcached proxy Server software with the project URL: http://code.google.com/p/memagent/

1. Installation:

1.mkdir magent
2.CD magent/
3.wget http://memagent.googlecode.com/files/magent-0.5.tar.gz
4.tar ZXVF magent-0.5.tar.gz
5./sbin/ldconfig
6.sed-i "s#libs =-levent#libs =-levent-lm#g" Makefile
7.make
8.CP magent/usr/bin/magent
9.cd.. /

2. Magent Command parameter Description:
1.-h this message
2.-u UID
3.-g GID
4.-p port, default is 11211. (0 to disable TCP support)
5.-s Ip:port, set memcached server IP and port
6.-b Ip:port, set backup memcached server IP and port
7.-l IP, local bind IP address, default is 0.0.0.0
8.-n number, set Max connections, default is 4096
9.-d don't go to background
10.-k use Ketama key allocation algorithm
11.-f file, UNIX socket path to listen on. Default is Off
12.-i number, Max keep alive connections for one memcached server, default IS20
13.-v verbose

Second, memcached http://memcached.org/

1. Installing Libevent

1. Tar zxvflibevent-1.4.13-stable.tar.gz
2. Cdlibevent-1.4.13-stable
3. ./configure--PREFIX=/USR
4. Make
5. Make install

2. Installing memcached

1. Tar zxvfmemcached-1.4.5.tar.gz
2. Cdmemcached-1.4.5.tar.gz
3. ./configure--WITH-LIBEVENT=/USR

Configure need to add Libevent installation path

1. Make
2. Make install
3. When the installation is complete, the memcached will be placed on the/usr/local/bin/memcached

Start:/usr/local/bin/memcached-d-M 512-uroot-p 11211-c 10000-m-F 1.1-p/tmp/memcached.pid

Third, the configuration of memcached and magent, no picture no truth, a picture of thousands of words:

Magent and memcached can be mixed, not the rigid one magent s-memcached s-memcached b-memcached

This model has been able to solve a node, a set of server cache data Services, but if the North Netcom set up a group of servers, while in southern Telecom also set up another group of servers, then the two sets of relatively independent nodes between the data synchronization and sharing, The solutions based on Magent and memcached are as follows:

It is important to note that the configuration of the two groups of magent is best consistent, such as:

The northern Magent is configured as: Magent s-memcached1s-memcached2 b-memcached3

Then the southern magent configuration is also: Magent s-memcached1s-memcached2 b-memcached3

The order is consistent because magent is simply using the hash remainder algorithm when assigning keys to memcached.

Of course if you are lazy, then you can directly connect the backup magent, because all the data above.

One particular area to note is:

1: One of the memcached dies, the data is retrieved from the Magent, and the data is removed from the backup memcached to ensure that the user is unaffected.

2:memcached Restart the resurrection, because there is no data after the two memcached restarts, so magent will get a null value, although the backup memcached has data. The memcached can be restored by a scheduled maintenance server.

3. If memcached dies and the backup machine dies at the same time, it can only indicate that you are unlucky enough that you may be able to see God at this moment.

Iv. Java Client

Recommended Use: xmemcached

http://code.google.com/p/xmemcached/

V. Synchronization of the cache with the DB

It is more safe to do this: when querying from the cache, add, Updae, delete while operating the cache and DB.

Of course, you can also synchronize the cache with the DB data, personally think that different business should have a different choice!

I use these two methods in the actual application, such as the user's personal information and so on, using the timing synchronization method.

VI. Search Engine + cache +db

This theme is relatively large and can be divided into:

1. Storage of file structures instead of DB persisted storage.

2. The use of caching in the search engine-the cache implementation of the text library and the index library.

3. Use search engines for unified data queries.

4. Files are read and written synchronously.

Memcached+magent implementing memcached Clusters