Ubuntu 16.04 builds memcached clusters through magent

First, download magent

Official website: https://code.google.com/archive/p/memagent/downloads

Offline Version: (Link: https://pan.baidu.com/s/1kU9DLI3 password: K6QG)

Second, compile and install

Description: Magent because it is a 10 product, so compile will be error, but do not follow the online modification to operate, should first execute make, and then look at the error to solve, because each system solution is different, should according to different systems to deal with is the best solution.

Prerequisite: First install memcached, reference: http://www.cnblogs.com/EasonJim/p/7572264.html

Extract:

mkdir magent    tar-zxvf magent-0.6.tar. gz-c magentsudo mv magent/usr/local
Cd/usr/local/magent

Compile:

1. Add at the beginning of the Ketama.h file

#ifndef ssize_max    #define SSIZE_MAX 32767    #endif

2. Modify the Makefile file

ARCH: = $ (ShellUNAME-M)X64=x86_64cc=Gccprogs=Magentifeq($ (ARCH), $ (X64))M64=-m64 LIBS =/usr/local/lib/libevent.a/usr/lib/x86_64-linux-gnu/libm.soElseLIBS=-levent-lm-l/usr/local/Libendifcflags=-wall-g-o2-i/usr/local/include $ (M64)All :$ (progs)Stprog= MAGENT.O Ketama.Oketama. O:KETAMA.C Ketama.h$ (CC) $ (CFLAGS)-c-o [email protected] Ketama.cmagent. o:magent.c Ketama.h$ (CC) $ (CFLAGS)-c-o [email protected] magent.cmagent:$ (Stprog) $ (CC) $ (CFLAGS)-o [email protected] $^ $ (LIBS)Clean:rm-f *. o *~ $ (progs)

Description: The red part can be modified.

3. Make compilation successful

Iii. Deployment Scenarios and descriptions

How to do memcached cluster and how to be highly available when the website visit reaches a certain level, is the next question to be discussed.

There is a text to describe the "memcached cluster"

Memcached How to handle fault-tolerant?

No Deal! :) In the case of memcached node failure, there is no need for the cluster to do any fault-tolerant processing. If a node fails, the measures to be taken depend entirely on the user. When a node fails, here are a few scenarios to choose from:

* Ignore it! There are many other nodes that can deal with the effect of node failure before the failed node is restored or replaced.

* Remove the failed node from the list of nodes. Be careful with this operation! By default (the remainder hash algorithm), the client adds or removes nodes, causing all cached data to be unavailable! Because the list of nodes for the hash reference changes, most of the keys are mapped to different nodes (as they were) because of the change in the hash value.

* Start the hot standby node and take over the IP occupied by the failed node. This prevents hash disturbances (hashing chaos).

According to the above, memcached one of the nodes fails, memcached itself is not any strategy to maintain the failure of forwarding, which is an unacceptable fact for large systems.

To illustrate:

When the client connection is written to multiple server-side IP addresses, the client will automatically distribute the cached data on each of the different machines:

Defect Description:

If one of the cache nodes of the machine down, then the client's cached data will be lost a part of the figure is described in the red font "losed 33% cache Data", that part of the data is completely gone! If it is the user's critical information then play big,:

Solution: using the Cache proxy Server

Using Magent cache proxy to prevent a single point of the phenomenon, the cache proxy can also do backup, through the client connection to the cache proxy server, the cache proxy Server connection cache server, the cache proxy server can connect more than one memcached machine, as shown, the accessories list is as follows:

magent Proxy Server : 2 units, respectively, 192.168.1.2:12000, 192.168.1.3:12000

memcached master server : 3 units, respectively, 192.168.1.4:11211, 192.168.1.5:11211, 192.168.1.6:11211

memcached Server : 2 units, respectively, 192.168.1.5:11211, 192.168.1.6:11211

build memcahced Server :

Install and run on 192.168.1.4, 192.168.1.5, 192.168.1.6, 192.168.1.7, 192.168.1.8, respectively memcached

build magent Proxy Server :

Compile the installation separately on 192.168.1.2, 192.168.1.3 magent

magent command details (using Google to translate):

-H This message
-U UID
-G GID
-P port, default is 11211. (0 to disable TCP support)
-S Ip:port, set memcached server IP and port
-B ip:port, set backup memcached server IP and port
-l IP, local bind IP address, default is 0.0.0.0
-N number, set Max connections, default is 4096
-D do not go to background
-K Use Ketama key allocation algorithm
-f file, UNIX socket path to listen on. Default is Off
-I number, max keep alive connections for one memcached server, default is 20
-V Verbose

Run the magent on 192.168.1.2, 192.168.1.3, respectively:

Magent-u root-n 51200-l 192.168.1.2-p 12000-s 192.168.1.4:11211-s 192.168.1.5:11211-s 192.168.1.6:11211-b 192.168. 1.7:11211-b 192.168.1.8:11211

Description: Common two-s primary node,-B is the standby node.

test the distribution of cached data Description:

1, Java connection magent, the cache key1 to magent,magent according to their own configuration parameters, coupled with a certain hashing algorithm, will be calculated key1 There are 3 main memcached server on a platform, and then the same algorithm, The Key1 is also on one of the 2 standby memcached servers, and a copy of the data is saved. That is, the primary server is distributed storage, at the same time, from the server is also distributed storage;

2, when the Java obtains the cache data Key1, Magent once knew that the data stored the main memcached server hangs, it will turn to obtain the data from the alternate memcached server. Note: The location selection algorithm of the server is the same as when it is stored.

3, there is a flaw, when the down of the main memcached server back to normal, the memcahed is no data, that is, all the data lost, but at this time the standby memcached server will not synchronize data to the primary server.

4, through the memcached management software memadmin (click to download) to see the above data distribution, as follows:

192.168.1.4 SNSGOU6,SNSGOU3
192.168.1.5 SNSGOU4,SNSGOU1
192.168.1.6 SNSGOU5,SNSGOU2
192.168.1.7 SNSGOU5,SNSGOU3,SNSGOU1
192.168.1.8 SNSGOU4,SNSGOU6,SNSGOU2

5, the Java Connection Agent, it is best to only one per random randomness, so that once an agent hung up (that is, not connected), you can switch to another proxy server. and random to connect, but also to ensure a certain load balance.

Four, the test example:

# Launch Magent instance    /usr/local/magent/magent-u root-n 4096-l 192.168.11.51-p 11200-s 192.168.11.51:11211-s 192. 168.11.52:11211-b 192.168.11.68:11211    /usr/local/magent/magent-u root-n 4096-l 192.168.11.52-p 11200-s 192.168.1 1.51:11211-s 192.168.11.52:11211-b 192.168.11.68:11211

1, login 51 on the magent, storage Key1 to Key5:

Telnet 192.168.11.51 11200Trying192.168.11.51 ...Connected to192.168.11.51.Escape character is ' ^] '.statsmemcached Agent V0.4Matrix1-192.168.11.51:11211, pool size 0Matrix2-192.168.11.52:11211, pool size 0ENDSetKey1 0 0 11STOREDSetKey2 0 0 222STOREDSetKey3 0 0 3333STOREDSetKey4 0 0 44444STOREDSetKey5 0 0 555555Storedquitconnection closed by foreign host.

Log on to memcached on 51 and get to Key2 and Key4:

Telnet 192.168.11.51 11211192.168.11.51 ...  192.168.11.51.Escape character is ' ^] '. Get Key1 END  0 222Endget Key3end0 44444endget key5End  quitconnection closed by foreign host.

Log on to memcached on 52 and get to Key1, Key3, and Key5:

Telnet 192.168.11.52 11211192.168.11.52 ...  192.168.11.52.Escape character is ' ^] '.  0 Endget key2end0 3333endget Key4end  0 555555ENDquitconnection closed by foreign host.

Log on to memcached on 68 and get to Key1 to Key5:

Telnet 192.168.11.68 11211192.168.11.68 ...  192.168.11.68.Escape character is ' ^] '.  0end0 222end0 3333end0 44444End  0 555555ENDquitconnection closed by foreign host.

Stop 52 of the memcached process and get Key1 to Key5 by Magent on 51:

Kill ' cat/var/run/memcached/memcached.pid '

Telnet 192.168.11.51 11200192.168.11.51 ...  192.168.11.51.Escape character is ' ^] '.  0end0 222end0 3333end0 44444End  0 555555ENDquitconnection closed by foreign host.

Recover 52 of the memcached process, through the magent on 51, only get to Key2 and Key4:

Telnet 192.168.11.51 11200192.168.11.51 ...  192.168.11.51.Escape character is ' ^] '. Get Key1 END  0 222Endget Key3end0 44444endget key5End  quitconnection closed by foreign host.

Through the above tests, we can draw a conclusion:

1. The value stored through the Magent connection pool will be available for all memcached of the magent agent.

2. If there is a memcached outage through the Magent proxy mode can also fetch the value.

3. If the value that is taken by the Magent proxy after the memcached repair restart is null, this is because the value inside the memcache after the reboot disappears as the Memcache service stops (because in memory), However, the magent is assigned to a machine by hashing the key, and the memcache is also taken from the machine after the reboot, and all the values taken are empty.

Workaround:

1. After each memcache outage, you can write a program that copies all of the information from other memcache in the cluster to the memcache of the current downtime fix.

2. Write the agent yourself, and then go to the other memcached when the value from a memcached service is null.

Precautions:

Magent is called in the same way as memcached, the client can switch to magent mode without changing the code.

V. How to synchronize the cache with the DB:

It is more safe to do this: when querying from the cache, add, Updae, delete while operating the cache and DB.

Of course, you can also synchronize the cache and DB data regularly, different business should have different choices.

Reference:

http://kb.cnblogs.com/page/69074/

http://carvin.iteye.com/blog/825070

HTTP://ULTRASQL.BLOG.51CTO.COM/9591438/1636374/(solve the problem of compiling magent in Ubuntu)

http://xiemingmei.iteye.com/blog/1418291

Https://www.2cto.com/database/201702/595692.html

http://www.iteye.com/topic/1129147

http://www.iteye.com/topic/1128229

Http://www.cnblogs.com/52php/p/5675711.html (most of the above is transferred from this article)

http://blog.csdn.net/cutesource/article/details/5848253

http://tim-fly.iteye.com/blog/1764519

Http://www.cnblogs.com/zhangrumingbj/p/3887683.html

http://kaibinyuan.blog.51cto.com/7304008/1612364

Http://www.cnblogs.com/szuyuan/p/4866891.html

http://tim-fly.iteye.com/blog/1756936

http://blog.csdn.net/haozhongjun/article/details/52936695

Http://www.cnblogs.com/happyday56/p/3461113.html

http://dba10g.blog.51cto.com/764602/1839379

Ubuntu 16.04 build memcached cluster via magent (Turn)