Memcache Study Notes

The following is mostly from the network, a small part of the problem I encountered after the fusion of records.

Put on a position of rise first ~

Memcache is what

Memcache is a danga.com project, the first to serve LiveJournal, many people around the world use this cache project to build their own heavy load of the site, to share the pressure of the database.
It can handle any number of connections, using non-blocking network IO. Because it works by opening up a space in memory, it then builds a hashtable,memcached self-managing these hashtable.
Memcache Official website: http://www.danga.com/memcached, more detailed information can come here to understand

Why are there two names of Memcache and memcached?
In fact Memcache is the name of this project, and memcached is its server-side main program file name.

Memcache the Use
The use of Memcache website general traffic is relatively large, in order to alleviate the pressure of the database, let Memcache as a cache area, the part of the information stored in memory, in the front-end can be quickly accessed. Then the general focus is to focus on how to share the database pressure and distributed, after all, the memory capacity of a single memcache limited. Here I briefly put forward my personal views, without practice, right when reference.

Distributed applications
Memcache originally support distributed, our client a little transformation, better support. Our key can be appropriately in a regular package, such as the user-oriented site, each users have a customer ID, you can follow a fixed ID to extract and access, such as 1, the beginning of the user is saved on the first memcache server, Data for users starting with 2 are saved on the second Mecache server, and access data is converted and accessed according to the user ID first.

However, this has the disadvantage that the user ID needs to be judged, if the business is inconsistent, or other types of applications may not be appropriate, then according to their actual business to consider, or to think of a more appropriate method.

Reduce database pressure
This is more important, all the data is basically stored in the database, every time the database is frequently accessed, resulting in a very slow database performance, can not serve more users at the same time, such as MySQL, especially frequent lock table, then let memcache to share the pressure of the database. We need to change the current architecture in a way that is small and does not change the front-end on a large scale.

An easy way for me to consider:
Back end of the database operation module, all the select operation is extracted (Update/delete/insert no matter), and then the corresponding SQL for the corresponding hash algorithm to calculate a hash data key (such as MD5 or SHA), Then the key to Memcache to find the data, if the data does not exist, the description has not been written to the cache, then extract the data from the database, an array class format, and then the data in set to Memcache, key is the SQL hash value, Then the corresponding setting of a failure time, such as one hours, then one hours of data are extracted from the cache, effectively reduce the pressure of the database. The disadvantage is that the data is not real-time, when the data has been modified, can not be real-time to the front-end display, and there may be a large memory footprint, after all, the number of select out of the data may be relatively large, this is a factor to consider.

Memcache the security
Our above Memcache server is directly through the client connection after direct operation, there is no verification process, so if the server is directly exposed to the Internet is more dangerous, light data leakage by other unrelated personnel view, heavy server was compromised, Because the Mecache is run as root, and there may be some of our unknown bugs or buffer overflow situation, these are unknown to us, so the danger is predictable. For the sake of safety, I do two suggestions, can be a little to prevent hacking or data leakage.

Intranet access
It is best to put the access between the two servers in the intranet form, usually between the Web server and the Memcache server. Universal servers are two network card, a point to the Internet, a point to the intranet, then let the Web server through the intranet network card to access the Memcache server, we memcache on the server when the boot on the network to listen to the IP address and port, Access between the intranet can effectively block other illegal access.
# memcached-d-M 1024-u root-l 192.168.0.200-p 11211-c 1024-p/tmp/memcached.pid
Memcache server-side setup listens on 11211 ports of 192.168.0.200 IP over the intranet, consumes 1024MB of memory, and allows up to 1024 concurrent connections

Set up a firewall
Firewall is a simple and effective way, if the two servers are hanging in the network, and need to access the Memcache through the extranet IP, then you can consider using a firewall or agent to filter illegal access.
Generally we can use iptables or FreeBSD under the Linux under the IPFW to specify some rules to prevent some illegal access, such as we can set only allow our Web server to access our Memcache server, while blocking other access.
# iptables-f
# iptables-p INPUT DROP
# iptables-a input-p tcp-s 192.168.0.2–dport 11211-j ACCEPT
# iptables-a input-p udp-s 192.168.0.2–dport 11211-j ACCEPT
The iptables rule above is to allow only 192.168.0.2 this Web server to the Memcache server access, can effectively prevent some illegal access, the corresponding can also add some other rules to enhance security, which can be done according to their own needs.

How to install memcached under the Windows platform

First, download memercached for Windows:http://www.2cto.com/uploadfile/2012/0522/20120522094758371.rar Two, installation step www.2cto.com 1, Extract to the specified directory, such as: C:\Memcached\memcached-win32-1.4.4-14.(it doesn't have to be C, I'll pack D, test pass!)2, open the command window with CMD, go to the extracted directory, enter "memcached.exe-d install".(I encountered a problem in this step can not execute the command, Baidu learned that the original is not enough authority, the same can not execute the command of children's shoes, and I like from [Start]->[attachment]->[command Prompt] Right-->[run as Administrator], This opens the command window to execute the command!)3, open the Control Panel, open the service, you can see memcached already on the above can, if not started, then manually start.(in fact, this step does not have to be opened manually, the command can be opened, after the completion of the previous command, the execution of "memcache.exe-d start" can start the service!~ default allocation of 64M memory, using 11211 port)4. Use the Telnet command to verify that the cache server is available.(this telnet command again, f**k Win7, Baidu after the solution is: first open [Control Panel], find [programs and features], and then click on the left [to open or close Windows features], find [Telnet client], the front tick, and then click OK, Then wait a minute and open the [CMD] command after the installation is complete. )there are many times when you need to monitor the memcached running on the server, such as the number of queries cached, the hit rate, and so on. But I found it.
That Memcached-tool was written in Perl under Linux, and I haven't tried windows. Later, I found a simple way to do that is to use Telnet.

1. On the Windows System connection memcached Port cmd command line, type Telnet 192.168.1.1 11211 11211 is the port number of the memcached binding.
2, the connection port after the input stats command, you can get a description of the memcached server operating conditions parameters.    start doing nothing, enter command  stats   View statistics, such as, indicating that the server is functioning properly.    Parameters Introduction  1, the above installation and start are in the default environment, at the time of installation can be set the following parameters:  -p listening port  www.2cto.com   -l connected IP address , the default is native  -d start memcached service  -d restart restart memcached service  -d Stop|shutdown Close the running memcached service  -d Install installs the memcached service  -d uninstall uninstall memcached service  -u Run as (only valid when running as root)  -m maximum memory usage, in megabytes. The default 64mb -m memory is exhausted when an error is returned, instead of deleting the item  -c the maximum number of simultaneous connections, the default is the 1024 -F block size growth factor, the default is 1.25 -n minimum allocation space, key+value+ Flags by default is 48 -h display Help  www.2cto.com    such as: "Memcached-d install-l 127.0.0.1-m 1024-c2048".  2, if you do not add parameters at the time of installation, you can modify the registry information to set up, open the registry, find  hkey_local_machine\system\currentcontrolset\services\ Memcached has a "ImagePath" entry with a value of:  "C:\Memcached\memcached-win32-1.4.4-14\memcached.exe"-D Runservice followed by "-M 1024-c2048 ". such as:   3, enter the stats command, the page appears the parameters introduced.  stat PID 4356 Server process id stat uptime 56625 server uptime, per second  stat time 1225249079 Server current UNIX times  stat Version1 .1.0 Server version number  stat pointer_size 64 stat rusage_user 151.845489 User Time accumulated by the process (seconds: subtle)  stat Rusage_system 121.667603 system time accumulated by the process (seconds: subtle)  stat ibuffer_size 4096  www.2cto.com   stat curr_connections 13 Connections Quantity & nbsp STAT total_connections 54136 The total number of connections received since the server was running  stat connection_structures 318 the number of connection structures allocated by the server  stat cmd_get 100595 Total number of FETCH requests  stat Cmd_set 6510 Total number of storage requests  stat Get_hits 96543 overall number of successful requests  stat get_misses 4052 Total requests failed  stat Bytes_read 4427679 The total number of bytes that the server reads from the network  stat bytes_written 6585596 Total Bytes sent to the network note:  www.2cto.com    Uptime is the number of seconds the memcached is running, Cmd_get is the number of query caches. Cmd_get/uptime gets the average number of requests per second cache. Cmd_set is the number of times the Key=>value is set. The whole memcached is a big hash, with Cmd_get not found content, will call Cmd_set write into the cache. Get_hits is the number of cache hits, and the cache hit ratio = Get_hits/cmd_get *100%. Get_misses plus get_hits equals Cmd_get. Total_itemscurr_items represents the number of key-value pairs that are now in the cache. Total_items = = Cmd_set = = get_misses, but when the maximum available memory is exhausted, if the memcached is set to delete the contents, the above equation will not be memcache Server Security:Memcache server is directly through the client connection after direct operation, there is no verification process, so if the server is directly exposed to the Internet is more dangerous, light data leakage by other unrelated personnel view, heavy server was compromised, In addition, there may be some of our unknown bugs or buffer overflow situation, these are unknown to us, so the danger is predictable. For the sake of security, do two suggestions, can be a little to prevent hacking or data leakage.

The problem with modifying the Memcache server configuration now is described below:

1> provides Web application server calls in the form of an intranet IP, and does not allow direct calls through an external network, such as placing the Memcache server on a 192.168.1.55 server

2> Modify the port, such as change to 11200

3> allocating memory, such as allocating 1024M (1G memory)

Here's how:

1> Start > Run: CMD (OK)

2>CD C:\memcached (carriage return)

3>memcached-m 1024-p 11200-l 192.168.1.55 (carriage return)

Note that at this point the command line does not return to the c:\memcached> state, and the Memcache server is actually quietly becoming a stop state. This window cannot be closed. Open a new CMD window

4> Start > Run: CMD (OK)

5>CD C:\memcached (carriage return)

6>memcached-d Start (enter) to close this cmd window.

You can now use the newly configured memcache server.


Although the above method resolves the problem of modifying the default configuration, there will always be a CMD window that cannot be closed, otherwise it will return to the default configuration of Port 11211.

A better solution is to modify the service's registry configuration by:

1> Start > Run: regedit (enter)

2> found in the registry: hkey_local_machine\system\currentcontrolset\services\memcached Server

3> the value of the default ImagePath key is: "C:\memcached\memcached.exe"-D runservice, instead: "C:\memcached\memcached.exe"-D runservice-m 512-p 11200-l 192.168.1.55 (OK, close the registry)

4> My Computer (right-click) > Management > Services to find memcache services, reboot once to take effect.


At this point, the same network segment of the computer can still use this memcache server, we limit the specified Web application server to be able to use, through the firewall way. If only allow 192.168.1.2 this Web server to the Memcache server access, can effectively prevent some illegal access, the corresponding can also add some other rules to enhance security, which can be done according to their own needs.

Memcache server is directly through the client connection directly after the operation, there is no verification process, so if the server is directly exposed to the Internet is more dangerous, light data leakage by other unrelated personnel view, heavy server was compromised, because Mecache is running with root privileges, In addition, there may be some of our unknown bugs or buffer overflow situation, these are unknown to us, so the danger is predictable. For the sake of safety, I do two suggestions, can be a little to prevent hacking or data leakage.

Intranet access
It is best to put the access between the two servers in the intranet form, usually between the Web server and the Memcache server. Universal servers are two network card, a point to the Internet, a point to the intranet, then let the Web server through the intranet network card to access the Memcache server, we memcache on the server when the boot on the network to listen to the IP address and port, Access between the intranet can effectively block other illegal access.
# memcached-d-M 1024-u root-l 192.168.0.200-p 11211-c 1024-p/tmp/memcached.pid
Memcache server-side setup listens on 11211 ports of 192.168.0.200 IP over the intranet, consumes 1024MB of memory, and allows up to 1024 concurrent connections

Set up a firewall
Firewall is a simple and effective way, if the two servers are hanging in the network, and need to access the Memcache through the extranet IP, then you can consider using a firewall or agent to filter illegal access.
Generally we can use iptables or FreeBSD under the Linux under the IPFW to specify some rules to prevent some illegal access, such as we can set only allow our Web server to access our Memcache server, while blocking other access.
# iptables-f
# iptables-p INPUT DROP
# iptables-a input-p tcp-s 192.168.0.2–dport 11211-j ACCEPT
# iptables-a input-p udp-s 192.168.0.2–dport 11211-j ACCEPT
The iptables rule above is to allow only 192.168.0.2 this Web server to Memcache server access, can effectively prevent some illegal access, the corresponding can also add some other rules to enhance security, which can be done according to their own needs

Memcached client program

There are three types of memcached Java clients:

? Officially provided client based on the traditional blocking IO maintained by Greg Whalin

? Java NIO-based spymemcached implemented by Dustin Sallings

? Xmemcached

1. Three different API comparisons
1) memcached Client for Java

The Memcached Java Client API, which has been introduced earlier, is widely used and runs more stably.

2) spymemcached

A simple, asynchronous, single-threaded memcached client written in Java. Support asynchronous, single-threaded memcached client, using the java1.5 version of concurrent and NIO, access speed will be higher than the former, but the stability is not good, test often reported timeout and other related anomalies.

3) xmemcached

Xmemcached is also a Java NIO-based client, and Java NiO has the advantage of being more efficient (especially at high concurrency) and less resource-intensive than the traditional blocking IO model. Traditional blocking IO in order to increase efficiency, a certain number of connections need to be created to form a connection pool, and NIO requires only one connection (of course, NIO can also be pooled), which in contrast reduces the overhead of thread creation and switching, which is particularly noticeable in high concurrency. So xmemcached and spymemcached in the performance are very good, in some aspects (stored data relatively small case) xmemcached than spymemcached performance is more excellent, specifically can see this Java Memcached Clients Benchmark.

2. Recommendations

memcached Client for Java is recommended because memcached Client for Java publishes a new version, improves performance and runs stably.

Xmemcached is also widely used, and has a more detailed Chinese API documentation, with the following features: High performance, support for a complete protocol, support client distribution, allow to set node weights, dynamic addition and deletion of nodes, support JMX, Integration with spring framework and hibernate-memcached, client connection pooling, scalability, and more.

Here are the sample programs for these three types of clients.

3. Sample Programs
1) memcached Client for Java

From the memcached client program URL of the Java environment described earlier, download the latest version of the client package: Java_memcached-release_2.5.1.zip, unzip, find the folder java_memcached- Release_2.5.1.jar, this is the client's jar package. Add this jar package to the project's build path, and you can use memcached in your project.

The sample code is as follows:

Package temp;

Import com.danga.memcached.*;

Import org.apache.log4j.*;

/** I tested this, the test passed, like a slightly!~** *

public class Cachetest {

public static void Main (string[] args) {

/**

* Initialize Sockiopool, manage memcached connection pool

* */

String[] Servers = {"10.11.15.222:10000"};

Sockiopool pool = sockiopool.getinstance ();

Pool.setservers (servers);

Pool.setfailover (TRUE);

Pool.setinitconn (10);

Pool.setminconn (5);

Pool.setmaxconn (250);

Pool.setmaintsleep (30);

Pool.setnagle (FALSE);

Pool.setsocketto (3000);

Pool.setalivecheck (TRUE);

Pool.initialize ();

/**

* Establish Memcachedclient instances

* */

Memcachedclient memcachedclient = new Memcachedclient ();

for (int i = 0; i <; i++) {

/**

* Adding objects to the memcached cache

* */

Boolean success = Memcachedclient.set ("" + I, "hello!");

/**

* object is taken from the memcached cache by the key value

* */

String result = (string) memcachedclient.get ("" + i);

System.out.println (String.Format ("Set (%d):%s", I, success));

System.out.println (String.Format ("Get (%d):%s", I, result));

}

}

}

2) spymemcached

Spymemcached Current version is version 2.5, the official website is: http://code.google.com/p/spymemcached/. You can download the latest version from address: Http://spymemcached.googlecode.com/files/memcached-2.5.jar to use.

The sample code is as follows:

Package temp;

Import java.net.InetSocketAddress;

Import Java.util.concurrent.Future;

Import net.spy.memcached.MemcachedClient;

public class Testspymemcache {

public static void Main (string[] args) {

Save Object

try {

/* Establish an Memcachedclient instance and specify the IP address and port number of the memcached service */

memcachedclient mc = new Memcachedclient (New Inetsocketaddress ("10.11.15.222", 10000));

Future<boolean> B = null;

/* Set the key value, the expiration time (in seconds), and the object to be cached to memcached */

b = Mc.set ("Neea:testDaF:ksIdno", "someobject");

if (B.get (). Booleanvalue () = = True) {

Mc.shutdown ();

}

} catch (Exception ex) {

Ex.printstacktrace ();

}

Get Object

try {

/* Establish an Memcachedclient instance and specify the IP address and port number of the memcached service */

memcachedclient mc = new Memcachedclient (New Inetsocketaddress ("10.11.15.222", 10000));

/* Find the cache from memcached by key value and return NULL if not present */

Object B = Mc.get ("Neea:testDaF:ksIdno");

System.out.println (B.tostring ());

Mc.shutdown ();

} catch (Exception ex) {

Ex.printstacktrace ();

}

}

}

3) xmemcached

Xmemcached's official website is: http://code.google.com/p/xmemcached/, which can download the latest version of 1.2 from its official website. 4来 use. The address is: http://xmemcached.googlecode.com/files/xmemcached-1.2.4-src.tar.gz.

The sample code is as follows:

Package temp;

Import java.io.IOException;

Import java.util.concurrent.TimeoutException;

Import Net.rubyeye.xmemcached.utils.AddrUtil;

Import net.rubyeye.xmemcached.MemcachedClient;

Import Net.rubyeye.xmemcached.MemcachedClientBuilder;

Import Net.rubyeye.xmemcached.XMemcachedClientBuilder;

Import net.rubyeye.xmemcached.exception.MemcachedException;

public class Testxmemcache {

public static void Main (string[] args) {

Memcachedclientbuilder builder = new Xmemcachedclientbuilder (addrutil

. getaddresses ("10.11.15.222:10000"));

Memcachedclient memcachedclient;

try {

Memcachedclient = Builder.build ();

Memcachedclient.set ("Hello", 0, "hello,xmemcached");

String value = memcachedclient.get ("Hello");

System.out.println ("hello=" + value);

Memcachedclient.delete ("Hello");

Value = Memcachedclient.get ("Hello");

System.out.println ("hello=" + value);

Close Memcached Client

Memcachedclient.shutdown ();

} catch (Memcachedexception e) {

System.err.println ("Memcachedclient operation Fail");

E.printstacktrace ();

} catch (TimeoutException e) {

System.err.println ("Memcachedclient operation timeout");

E.printstacktrace ();

} catch (Interruptedexception e) {

Ignore

}catch (IOException e) {

System.err.println ("Shutdown memcachedclient fail");

E.printstacktrace ();

}

}

}

Memcache Study Notes