Memcached High Performance Memory Object cache system

I. Overview of memcached

• Memcached is a high-performance distributed memory object caching system for dynamic Web applications to mitigate database load. It improves the speed of dynamic, database-driven Web sites by caching data and objects in memory to reduce the number of times a database is read. Memcached is based on a hashmap that stores key/value pairs. Its daemon (daemon) is written in C, but the client can write in any language and communicate with the daemon through the memcached protocol.
• Traditional data are stored in the relational database management system (RDBMS relational database management system), the client request will be read from the RDBMS and displayed in the browser, so that when the traffic is too large or centralized, resulting in Rsbms overload, the database response deteriorated, Serious issues such as delay display in the browser
• Memcached is a set of distributed memory object cache system, mainly through cache database query results, reduce database access times, to improve the response speed of dynamic WEB applications, improve scalability

Second, memcached cache mode

• Unlike local caches such as APC, Memcached is distributed, that is, it is not local, and it is based on a network connection (which, of course, it can also use localhost) to complete the service itself as an application-independent program or daemon (daemon mode)
• Memcached uses the Libevent library (the program library, which encapsulates the event processing capabilities of Linux Epoll, BSD-like operating systems, etc.) to enable network connectivity services, which theoretically can handle an unlimited number of connections, but unlike Apache, It is more of a stable continuous connection, so its actual concurrency is limited, in the conservative case the maximum number of simultaneous connections memcached 200, which is related to the Linux threading ability, of course, this value can be adjusted
• Memcached memory usage also differs from APC. APC is based on shared memory and mmap (mapping a file or other object into memory), MEMCACHD has its own memory allocation algorithm and management mode, it is not related to shared memory, there is no limit for shared memory, typically, Each memcached process can manage 2GB of memory space and increase the number of processes if more space is needed
  

1.Memcached Memory Algorithms

• Memcached uses the [slab allocation][1] mechanism to allocate and manage memory, dividing the allocated memory into chunks of memory of a specific length according to a predetermined size, and then dividing the same memory blocks into groups, which, when stored, match the slab size according to the size of the key value. Find the nearest Slab store

2.Memcached Cache Policy

• LRU (least recently used) plus expiry expiration policy, when storing data items within memcached, you can specify the expiration time of the cache, default to permanent, and when the memcached server runs out of allocated memory, the invalidated data is replaced first, then the data that is not used recently
• In LRU, memcached is using a lazy expiration policy (setting the time to live for a given key, when the key expires (survival time is 0), it is automatically deleted), and does not monitor whether the stored key/vlue is expired. Instead, look at the timestamp of the record when you get the key value, and check to see if the key/value is out of space, which can reduce the load on the server

3.Memcached failure Policy lazy expiration + LRU

• Lazy expiration function: If the data items stored in a very large number of times, at this time to monitor, the cost is considerable, so memcached will not spend too much CPU time on the expiration monitoring, so as to play a certain role in performance optimization
• Lru:memcached will take precedence over the time-out space, but there will be a state of insufficient space for the append information, which will use the least recently used (LRU) mechanism to allocate space, search from records that have not been used recently, and allocate their space to new records

4.Memcached Distributed Algorithms

When Key/value is deposited/removed to the memcached cluster, the Memcache client is credited to that server based on an algorithm (step one: Select a server, step two: Access data)

• Remainder algorithm: The integer hash value of the key is obtained, divided by the number of servers, the server is stored according to the remainder (features: simple, efficient; but poor scalability, when the number of servers changes, almost all of the cache will expire)
• Hashing algorithm: First computes the hash value of the memcached, distributes it on the 0-2^32 circle, and then calculates the hash value of the stored data key and maps it to the circle, and finally begins to search from the location where the data is mapped to, and saves the data in the first server found, if more than 2^ 32 is still not found, the data is saved on the first memcached server; If you add a memcached server, the keys on the first server that are added counterclockwise only on the circle will be affected
  Principle of one-off HA system

After adding a server

Three, memcached use scene

• Memcached should be used in situations where "distributed" is required, does not require "sharing" or is not suitable for small servers, and if memcached is used for local caching, it is dozens of times times slower than PHP's APC
• General memcached are used as the database front-end cache (reducing disk overhead, SQL parsing), and memcached is using memory to cache data, more than the user directly access the database has higher performance, then in large systems, such as access to data frequently, Memcached can greatly reduce database pressure and improve efficiency
• and memcached not only for the database cache, except that the resources in PHP cannot be saved, other data can be stored (string, value, Array, object, Boolean, NULL, binary (picture, video))
  Note: But note that memcached cannot persist data because memcached uses memory cache and memory is volatile (loss of power lost data)
• So where is the memcached bottleneck? Memcached bottleneck is the network, because the memcached with "distributed", need a lot of data exchange, so network bandwidth will often full load

1. Memcached not suitable for sites with little PV
2. Memcached does not apply to data changes frequently, but data changes need to be in the storage situation
3. Memcached is suitable for frequent data changes and frequent queries.
4. Memcached is applicable to the situation that the database changes the frequency, the query frequently

Iv. Principles of memcached 1.Memcached No Cache

• Client Access Web server
• The Web server calls the Memcache client library interface to connect to the memcached server
• If memcached does not cache the data requested by the client, the Memcache client program forwards the request to the database server
• The database server returns data to the Memcache client program
• When the Memcache client program receives the data, it uses the distributed algorithm to cache the data to the specified memcached server for the next use
• After the memcached cache is complete, the Memcache client program returns the data to the Web client, and the Web client returns the data to the client

2.Memcached with Cache

• Client Access Web server
• The Web server calls the Memcache client library interface to connect to the memcached server
• Memcached returns the database to the Memcache client program when the data required by the client is cached in memcached
• The Memcache client then hands the data to the Web server
• The Web server returns data to the client

Memcached will maintain a huge hash table in memory, storing some of the array and files that are often read and written, starting the service process on the server side, specifying the listening IP at startup, its own port number, the memory size used, and several key parameters (single process, single thread, asynchronous i/ O, Event-based (event_based) service mode) uses Libevent as an event notification implementation; Each server simply manages its own data on the client side by specifying the server-side IP address; data in key-->value form, The value of key is converted by hash, and then the distributed algorithm is used to determine which sever to store/fetch the data

V. Case: lamp+memcached

Host	system	IP	Network card	Software
Lamp+memcache (Client)	Centos 6.7 64Bit	192.168.1.10	Vmnet1	LAMP, Memcache
Memcached (Server)	Centos 6.7 64Bit	192.168.1.100	Vmnet1	Libevent memcached

Memcached (Server) 1. Environmental Preparedness (Memcached)

vim /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0                             //网卡名称TYPE=Ethernet                           //网卡类型为以太网ONBOOT=yes                          //开机自启该网卡NM_CONTROLLED=no                        //关闭NetworkManagerBOOTPROTO=static                        //网卡设置为静态方式IPADDR=192.168.1.100                    //IP地址配置NETMASK=255.255.255.0                   //子网掩码配置

/etc/init.d/network restart     //重启网络服务

2. Installing Memcached (Memcached)

Installing Libevent
Libevent is the asynchronous time notification library that memcached relies on to complete the installation as a memcached dependency

tar -zxvf libevent-1.4.9-stable.tar.gz -C /usr/src/

cd /usr/src/libevent-1.4.9-stable/

./configure --prefix=/usr/local/libevent

make && make install

Installing memcached

tar -zxvf memcached-1.2.6.tar.gz -C /usr/src/

cd /usr/src/memcached-1.2.6/

./configure --prefix=/usr/local/memcached --with-libevent=/usr/local/libevent

Options:
--with-libevent: Specify Libevent Event Library location

make && make install

echo "PATH=$PATH:/usr/local/memcached/bin">>/etc/profile

source /etc/profile

memcached -d -m 32m -p 11211 -u root                //启动Memcached服务

Parameter description:
-D: Start as daemon
-M <num>: Amount of memory allocated to memcached, in MB, default 64MB
-U <username>: User running memcached only as root runtime
-L <IP_ADDR>: (lowercase L) the server IP address of the listener, default to the value of the environment variable Indrr_any
-P <num>: Set memcached listening port, preferably more than 1024 ports
-C <num>: Set maximum number of concurrent connections, default is 1024
-P <file>: Set the PID file to save the memcached, use it with the-D selection

Note: The memcached:error while loading shared libraries:libevent-1.4.so.2:cannot open shared object File:no Such fi will appear during startup Le or directory "error, this is because memcached did not find the Libevent library, need to add libevent library file location in the system
Solution:
VIM/ETC/LD.SO.CONF//Open system additional load library definition file
/usr/local/libevent/lib//Add Libevent Event Library folder path
Ldconfig//re-reading/etc/ld.so.conf file contents
Memcached-d-M 32m-p 11211-u root//No error
NETSTAT-UTPLN | grep 11211//view memcahced listening port

Note: Stop memcached, can use "Pkill memcahced" and "Killall memcached" and other related methods

Lamp+memcache (Client) 1. Environmental Preparedness (Lamp+memcache)

vim /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0                             //网卡名称TYPE=Ethernet                           //网卡类型为以太网ONBOOT=yes                          //开机自启该网卡NM_CONTROLLED=no                        //关闭NetworkManagerBOOTPROTO=static                        //网卡设置为静态方式IPADDR=192.168.1.10                     //IP地址配置NETMASK=255.255.255.0                   //子网掩码配置

/etc/init.d/network restart     //重启网络服务

yum -y install zlib-devel libxml2-devel libtool-ltdl-devel      //安装依赖包

2. Install LAMP (lamp+memcache)

Installing httpd

rpm -e httpd httpd-manual webalizer subversion mod_python mod_ssl mod_perl system-config-httpd php php-cli php-ldap php-common mysql dovecot --nodeps//忽略依赖关系卸载系统已安装的RPM包软件（如果有的话，最好输入确认下）

tar -zxvf httpd-2.2.17.tar.gz -C /usr/src/

cd /usr/src/httpd-2.2.17/

./configure --prefix=/usr/local/httpd --enable-so --enable-rewrite --enable-charset-lite --enable-cgi

Options:
--ENABLE-SO: Start dynamic load module support
--enable-rewrite: Start Page address rewriting feature
--enable-charset-lite: Enable character set support to support Web pages that use various character set encodings
--ENABLE-CGI: Enable CGI scripting support

make && make install

echo "PATH=$PATH:/usr/local/httpd/bin">>/etc/profile

source /etc/profile

cp /usr/local/httpd/bin/apachectl /etc/init.d/httpd

vim /etc/init.d/httpd\# chkconfig:35 67 55\# description: Apache Service

chkconfig --add httpd

Install MySQL

rpm -e mysql-server mysql           /卸载使用RPM方式安装MySQL软件包

yum -y install ncurses-devel cmake  //安装依赖软件包

useradd -M -s /sbin/nologin mysql   //新建程序用户并加入mysql组，不允许登陆系统

tar -zxvf mysql-5.5.22.tar.gz -C /usr/src/

cd /usr/src/mysql-5.5.22/

cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DWITH_EXTRA_CHARSETS=all -DSYSCONFDIR=/etc

Options:
-dcmake_install_prefix: Specify MySQL installation location
-ddefault_charset: Specifying the default character set encoding
-ddefault_collation: Specifies that character set proofing rules are used by default
-dwith-extra-charsets: Specify additional support for other character set encodings
-dsysconfdir: Configuration file storage location

make && make install

chown -R mysql:mysql /usr/local/mysql/

cp support-files/my-medium.cnf /etc/my.cnf

/usr/local/mysql/scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql/ --datadir=/usr/local/mysql/data/             //执行初始化脚本

Options:
--user: Specify running User
--basedir: Specifying the MySQL database location
--datadir: Specifying the MySQL data storage location

echo "PATH=$PATH:/usr/local/mysql/bin">>/etc/profile//添加MySQL到搜索路径，方便执行命令

source  /etc/profile                    //立即让profile文件内容生效

cp support-files/mysql.server /etc/init.d/mysqld//拷贝服务脚本文件到/etc/rc.d/init.d位置

chmod +x /etc/init.d/mysqld

chkconfig --level 35 mysqld on

/etc/init.d/mysqld start 或 service mysqld start

netstat -utpln | grep mysqld

Installing the PHP encryption tool

rpm -e php php-cli php-ldap php-common php-mysql --nodeps//忽略依赖关系卸载系统已安装的RPM包软件（如果有的话，最好输入确认下）

tar -zxvf libmcrypt-2.5.8.tar.gz -C /usr/src/

cd /usr/src/libmcrypt-2.5.8/

./configure && make && make install

ln -s /usr/local/lib/libmcrypt.* /usr/lib/

tar -zxvf mhash-0.9.9.9.tar.gz -C /usr/src/

cd /usr/src/mhash-0.9.9.9/

./configure && make && make install

ln -s /usr/local/lib/libmhash.* /usr/lib/

tar -zxvf mcrypt-2.6.8.tar.gz -C /usr/src/

cd /usr/src/mcrypt-2.6.8/

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

./configure && make && make install

Install PHP

tar -zxvf php-5.3.28.tar.gz -C /usr/src/

cd /usr/src/php-5.3.28/

./configure --prefix=/usr/local/php --with-mcrypt --with-apxs2=/usr/local/httpd/bin/apxs --with-config-file-path=/usr/local/php --enable-mbstring --with-mysql=/usr/local/mysql/

Options:
--with-mcrypt: Extension tools support for loading data encryption
--WITH-APXS2: Set the file location of the APXS module support program provided by Apache HTTP Server
--with-config-file-path: Set the PHP configuration file php.ini The location you want to store
--enable-mbstring: Start multibyte string feature to support code such as Chinese
--with-mysql: Setting the installation location of the MySQL database service program

make && make install

PHP Configuration (Lamp+memcache)

cp /usr/src/php-5.3.28/php.ini-development /usr/local/php/php.ini//拷贝PHP模板文件到PHP工作目录

vim /usr/local/php/php.ini        226  short_open_tag = On        //允许识别PHP短语法标记，即<?...?>        784  default_charset = "utf-8"  //设置默认字符集为utf-8，注意删除前面";"

tar -zxvf ZendGuardLoader-php-5.3-linux-glibc23-x86_64.tar.gz  -C /usr/src/

cd /usr/src/ZendGuardLoader-php-5.3-linux-glibc23-x86_64/

cp php-5.3.x/ZendGuardLoader.so /usr/local/php/lib/php/

vim /usr/local/php/php.inizend_extension=/usr/local/php/lib/php/ZendGuardLoader.sozend_loader.enable=1

vim /usr/local/httpd/conf/httpd.conf        310 AddType application/x-httpd-php .php        167 DirectoryIndex index.php index.html

httpd -t

/etc/init.d/httpd start

vim /usr/local/httpd/htdocs/test1.php               //测试PHP网页能否正确显示<?phpphpinfo();?>

vim /usr/local/httpd/htdocs/test3.php<?php        $link=mysql_connect(‘localhost‘,‘root‘,‘‘);        if($link) echo "恭喜你,数据库连接成功啦！";        mysql_close();        ?>

3. Installing Memcache (Lamp+memcache)

tar -zxvf memcache-2.2.7.tgz -C /usr/src/

cd /usr/src/memcache-2.2.7/

/usr/local/php/bin/phpize               //根据系统信息生成对应的configure文件

If an error occurs:
......
Cannot find autoconf. Please check your autoconf installation and the $PHP _autoconf environment variable. Then, rerun the this script.
Workaround:
Yum-y Install autoconf
Note: The above is the solution, only to perform "/usr/local/php/bin/phpize" error when using

./configure --enable-memcache --with-php-config=/usr/local/php/bin/php-config

make && make installBuild complete.Don‘t forget to run ‘make test‘.Installing shared extensions: /usr/local/php/lib/php/extensions/no-debug-non-zts-20090626/      //复制扩展目录

vim /usr/local/php/php.ini        819 extension_dir = "/usr/local/php/lib/php/extensions/no-debug-non-zts-20090626/"      //指定目录        820 extension = memcache.so                             //启动扩展模块

/etc/init.d/httpd restart

vim /usr/local/httpd/htdocs/test2.php<?php$memcache = new Memcache();        $memcache->connect(‘192.168.1.100‘,11211);$memcache->set(‘key‘,‘Memcache test successful!‘,0,60);        $result = $memcache->get(‘key‘); unset($memecache);        echo $result;?>//测试调用memcache程序接口来测试memcached服务器与客户端系统工作是否正常

Vi. Use of memcached

memcached each accessed object has a unique identifier key, access operations are through key, for example, the back-end database can be extracted from the select operation, and then hash the corresponding SQL to derive key, Then use this key to find the data in the memcached, if the data does not exist, it has not been written to the cache, the cache does not exist when the key is stored in the cache, and set a failure time (such as 1 hours), the data during the expiration time is extracted from the cache, This effectively reduces the pressure on the database
Data Access command:

• Set: Save a data to the server, overwrite if it already exists
• Add: Adds a data to the server, but the server must not exist for this key to guarantee that the data will not be overwritten
• Replace: Replaces an already existing data, if the data does not exist, is similar to the SET function
• Get: The format is: Get < key >key; a non-empty string combination, after sending this instruction, wait for the server to return; if there is no data on the server side, then return "End", proof that there is no such key, no data, if there is data, Returns the specified format: VALUE < key >< tag >< data length >
• Delete: remove command; Delete < key >< timeout time >< timeout >-Timeout
• Flush_all: After this instruction is executed, all cached data on the server is deleted and returned
• Stats: Displays status information for all current Memcache server runs

If you just want to get information about some items, you can specify parameters, format: stats < parameters;: This command will return only the project state information for the specified parameter.
Parameters:

stats        Pid：memcache 服务器的进程 ID        uptime：服务器已经运行的秒数        Time：服务器当前的 unix 时间戳        version：memcache 版本        pointer_size：当前操作系统的指针大小（32 位系统一般是 32bit）        rusage_user：进程的累计用户时间        rusage_system：进程的累计系统时间        curr_items：服务器当前存储的 items 数量        Total_items：从服务器启动以后存储的 items 总数量        Bytes：当前服务器存储 items 占用的字节数        curr_connections：当前打开着的连接数        Total_connections：从服务器启动以后曾经打开过的连接数        connection_structures：服务器分配的连接构造数        cmd_get：get 命令（获取）总请求次数        cmd_set：set 命令（保存）总请求次数        get_hits：总命中次数        get_misses：总未命中次数        evictions：为获取空闲内存而删除的items数（分配给memcache的空间用满后需要删除旧的items来得到空间分配给新的items）        Bytes_read：总读取字节数（请求字节数）        Bytes_written：总发送字节数（结果字节数）        Limit_maxbytes：分配给 memcache 的内存大小（字节）        threads：当前线程数   quit：退出

1. Case (Memcached)

yum -y install telnet

Telnet 192.168.1.100 11211//Connect memcached 11211 port Trying 192.168.1.100 ...        Connected to 192.168.1.100. Escape character is ' ^] '. Set key 0 2he//Set the value of Save key STORED//server returns the result, which represents the Save Success Get key//Get key value corresponding to keys 0 2 he endstats//Show All current Memcache Status information of the server running stat PID 12863//memcached initiated process ID stat uptime 4008//How many seconds has it been activated so far stat ti        Me 1464713917//unix time (number of seconds from January 1, 1970) Stat version 1.2.6//memcached release information stat pointer_size 64        Pointer size for current operating system stat Rusage_user 0.278957//Process Cumulative User Time stat Rusage_system 0.371943//Process Cumulative system time Stat Curr_items 1//server currently stores the number of items stat Total_items 2//The total number of items stored after startup from the server stat byt        ES 79//Current server stores the number of bytes used in the items stat curr_connections 2//Current number of concurrent connections stat Total_connections 4    Total number of connections    Stat Connection_structures 3//server allocated number of connection constructs stat Cmd_get 2//number of times the GET command was executed stat Cmd_set 2        Number of times the SET command was executed stat get_hits 2//get hit count stat get_misses 0//get non-hits Stat Evictions 0//number of items deleted for free memory stat bytes_read 98//Total Read bytes (request bytes) STA              T Bytes_written 97//Total Bytes sent (result bytes) stat limit_maxbytes 33554432//allowable maximum memory capacity stat threads 1 Current number of threads Endquit//Exit Connection closed by foreign host.

2. Summary

Error
The error portion of the Memcache protocol is primarily a three-tip instruction for error hints

• Common error message: Error
• Client Error: Client_error < error message >
• Server-side error: Server_error < error messages >

Data Save Instructions

• Data preservation is the basic function, that is, the client through the command to return the data, the server side after receiving processing

Instruction format

  <命令><键><标记><有效期><数据长度>

Set key 0 60 10

Server-side return

• Data saved successfully (STORED)
• Data save failed (not_stored), generally because the server side of this data key already exists
• If there is no return after set, the data length of the set is not used, the server waits for it to be finished.

Memcached High Performance Memory Object cache system