Memcached detailed introduction, installation, high availability, distributed cases, monitoring

Memcached is a set of memory cache system or software, used in dynamic application system to cache database data, reduce database access pressure, achieve performance improvement purposes

1, commonly used in the enterprise database cache

2, as a storage for session data sharing between cluster architecture node application servers

Memcached is to store data by pre-allocating the specified memory space, Java, PHP applications, and more than 1000 single mysql200-300 is already very large.

Memcached (cannot persist), Redis (persistent cache) cache of back-end databases, dynamic data. Blog, BBS

Nginx, squid,varnish cache data cache for front-end Web applications, images, attachments, js,css,html

Need to allocate a piece of memory in advance, through API access, read in-memory cache data, each piece of data is stored in key-value form

When a program is updated to delete data that is already in the database, the client simultaneously sends a request to notify memcached that the old data for the same ID content that has been cached is invalidated, thus guaranteeing the data in the memecached and the database

Data consistency

In the case of high concurrency, in addition to notifying the memcached of expired cache invalidation, it will also be through the mechanism to enable users to access new data, through the program pre-pushed the updated data into the memcached cache, and then

Access, which can reduce the pressure of data access, so that the efficiency is very high

Disadvantages:

1, if the memcached reboot (power outage), the cache is lost, may cause data avalanche

How to deal with??

First, the front-end proxy settings prohibit external access, through the program to initialize, the MySQL data cache, cache to memcached, and then slowly mount the server, and then let the front-end to find the cache

When the memcached memory space is exhausted, the memecached itself will be used (LRU least recently used least recently) added to the option invalidation strategy, the invalid data is first replaced, and then is not used recently

The data is replaced

Almost all sites, when the traffic is very large, the first to appear is the database role server, and storage role server bottlenecks, we design should rely on the principle of the former

Tens PV/IP scale High performance concurrent Web site architecture

http://oldboy.blog.51cto.com/2561410/736710

memcached "Distributed Application 1"

memcached support distributed, we can be modified on the application server program, it is good to support, for example, our key can be properly carried out by the regular

Encapsulation, such as a user-oriented web site, each user has a userid, then can follow a fixed ID to extract and access, such as 1-100w beginning to save the user on the first memcached server, to

100-200w the beginning of the storage on the second server, access to data are first followed by UserID for the corresponding conversion store.

But this has the disadvantage that it is necessary to judge the UserID, if the business is inconsistent, or other types of applications, it is certainly not so appropriate, according to their own business judgment.

memcached "Distributed Application 2"

In the application server through the program Url_hash, the suppression hashing algorithm accesses the memcached service, all memcached server address pools can be simple in each program configuration file

Memcached "Distributed Application 3"

Portal Baidu. is responsible for requesting the backend cache service through a middleware proxy and then connecting to the Web

Memcached "Distributed Application 4"

You can use the common LVs haproxy to do cache load balancing. The key is the scheduling algorithm, the cache general choice Url_hash and the consistent hash algorithm, will sacrifice the performance of LVS

Memcahed Service Application Optimization case

The database is responsible for very high load 20-30

Log in to Database

Show Processlist;

Show full processlist;

Mysql-u-p-e "Show full Processlist | Grep-vi Sleep "

The results show that like "* * *" similar statements are many

Optimization ideas:

1, from the business implementation of the user login after the search, you can reduce the number of searches, thereby reducing the database pressure

2, if there is a large number of ordinary search, generally crawler crawling your site, IP sealed off

3, configure master-slave synchronization, the program to achieve read and write separation, the best case like the statement to go from the library query, reduce the main library pressure

4, generally like the statement, generally difficult to optimize in MySQL, you can search service Sphinx to achieve the search

5. Add memcached server to the front of the database

Memcached Features:

Memcached, as a high-concurrency, high-performance caching service, has the following characteristics:

Simple protocol

Memcached protocol implementation is relatively simple, using the Basic text protocol, can be operated between Telent memcached server access to data

Libevent-based event handling

Built-in memory management method (slab allocation)

This built-in management is efficient, all data is stored in the Memecache, when the amount of data deposited in the full space, memcached using the URL algorithm to automatically delete the non-applicable cache data, that is, the reuse of outdated data within the

Storage space. But easy to lose data, can be developed with Sina memcacedb Persistent memory cache system, of course, there are redis,mongodb

memcached servers do not communicate, are independent storage of data, do not share any information, through the design of the client, so that memcached has distributed, support large-scale application of massive data

.

memcached How the software works:

Memcached is a C/s architecture software that starts the service daemon on the server and can specify the listening IP address, port number, number of concurrent connections, and how much memory is allocated for the memcached servers to handle the client's request parameters.

Number

With asynchronous I/O, the application can connect to the Memcached service by specifying the IP address of the server and the port to communicate with each other

The data that needs to be cached is stored in the server-side allocated memory in the form of a Key/value key-value pair, and each cached data has a unique key, and the data in the operation Memcached is performed by this unique key.

The data that is cached to memcached is placed in the allocated memory instead of being allocated to disk, so the readings are very fast

Considerations, consider the impact of restarting lost data and the loss of data in high concurrency scenarios

memcached Delete mechanism

Memcached does not release allocated memory space (unless the setting expires when the data is added or when the memory is full)

Lazy expiration policy, you do not monitor the key/value to expire, but get the key value to view the record timestamp check Key/value to the space is expired, this policy will not expire check on the waste CPU

Resources

The LRU algorithm allocates space, removes the least recently used Key/value pair, and if the memory is large enough that you do not want to use the LRU algorithm, then you can start with the-m parameter so that the memcached will return when the memory runs out

An error

Installing memcached

1,wget http://www.monkey.org/~provos/libevent-1.4.13-stable.tar.gz

[Email protected] ~]# tar xvf libevent-1.4.13-stable.tar.gz

[Email protected] ~]# CD libevent-1.4.13-stable

[Email protected] libevent-1.4.13-stable]#

E

Make

Make install

2,memcached

Divided into client and server side

MEMCACHED-2.2.5.TGZ---Client

Memcached-1.4.15.tar--server

wget http://memcached.googlecode.com/memcached-1.4.13.tar.gz

CD memcached-1.4.15

1004./configure && make && make install

1010 ls/usr/local/lib/

1011 echo "/usr/local/lib" >>/etc/ld.so.conf

1012 Ldconfig

1013 which memcached

/usr/local/bin/memcached

Memcached 1.4.15

-P <num> TCP port number to listen on (default:11211)

-U <num> UDP port number to listen on (default:11211, 0 is off)

-L <addr> interface to listen in (Default:inaddr_any, all addresses)

<addr> may specified as Host:port. If you don ' t specify

A port number, the value of specified With-p or-u is

Used. Specify multiple addresses separated by comma

or by using-l multiple times

-D Run as a daemon

-R Maximize Core file limit

-U <username> assume identity of <username> (only if run as root)

-M <num> max memory to use for items in megabytes (default:64 MB)

-M return error on memory exhausted (rather than removing items)

-C <num> Max simultaneous connections (default:1024) concurrency

-P <file> Save PID in <file>, only used with-d option

Memcached-p 11211-u root-m 16m-c 10240-d

[Email protected] ~]# lsof-i:11211

COMMAND PID USER FD TYPE DEVICE size/off NODE NAME

memcached 20124 root 26u IPv4 139731 0t0 TCP *:memcache (LISTEN)

memcached 20124 root 27u IPv6 139732 0t0 TCP *:memcache (LISTEN)

memcached 20124 root 28u IPv4 139735 0t0 UDP *:memcache

memcached 20124 root 29u IPv6 139736 0t0 UDP *:memcache

Start memcached multiple instances are relatively independent of each instance

Memcached-p 11212-u root-m 16m-c 10240-d

/etc/rc.local

(4) Write data check results

Adding data to the memcached

1, via NC Write

[[email protected] ~]# printf "set key007 0 0 10\r\noldboy0987\r\n" |nc 127.0.0.1 11211

STORED the byte of the----command is 10, then 10 characters later, otherwise the addition is unsuccessful

[[email protected] ~]# printf "Get key007\r\n" | NC 127.0.0.1 11211

VALUE key007 0 10

oldboy0987

END

[Ro[email protected] ~]# printf "set key001 0 0 10\r\noldboy0098\r\n" |nc 127.0.0.1 11211

STORED

Delete:

[[email protected] ~]# printf "Delete key001\r\n" | NC 127.0.0.1 11211

DELETED

[[email protected] ~]# printf "Get key001\r\n" | NC 127.0.0.1 11211

END

Telent 127.0.0.1 11211

Stats---view memcached service status

STAT PID 20124

STAT Uptime 10911

STAT Time 1437183342

STAT version 1.4.15

STAT libevent 1.4.13-stable

STAT Pointer_size 64

STAT Rusage_user 0.146977

STAT Rusage_system 0.469928

STAT Curr_connections 10

STAT Total_connections 54

STAT Connection_structures 12

STAT Reserved_fds 20

STAT Cmd_get 28

STAT Cmd_set 4

STAT Cmd_flush 0

STAT Cmd_touch 0

STAT get_hits---------hit rate, incremented by 1 per get

STAT get_misses---------Loss Rate

STAT delete_misses 0

STAT Delete_hits 1

STAT incr_misses 0

STAT Incr_hits 0

STAT decr_misses 0

STAT Decr_hits 0

STAT cas_misses 0

STAT Cas_hits 0

STAT Cas_badval 0

STAT Touch_hits 0

STAT touch_misses 0

STAT Auth_cmds 0

STAT auth_errors 0

STAT Bytes_read 3888

STAT Bytes_written 2571

STAT limit_maxbytes 16777216

STAT Accepting_conns 1

STAT Listen_disabled_num 0

STAT Threads 4

STAT Conn_yields 0

STAT Hash_power_level 16

STAT hash_bytes 524288

STAT hash_is_expanding 0

STAT bytes 161

STAT Curr_items 2

STAT Total_items 3 Adds one, this value is incremented

STAT expired_unfetched 0

STAT evicted_unfetched 0

STAT Evictions 0

STAT reclaimed 0

END

Restart memcached, data loss

[[email protected] ~]# printf "Get key\r\n" |nc 127.0.0.1 11211

VALUE Key 0 10

Wybwyb

END

[Email protected] ~]# Ps-ef | grep memcached

Root 20124 1 0 06:33? 00:00:00 memcached-p 11211-u root-m 16m-c 10240-d

Root 22920 22899 0 09:47 pts/1 00:00:00 grep memcached

[Email protected] ~]# kill-9 20124

[Email protected] ~]# Ps-ef | grep memcached

Root 22922 22899 0 09:48 pts/1 00:00:00 grep memcached

[Email protected] ~]# memcached-p 11211-u root-m 16m-c 10240-d

[Email protected] ~]# Ps-ef | grep memcached

Root 22924 1 0 09:49? 00:00:00 memcached-p 11211-u root-m 16m-c 10240-d

Root 22931 22899 0 09:49 pts/1 00:00:00 grep memcached

[[email protected] ~]# printf "Get key\r\n" |nc 127.0.0.1 11211

END

memcached command syntax

Set Key 0 0 10

<command name> <key> <flags> <exptime> <bytes>\r\n

Command name is set get add replace

Get Data

Key is the value of the data that is required to be stored by the next client

Session shared storage in a cluster

http://oldboy.blog.51cto.com/2561410/1331316

Use memcached to store session features:

Advantages:

1) reading and writing speed will be much faster than the normal files.

2) can solve the problem of multiple servers sharing session.

Disadvantages:

1) session data are stored in memory, the persistence of the lack of, but the session data is not a problem.

2) Single point, deploy multiple units, also can not synchronize data. The problem of Sesson loss is still assigned by hash algorithm.

Large-scale enterprise solutions:

2) sessons can be stored with other persistent systems, for example: Redis,ttserver, instead of memcached.

3) high-performance concurrency scenarios, the efficiency of cookies is much better than the session, so the big website will use cookies to solve the problem of session sharing.

4) There are primary OPS users by sacrificing LB load balancing strategy implementation, such as: Lvs-p,nginx Ip_hash, and so on, these are not good methods.

How to solve the problem that the session does not share?

http://oldboy.blog.51cto.com/2561410/1331316

http://oldboy.blog.51cto.com/2561410/1323468

1, start a memcache instance on the server, take this instance as a shared memcached-p 11211-u root-m 16m-c 10240-d

The start of the real configuration

To modify the configuration file, set the global settings in php.ini:

################## #web集群session共享存储设置:

1, allow all Web servers to save the session in the same path cache (/tmp)

Can do shared NFS, the/TMP share, (A-B server), but NFS high-access, response is not timely, so all stored in the memcached

Ini

Session.save_path = "/tmp"

2, using memcached

Change the php.ini of all Web sites to the same memcached Ip+port

The type and configuration path of the session in the default php.ini:

#session. Save_handler = Files

#session. Save_path = "/tmp"

Modify the configuration as follows:

Session.save_handler = Memcache

Session.save_path = "tcp://10.0.0.18:11211"---

Tips:

1) 10.0.0.18:11211 is the IP and port of the memcached database cache.

2) The above is suitable for lnmp,lamp environment.

3) memcached server can also be multiple units via hash dispatch.

################# #针对memcached单点故障出现的问题解决方案: ( also not perfect, no fix master hangs, re-open Data unrecoverable problem)

1, high availability for memcached

Http://blog.snsgou.com/post-800.html

2, the use of Sina Memcacheddb, the original client is still memcached, but on the server he can persist storage.

3, Japanese-developed Tokyo Tyrant+tokyo Cabinet

################ #memcached的管理方式

1, via Telent + Port +ip

2, via NC---printf "Get key008\r\n" | NC 127.0.0.1 11211

############## #memcached monitoring and management tools

memadmin-1.0.12.tar.gz-----placed in the PHP Site Directory----(LAMPORONMP preferably source code compiled)

Tar xvf memamin

Www.etiantian.com/memcache

With the Memadmin tool, you can see the status of Memcached, the number of connections, the current number of concurrent

The analysis shows that the current memcached change in the change of the shot is more serious, the capacity is sufficient

This article is from the "Crazy_sir" blog, make sure to keep this source http://douya.blog.51cto.com/6173221/1677869

Memcached detailed introduction, installation, high availability, distributed cases, monitoring