Large website Optimization-memcache technology, Optimization-memcache Technology

Large website Optimization-memcache technology, Optimization-memcache Technology
Large website Optimization-memcache Technology

Memory + cache memory cache

Introduction to memcache

Memcache is a setDistributedHigh SpeedCache SystemDeveloped by Brad Fitzpatrick of LiveJournal, it is currently used by many websites to speed up Website access, especially for large websites that require frequent access to databases. This is an open-source software, which is released with BSD license authorization. [Retrieved from Baidu encyclopedia]

Official Website: http://memcached.org/

 

Distributed: Multiple Memcache servers are used to manage the data architecture.

Cache System: caches the data queried by the user into the memory, so that it can be obtained directly from the memory next time. Reduces disk IO overhead.

What is nosql? (SQL Relational Database Service)

A: MySQL is called a relational database. It is mainly characterized by a two-dimensional table structure (rows and columns in a table). The table and table are in a relationship. (oracle (Java), db2,Sqlserver)

Non-relational data: the database that does not use SQL statements as queries (the system that stores data), and does not have the concept of a two-dimensional table in a strict sense. Its data structure is a huge hash table (key-value)

Hash Table advantages: the time complexity is 0 (1): As data increases, the query time does not change by an order of magnitude (1 s.

Disadvantages of the hash table: the hash collision of different keys corresponds to the same value

 

Key1 = MD5 (SQL)	Asion
Key2	12
Key3	Shenzhen
Key4	Iphone

 

Selinux is not disabled

Make selinux effective immediately

 

2.1 prepare the environment

In Linux, tools such as gcc, g-c ++, make (makefile), cmake, autoconfig (configure), and libtool are required.

In LinuxNetworkingUse the following command

# Yum install-YGcc make cmake autoconfig libtool

-Y does not require confirmation Interaction

 

2.2 compile and install memcached

Memcached depends onLibeventLibrary, so you need to first install, respectively to their respective official website to download stable version

Libevetnt Official Website: http://libevent.org/

Memcache Official Website: http://memcached.org/

 

Compile and install libevent, compile and install memcached, and specify the installation path of libevent when installing memcached.

Procedure:

Upload

Download the libevent, decompress it, and install it (generally, the software installation in Linux is in the/usr/local/src directory, and the installed software is usually in/usr/local/NAME)

#./Configure -- prefix =/usr/local/libevent & make install

B. Install memcache, decompress, compile, and install memcache

#./Configure -- prefix =/usr/local/memcached -- with-libevent =/usr/local/libevent & make install

 

2.3 memcached startup

#/Usr/local/memcached/bin/memcached-m 64-p 11211-u nobody-vv

Note:: Memcached is successfully started, but the information is output to the console.

If you want to enable memcached as a service in the background, you only need to add the-d option (daemon background)

#/Usr/local/memcached/bin/memcached-m 64-p 11211-u nobody-d

 

How can I check whether the server is started normally?

# Ps axu | grep memcached

 

To view the parameter information, use memcached-h to view the help information:

Operate memcache

The communication between the memcached client and the server is very simple. The text-based protocol, similar to the http protocol, can be directly used for interaction through Telnet.

 

Use Telnet (quit)

# Telent Server IP 11211

Windows

Use ctrl +] to enable Telnet echo after connection

Linux

 

2. Basic commands:

Learn about the add, delete, modify, and query commands of memcache:

Add key flag expire length

Key: Name

Flag: 1 memcache is saved as a string

Expire: expiration time, memcache time

Length: Data length (B)

※Add

# Add name 1 0 2 # add a key to the memcache server with a name of 2 bytes in length and valid for a long time

 

How to Understand expire

Set the cache validity period in three formats

Set the number of seconds. It starts with the setting and expires after n seconds.

Timestamp, which expires after the specified Timestamp

It is set to 0 and does not expire automatically (note that it is not permanently valid. A. During memcache installation, a maximum validity period is specified. The default value is 30 days (Source Code). B. It may be squeezed out in less than 30 days)

Minimum recently used Principle

 

※Delete

# Delete key

 

※Replace

# Replace key flag expire length

 

※Get

# Get key

 

※If set contains replace, add

# Set key flag expire length

Name exists:

Age does not exist:

 

※Incr increase

# Incr age NUMBER

 

※Decr reduction

# Decr age NUMBER

 

※Stats collects memcache server information.

# Stats

 

※Flush_all clears all data.

# Flush_all

Use php to operate memcache

Windows

Download the extension dll file to (http://downloads.php.net/pierre/)

Put the file in the directory corresponding to extension_dir of php.

Modify the php. ini file, add extension = php_memcache.dll, and introduce the file

Restart Apache

 

Linux (how to enable an extension for php in Linux, let's talk about the general solution)

Beta: test version: Generally, there are some minor bugs, but you need to use them to find out. If there are any problems, you can give official feedback. Then repair

Alpha: Internal beta version: The version used during internal development. Generally, this version has many bugs. However, this version often has new features. (This is usually done by a new company) alpha has some compensation.

Stable: stable version: it has no bugs and can run stably.

 

Notes for using ftp:

Whether the service vsftpd status is enabled normally in vsftpd

Generally, only common users can connect to ftp (root cannot connect to ftp) to obtain sudo

The size and space of your home directory are insufficient (the file is successfully uploaded, but the displayed size is 0)

Whether the firewall is disabled (iptables-F is disabled) (iptables-L)

 

 

# Cls

# Cd

# Vim. bashrc

 

Linux download Mode

#: Exit wq in lowercase x

Linux in editing mode

# Z capital

 

 

#/Usr/local/php/bin/phpize absolute path phpize

D. Use the configure file generated above to collect system information without specifying the installation path

#./Configure -- with-php-config =/usr/local/php/bin/php-config tell it how to find the php configuration file

E. Execute compilation and Installation

# Make & make install

Note: You can view the structure after the preceding command is executed.

# Ls/usr/local/php/lib/php/extend/no-debug-non-zts-20090626/

 

Note: in Linux, how does one view the location of the php. ini configuration file?

Solution: phpinfo ();

 

Note: in Linux, you must back up the configuration file before modifying it.

Php. ini-backup-2016-1-12

 

Note: What is. so?

. So is a shared object in Linux. It is a. dll file in Windows.

 

 

 

Note: in Linux, you can disable a service.

# Pkill-9 httpd

# Ps aux | grep httpd

# Ps uax | grep httpd

 

Add the address generated above to the php. ini configuration file, as shown below:

Extension_dir =/usr/local/php/lib/php/extensions/no-debug-non-zts-20090626/

Extension = memcacahe. so

Create a test. php file to test whether php has a memcache module.

Operate memcache in php, save and get the value

Memcached memory management and deletion Mechanism

 

Note: The maximum value of Memcache can only be 1 MB.

 

Note: memory fragmentation always exists, but which method can minimize memory fragmentation.

 

1. What is memory fragmentation?

When using this memory cache system, due to constant application and release, some small memory fragments will be formed and cannot be used. This phenomenon is called memory fragmentation. This small red block is the space that the operating system cannot use.

 

How can this problem be solved?

Memcache uses slab allocator to manage (The size of each slab class is 1 MB.)

The smallest unit is chunk (small block): the data warehouse.

Multiple small units form a chunks: composed of multiple small blocks (all small blocks have the same size)

The size of each slab class is 1 MB.

 

How do I select a proper memcache size?

Note:If the slab of 122Bytes is full, there is a bytes data. Where can I store it?

A: Certainly there will be no 144, but there will still be a 122 chunk, which uses the LRU Algorithm for data storage.

 

Memory waste caused by fixed chunk, 22B

 

Factor Optimization

When memcached is started, slab class will be organized according to a certain size. You can use-f to specify

The default value is 1.25. The ratio between adjacent chunks is the increase factor. You can adjust the cache factor based on your website's business.

Because each business is different, the minimum chunk required is different. This parameter makes our system more adaptable to our own business, because data can be set by ourselves.

 

Memcache inert Deletion

Memcached does not monitor whether the record expires, but checks the timestamp of the record during get to check whether the record expires. This behavior is called lazy (inert) expiration. Therefore, memcached does not consume CPU time on expired monitoring.

For example, if a set (name, asion, 0, 3600) expires after 3600 seconds, it will not be automatically deleted after it expires. Only when the get query is performed, the system checks whether it expires, if it expires, it is deleted.

For example, # add name 1 8 2 after 8 s, is it invalid or does not exist? Stats analysis

Analysis: It indicates that you are invalid, not nonexistent. Only when you get it the next time will memcache check whether it expires and delete it when it expires.

 

Memcache LRU Algorithm

Memcached will give priority to the space of records that have timed out, but even so, there will be insufficient space to append new records. In this case, we need to use the Least Recently Used (LRU) mechanism to allocate space.

As the name suggests, this is a mechanism to delete records that are "least recently used. Therefore, when the memory space of memcached is insufficient (when the new space cannot be obtained from the slab class), The memcached will be searched from the recently unused record and empty.

To the new records. From the perspective of caching, this model is ideal.

 

When the data space in the mecache (64 MB by default) is full, can you continue to store the data?

A: It can be stored. To delete expired data, if none of them have expired, delete the most inactive data and free up space to add data later.

For example, if a bytes data record is generated when the data is full, how can this problem be solved? (Even if it is permanently valid, it will be kicked)

Analysis: Memory ManagementLRU Algorithm, FIFO Algorithm

 

Some memcache parameters Ctrl + C

Note: If you enter ctrl + s in vim, you can exit with ctrl + q.

-P listening port

-L connected IP address. The default value is local

-D start: start the memcached service.

-D restart: restart the memcached service.

-D stop | shutdown the running memcached Service

-D install the memcached Service

-D uninstall memcached Service

-U runs as the identity (only valid when running as root)

-MB maximum memory usage, in MB. The default value is 64 MB.

Note: If the system is 32-bit, the maximum limit is 2 GB. if the system is 64-bit, there is no limit.

-An error is returned when M memory is used up, instead of deleting items.

-C: Maximum number of simultaneous connections. The default value is 1024.

-F block size growth factor, default value: 1.25

-N: minimum allocation space. The default value of key + value + flags is 48.

-H Show Help

-V output warning and error message

-Vv: print the client's request and return information

-I print the copyright information of memcached and libevent.

 

Use Cases

Save: 1. File 2. mysql

Q: What if there are too many session files in a folder and retrieval slows down?

Answer: hierarchical Processing

Memcache is used for storage, and Memcache is used for storage in a distributed manner (multiple Memcache servers are used for storage)

 

Use memcache to save the session

 

1. Modify the php. ini file. The configuration information is as follows:

Session. save_handler = memcache # indicates that the session is saved using memcache.

Session. save_path = "tcp: // 127.0.0.1: 11211" # specify the address and port of the memcache Server

Test

1. Enable session and save session

Get session from Memcache

Advanced features distributed memcache Configuration

What is distributed?

Because the service capability of a single memcache is limited, multiple memcache instances can be used to provide the cache function. This architecture is called the distributed slow storage (cluster) system of memcache.

 

How to implement it?

You can understand how to distribute data to multiple Memcache servers.

 

Before the client implements distributed data storage, the data is saved to the memcache server based on certain algorithms, obtain data from the corresponding memcache server based on the same algorithm.

 

Distributed algorithms

Touch Algorithm

The key value is used to obtain the remainder of the number of servers, and then the corresponding value is saved to the memcache server of the corresponding remainder. Generally, this hash FunctionCrc32 (key)% 3

The crc32 () function converts a string to a 32-bit integer.

 

Disadvantage: When a server goes down or needs to be added to a server, the cache data basically becomes invalid because the divisor has changed. Not strict formula. hit rate = Retrieved Data/total number 1/N indicates the number of servers

 

The problem is that when memcache goes down, the cache data becomes invalid, and the pressure on MySQL increases sharply,

At this time, MySQL will go down, and then restart MySQL. MySQL will go down again in a short period of time, and then a little delay (a part of the cache has been re-established ), downtime. Over time, MySQL becomes stable and the cache system is successfully established. Because the cached data does not exist, all requests must be forwarded to MySQL. This phenomenon is called the avalanche of memcache.

 

Overview:

 

 

Consistent hash for distributed

 

Assume there is a ring with so many positive integers ranging from 0 to 2 ^ 32.

 

After the number of nodes on the server is computed using the hash function (crc32 (), the nodes are mapped clockwise to the ring above.

 

Save the key of the data to be stored to a node that is no smaller than itself in a clockwise direction based on the hash function above.

 

Note:If memcache goes down, data will be lost. However, if you want to minimize data loss and use consistent hash, even if one server goes down, it only affects data on one server.

 

Virtual node: share the task

Cache avalanche

What is the cause of avalanche?

Because of improper algorithms and touch algorithms, a large number of cache failures may lead to an avalanche.

Solution: consistent hash Algorithm

 

The cache time is the same time, and the cache system will all expire at the same time, which will also cause an avalanche

Solution: Set the cache time to a random time range (3-9 hours)

 

Because the cache data of a memcache node is invalid, the cache hit rate of other memcache nodes is reduced. The missing data in the cache is queried in the MySQL database. In a short period of time, this causes a huge pressure on the MySQL server and downtime, which is called a cache avalanche.

 

After MySQL is restarted, it will go down again in a short period of time, but some cached data has been created. After MySQL is started multiple times, the cache is completely rebuilt and MySQL will not go down until it becomes stable.

 

Solution: Set the cache time to a random value in a range (3-9 hours), so that it will expire in different time periods and the reconstruction work will be shared to different time periods.

How to Make memcache highly available

UseRepcachedImplementation: replication cached is a memcached High Availability technology invented by the Japanese, short for replication Buffer technology.

 

MemcacheDBIs a distributed, key-value formatPersistent Storage System.Developed by sina personnel. It is not a cache component, but a reliable and fast persistent storage Engine Based on Object Access. The Protocol is consistent with memcache (incomplete), so many memcached clients can connect to it. MemcacheDB uses Berkeley DB as a persistent storage component. Therefore, many Berkeley DB features are supported. Extension

In Linux, how does one add an extension to a php and perform common steps?

A:

 

How does memcache ensure security?

A:

Because the memcache design is extremely concise, there is no permission restriction. Why not set permissions? Only the cache function is provided.

On the Intranet 192.168.1.110, the Intranet IP address cannot be accessed from the Internet.

Write oneFirewall verification rulesOnly the packets with the specified IP address can be transferred in, and all others can be discarded.

192.168.1.221 --- 224

3. What should I do if there are too many files when I use files to save session files?

Generally, when there are more than 65535 session files, the session acquisition will become abnormally slow, which means php code execution is slow. How can this problem be solved?

A:

Tiering: Create a folder starting with a A-Z under a folder and then A_Z creates

Use memcache for processing: If the processing capability of a single memcache is limited, distributed memcache is used for processing.

How does one understand that a Server Load balancer cluster is also a highly available cluster?

A: The Server Load balancer cluster provides high availability. If a server is down, it can still provide normal services, but it is a little difficult to provide services.

Because the Server Load balancer cluster does not provideKeepalive. Monitoring Mechanism

 

Case:

How to Use Memcache in projects?

Memcache caches data to reduce the pressure on MySQL.

 

Write a list page

Details page

 

Extension

Install a common php Extension Method in Linux

A:

Download the corresponding extended source http://pecl.php.net/package/memcache

Upload the extension to the Linux server and place it under (/usr/local/src)

Run tar-zxvf NAME.tar.gz

Cd NAME

The phpize name (/usr/local/php/bin/phpize) under the absolute path must be in the directory of the extension package.

Run. /configure -- with-php-config =/usr/local/php/bin/php-config (-- with-php-config = the php-config path under the absolute path, you do not need to specify the installation path)

Make & make install

A directory is generated, which contains the NAME. so file.

Modify the configuration file of php. ini and add two lines of information.

Extension_dir = 'Directory generated above'

Extension = 'name. so'

Restart Apache, use the phpinfo () function to test the function, enter the address in the browser, and search for Memcache, which indicates that the extension is successfully enabled.