1. Basic Introduction to varnish
Varnish's author, Poul-Henning Kamp, is one of FreeBSD's kernel developers. He believes that today's computers are much more complicated than in 1975. In 1975, there were only two storage media types: memory and hard disk. However, in addition to primary memory, the memory of the computer system also includes L1, L2 in the cpu, and even L3 cache. The squid cache has its own caching device on the hard disk, so it is impossible for the architecture of squid cache to handle the replacement of objects on its own to know these conditions, but the operating system can know these situations, therefore, this part of work should be handed over to the operating system, which is the Varnish cache design architecture.
Similar to general server software, Varnish is a web cache proxy server, which can be divided into master (management) processes and child (worker, mainly used for cache work) processes. The master process reads commands, performs initialization, fork, and monitors the child process. The child process allocates several threads for work, including some management threads and many woker threads.
The Management process mainly implements new application configuration, compiling VCL, monitoring varnish, initializing varnish, and providing a command line interface. The Management process detects the Child process every several seconds to determine whether it is running normally. If the Child process does not receive a response within the specified duration, Management restarts the Child process.
The Child process contains multiple types of threads, such:
Acceptor thread: receives and responds to new connection requests;
Worker thread: The child process starts a worker thread for each session. Therefore, hundreds or more worker threads may occur in high concurrency scenarios;
Expiry thread: clears expired content from the cache;
Varnish relies on the workspace to reduce the possibility of contention when threads apply for or modify memory. There are multiple different workspaces in varnish, the most critical of which is the session workspace used to manage session data.
Working Process principle and process:
2. Differences between varnish and squid
Varnish and squid are lightweight and easy to use in small and medium-sized applications, but for massive concurrent requests, A single varnish can carry about 5000 concurrent access requests, and more than 5000 requests may be unstable. squid can show good performance here, therefore, most enterprise-level applications are squid-based, while most varnish-based reverse proxy caches of small and medium-sized enterprises are used;
3. varnish log description
To interact with other parts of the system, the Child process uses the shared memory log that can be accessed through the file system interface. Therefore, if a thread needs to record information, it only needs to hold one lock, and then write data to a memory area in the shared memory, and then release the lock. To reduce competition, each worker thread uses the log data cache.
The size of shared memory logs is generally 90 MB, which is divided into two parts: the first part is the counter, and the second half is the data requested by the client. Varnish provides multiple tools, such as varnishlog, varnishncsa, or varnishstat, to analyze information in shared memory logs and display it in a specified manner.
4. Introduction to VCL
Varnish Configuration Language (VCL) is a tool for varnish to configure cache policies. It is a simple programming Language based on domain specific, it supports limited arithmetic and logical operations, regular expressions for string matching, set custom variables, and if statements, there are also built-in functions and variables. Cache policies written using VCL are usually stored in. vcl files. varnish can call these caching policies only after they are compiled into binary format. In fact, the entire cache policy is composed of several specific child routines, such as vcl_recv and vcl_fetch, which are executed at different locations (or time) respectively. If a child routine is not pre-defined for a specific location, varnish executes the default definition.
Before the VCL policy is enabled, the management process converts it to C code, and then the gcc compiler compiles the C code into a binary program. After compilation, management is responsible for connecting it to the varnish instance, that is, the child process. It is precisely because the compilation is completed outside the child process that it avoids the risk of loading the wrong format VCL. Therefore, the overhead of varnish Configuration modification is very small. It can retain several old versions that are still referenced at the same time, and make the new configuration take effect immediately. The old version configuration after compilation is usually discarded only when varnish is restarted. If you need to manually clear the configuration, you can use the vcl. discard command of varnishadm to complete the configuration.
5. varnish back-end Storage
Varnish cache objects are cleared and re-created every time the service is restarted, so these servers should not be restarted at will. varnish wants to store data more persistently, more storage mechanisms are introduced, so varnish supports a variety of different backend storage;
Varnish supports multiple types of backend storage, which can be specified by the-s option when varnishd is started. Backend storage types include:
(1) file: stores all cached data using a specific file, and maps the entire cache file to the memory area (if conditions permit) through the mmap () System Call of the operating system );
(2) malloc: Use the malloc () library to call the request to apply for a specified size of memory from the operating system at varnish startup to store cache objects;
(3) persistent (experimental): it has the same function as file, but can store data permanently (that is, it will not be cleared when varnish data is restarted); it is still in the testing phase;
Varnish cannot track whether a cached object has been stored in a cached file, so that it cannot know whether the cached file on the disk is available. Therefore, the file storage method clears data when varnish is stopped or restarted. The emergence of the persistent method makes up for this, but the persistent is still in the testing stage. For example, it is still unable to effectively process the situation where the overall size of the cached object exceeds the cache space. Therefore, it is only applicable to scenarios with large cache space.
Choosing the appropriate storage method helps improve the system. From an empirical perspective, we recommend that you use the malloc method when the memory space is sufficient to store all cached objects. Otherwise, file Storage delivers better performance. However, it should be noted that varnishd actually uses a larger space than the cache space specified by the-s option. Generally, varnishd needs to use about 1 k of storage space for each cached object, this means that for 1 million cached objects, the cache space used exceeds the specified size by about 1 GB. In addition, varnish itself occupies a large amount of memory space to save data structures.
6. Working Principle and workflow of varnish
Official workflow:
Vcl is implemented based on the state engine. Note:
If the results of vcl_recv can be queried and identified, it is necessary to go to the vcl_hash step. If it cannot be identified, it is sent to vcl_pipe through the pipe (pipeline). If it can be identified, if it is not a cacheable object, it is sent to vcl_pass through pass. After vcl_hash, you can check whether the object in the cache has hit (vcl_hit ), if there is no hit, it indicates that it is not hit (vcl_miss). If it is hit, it can be directly sent to vcl_deliver for response through deliver. If it is not hit, it is sent to vcl_fatch through fetch to fetch data from the backend server, after the data is retrieved, if the data can be cached, it will be cached. After the local cache is complete, a response will be built. If the data cannot be cached, it will not be cached and sent to vcl_deliver for response. If the data hit is handed to vcl_pass, after the pass is passed, it is necessary to go to the Fetch objet from backend server to retrieve data, because the hit object may expire or be required. Independent and additional processing; this is the status engine process of vcl.
I. installation and implementation process:
# Install varnish with version 3.0.4-1. el6
[Root @ node0 ~] # Rpm-ivh varnish-3.0.4-1.el6.x86_64.rpm varnish-docs-3.0.4-1.el6.x86_64.rpm varnish-libs-3.0.4-1.el6.x86_64.rpm
[Root @ node0 ~] # Rpm-ql varnish # view the Installation File of varnish[Root @ node0 varnish] # cp default. vcl default. vcl. bak
[Root @ node0 varnish] # mv default. vcl test. vcl
[Root @ node0 varnish] # vim test. vcl}
# Start the service
[Root @ node0 sysconfig] # service varnish start
Starting varnish HTTP accelerator: [OK]
[Root @ node0 sysconfig] #
Install and configure the backend web server and start it. The address is 172.16.27.1, and then access the backend server through the varnish server address. Here, we only define a point to the backend server, that is, we can only work now, however, no related cache attributes have been defined, so visit the varnish server first:
[Root @ node1 ~] # Yum-y install httpd php-mysql
[Root @ node1 ~] # Cd/var/www/html
[Root @ node1 html] # vim index.html
<H1> www.tanxw.com and varnish fo backend <H2> node1.tanxw.com
2. Set whether the response is hit, and then write the configuration file:
[Root @ node0 varnish] # vimtest. vcl
Sub vcl_deliver {# define a subroutine
If (obj. hits> 0) {# set X-Cache to HIT in the http Response Header if it is HITSet resp. http. X-Cache = "HIT from" server. ip;
} Else {# Otherwise, set X-Cache to MISS in the http response header.
Set resp. http. X-Cache = "MISS ";}
# Recompile and reload the configuration file in the varnish command line
Varnish> vcl. load test1/etc/varnish/test. vcl
200 13
VCL compiled.
Vcl. use test1
200 0
Varnish> vcl. show test1
200 191
Backend webserver {
. Host = "172.16.27.1 ";
. Port = "80 ";
}
Sub vcl_deliver {
If (obj. hits> 0 ){
Set resp. http. X-Cache = "HIT ";
} Else {
Set resp. http. X-Cache = "MISS ";
}
}
[Root @ node0 varnish] # curl-Ihttp: // 172.16.27.88/index.html # You can also request
Then visit the page to see if it has taken effect:
3. Some files cannot be cached, and configuration files are added intermittently.
[Root @ node0 varnish] # vim test. vcl # Add the following code:
Sub vcl_recv {# indicates that if test.html is matched in the request file, the request is passed and the request is not cached.
If (req. url ~ "Test.html "){}
# Re-load the configuration file and apply it to the command line of varnish.
Varnish> vcl. load test4/etc/varnish/test. vcl
200 13
VCL compiled.
Vcl. use test4
200 0
# Check the Cache in the command line request. No matter how you request X-Cache, it is missing.
[Root @ node0 varnish] # curl-I http: // 172.16.27.88/test.html
HTTP/1.1 200 OK
Server: Apache/2.2.15 (CentOS)
Last-Modified: Sat, 17 May 2014 09:51:07 GMT
ETag: "120905-1a-4f99578180449"
Accept-Ranges: bytes
Content-Length: 26
Content-Type: text/html; charset = UTF-8
Date: Sat, 17 May 2014 10:01:27 GMT
X-Varnish: 1309371381
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: MISS
Then, request the test.html page;
4. Set cache duration and define image anti-leech protection:
[Root @ node0 varnish] # vim default. vcl
Sub vcl_fetch {
If (req. url ~ "\. (Jpg | jpeg | gif | png) $") {# If the url is cached for 2 hours after the image format ends}
Sub vcl_recv {
If (req. url ~ "Test.html "){
Return (pass );
}
# Image anti-leech
If (req. http. referer ~ "Http ://.*"){
If (! (Req. http. referer ~ "Http: //. * tanxw \. com"
| Req. http. referer ~ "Http: //. * google \. com"
| Req. http. referer ~ "Http: //. * yahoo \. com"
| Req. http. referer ~ "Http: //. * google \. cn"
| Req. http. referer ~ "Http: //. * baidu \. com"
)){
Set req. http. host = "www.tanxw.com ";
Set req. url = "/templets/default/images/logl.gif ";
}
Return (lookup );
}
Return (lookup );
}
5. Remove a single cache object: purge is used to clear a specific object in the cache and its variants (variants). Therefore, this method can be used when there are cache objects explicitly to be trimmed. The http purge method can implement the purge function. However, it can only be used in vcl_hit and vcl_miss. It will release the memory and remove all Vary:-variants of the specified cache object, wait for the next client request to refresh the content when it arrives. In addition, it is generally used with return (restart.
[Root @ node0 varnish] # vim default. vcl
Acl purgers {# define acl Access Control. Only the following network segments or hosts are allowed to perform the purgers operation
"127.0.0.1 ";}
# Save and exit, and then test it in the command line:
[Root @ node0 varnish] # curl-I http: // 172.16.27.88/index.html
HTTP/1.1 200 OK
Server: Apache/2.2.15 (CentOS)
Last-Modified: Sat, 17 May 2014 08:07:17 GMT
ETag: "120904-47-4f99404c6fde2"
Content-Type: text/html; charset = UTF-8
Content-Length: 71
Accept-Ranges: bytes
Date: Sat, 17 May 2014 13:38:51 GMT
X-Varnish: 364681188 364681185
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT from 172.16.27.88 # Cache HIT
[Root @ node0 varnish] # curl-x purge http: // 172.16.27.88/index.html
<? Xml version = "1.0" encoding = "UTF-8"?>
<! DOCTYPE html PUBLIC "-// W3C // dtd xhtml 1.0 Strict // EN"
Http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd>
<Html>
<Head>
<Title> 200 Purged OK. </title> # The cache is cleared successfully.
</Head>
<Body>
<H1> Error 200 Purged OK. <P> Purged OK. </p>
<H3> Guru Meditation: <P> XID: 364681189 </p>
<Hr>
<P> Varnish cache server </p>
</Body>
</Html>
[Root @ node0 varnish] # curl-I http: // 172.16.27.88/index.html
HTTP/1.1 200 OK
Server: Apache/2.2.15 (CentOS)
Last-Modified: Sat, 17 May 2014 08:07:17 GMT
ETag: "120904-47-4f99404c6fde2"
Content-Type: text/html; charset = UTF-8
Content-Length: 71
Accept-Ranges: bytes
Date: Sat, 17 May 2014 13:42:39 GMT
X-Varnish: 364681194
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: MISS # Cache MISS after clearing
Vi. Varnish checks the health status of the backend Host:
Varnish can detect the health status of the backend host and automatically remove it from the list of available backend hosts when determining that the backend host is invalid, once it becomes available again, it can be automatically set to available. To avoid misjudgment, Varnish detects that the health status of the backend host changes (for example, a backend host suddenly becomes unavailable during a certain detection ), it usually takes several consecutive times to perform a probe to be in a new State to mark it as a converted State.
The health status detection method of each backend server is passed. probe. backend. the healthy variable can also be obtained through the Backend_health check in varnishlog or the debug of varnishadm. health Check.
Common probe commands in. probe include:
(1). url: the URL requested when detecting the health status of the backend host. The default value is "/".
(2). request: the detailed format of the request content when detecting the health status of the backend host. After definition, the request replaces the probe method specified by. url. For example:
. Request =
"GET/.healthtest.html HTTP/1.1"
"Host: www.magedu.com"
"Connection: close ";
(3). window: set the number of recent probes when determining the health status of the backend host. The default value is 8;
(4). threshold: In the number of times specified in. window, at least how many times are successful before determining that the backend host is healthy; The default value is 3;
(5). initial: the minimum number of successful detection times required for the backend host when Varnish is started. The default value is the same as. threshold;
(6). expected_response: the status code of the expected response from the backend host. The default value is 200;
(7). interval: The sending cycle of the probe request. The default value is 5 seconds;
(8). timeout: The expiration time of each probe request. The default value is 2 seconds;
. Url = "/.healthtest.html"; # obtain this page from the backend host
. Interval = 1 s; # Try once every 1 second
. Window = 5; # Try up to 5 times to determine the sample
. Threshold = 2; # If sampling is performed five times, if two times are incorrect, it is regarded as a failure, and the same is true when going online.
}VII. Varnish command line tool
Varnishadm command syntax: varnishadm [-t timeout] [-S secret_file] [-T address: port] [-n name] [command [...]
A tool that connects to varnishd through a command line for management. There are two methods to specify the varnish instance to be connected:
-N name -- connect to an instance named "name;
-T address: port -- connect to the instance on the specified socket;
There are two running modes. When "command" to be executed is not provided in the command line, it enters interactive mode; otherwise, varnishadm executes the specified "command" and exits. To view the locally enabled cache, run the following command.
# Varnishadm-S/etc/varnish/secret-T 127.0.0.1: 6082 storage. list
Summary:
Varnish is a powerful cache server that can also be used for static/dynamic separation. It is limited to space and will not be mentioned here. For more information, see the parameter official documentation, the scheduling of backend servers can also be done, so the summary here is too hasty. If there is something you can't do, I hope you can tell me more about it. If you have any questions, you can leave a message for us to learn.
This article from the "warm boiled frog" blog, please be sure to keep this source http://tanxw.blog.51cto.com/4309543/1412984
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
