Varnish principles and configurations

1. Introduction to varnish

Varnish is a high-performance and open-source reverse proxy server and HTTP accelerator. It adopts a brand new software system and works closely with the current hardware system. Compared with the traditional squid, varnish has many advantages such as higher performance, faster speed, and more convenient management. Many large websites are trying to replace squid with varnish, which promotes the rapid development of varnish.

1. varnish Architecture

Varnish mainly runs two processes: the management process and the child process (also called the cache process ). Their working principles are roughly as follows:

650) This. width = 650; "Title =" 12 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 12 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663523abZN.png "width =" 889 "Height =" 536 "/>

Management process:

The management process mainly implements new application configuration, compiling VCL, monitoring varnish, initializing varnish, and providing a command line interface. The management process detects the child process every several seconds to determine whether it is running normally. If the child process does not receive a response within the specified duration, management restarts the child process.

Child process:

The child process contains multiple types of threads. Common examples include: acceptor thread: receives new connection requests and responds. Worker thread: The child process starts a worker thread for each session. Therefore, in high concurrency scenarios, hundreds or even more worker threads may occur. Expiry threads: Clear expired content from the cache;

Varnish relies on the workspace to reduce the possibility of contention when threads apply for or modify memory. There are multiple different workspaces in varnish, the most critical of which is the session workspace used to manage session data.

Logs:

To interact with other parts of the system, the child process uses the shared memory log that can be accessed through the file system interface. Therefore, if a thread needs to record information, it only needs to hold one lock, and then write data to a memory area in the shared memory, and then release the lock. To reduce competition, each worker thread uses the log data cache.

The size of shared memory logs is generally 90 MB, which is divided into two parts: the first part is the counter, and the second half is the data requested by the client. Varnish provides multiple tools, such as varnishlog, varnishncsa, or varnishstat, to analyze information in shared memory logs and display it in a specified manner.

Ii. varnish back-end Storage

Varnish supports multiple types of backend storage, which can be specified by the-s option when varnishd is started. Backend storage types include:

(1) file: stores all cached data using a specific file, and maps the entire cache file to the memory area (if conditions permit) through the MMAP () System Call of the operating system );

(2) malloc: Use the malloc () library to call the request to the operating system for a specified size of memory space at varnish startup to store cache objects, similar to the dynamic application function of malloc in C language;

The file and malloc storage methods are similar. When the cache service is restarted, the previously cached data no longer exists.

Iii. varnish state engine)

Speaking of the varnish status engine, you have to say VCL (varnish configuration language: varnish tool for configuring cache policies ). It is a simple domain-based programming language that supports arithmetic operations, regular expressions, and if statements. Cache policies written in VCL are usually stored in. VCL files. varnish can call these caching policies only after they are compiled into binary format.

VCL is used for the Administrator to define a cache policy, and the defined policy will be analyzed by the varnish management process, converted to C code, compiled into a binary program, and connected to the child process. Varnish has several so-called states, which can be attached with a policy defined by VCL to implement the corresponding cache processing mechanism, therefore, VCL is often referred to as a "domain-specific" language or status engine. "domain-specific" means that some data only appears in a specific status.

The specific status is implemented by defining built-in functions. The specific process is as follows:

650) This. width = 650; "Title =" 11 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 11 "src =" http://img1.51cto.com/attachment/201409/25/8886857_14116635273XRo.png "width =" 901 "Height =" 835 "/>

Vcl_recv: vcl_recv is the first sub-routine to be executed after Varnish completes decoding request packets into a basic data structure. vcl_fetch: cache decisions are made based on server responses. vcl_pipe: used to send requests directly to backend hosts; vcl_hash: the data source for custom hash generation; vcl_pass: Used to directly transmit requests to backend hosts; vcl_hit: the operations to be performed when the cached object is found in the cache; vcl_miss: the operations to be performed when the cached object is found in the cache; vcl_deliver: the method used to respond the content of the user request to the client; vcl_error: When the varnish side synthesize the error response;

Iv. http protocol and varnish

Cache-related HTTP header:

HTTP provides multiple headers for page caching and cache invalidation. The most common ones are: (1) expires: used to specify the expiration date/time of a web object, it is usually in GMT format. Generally, the future of this setting should not be too long. The length of one year is sufficient for most scenarios; it is often used to specify cache cycles for pure static content such as javascripts style sheets or images. (2) cache-control: it is used to define cache instructions that all cache mechanisms must follow, these indicators are specific commands, including public, private, and no-Cache (indicating that they can be stored, but they cannot be used to respond to client requests until they are verified again), No-store, Max-age, S-maxage, and must-revalidate. the time set in cache-control overwrites the time specified in expires. (3) etag: response Header, used to define a version identifier for a Web Resource in the Response Message; (4) Last-mofified: Response Header, it is used to respond to the client's request for the last-modified-since or if-None-match header to notify the client of the most recent modification time of the requested web object. (5) if-modified-since: Conditional request header. If the requested web content changes after the time specified in this header, the server responds to the changed content. Otherwise, the response is 304 (not modified); (6) if-None-Match: Conditional request header; the Web server defines the etag header for a Web content, the client can obtain and save the value of this header (TAG) upon request ); then, in subsequent requests, the IF-None-match header will be used to append an authorized tag list and the server will check whether the original content has tags that can match a tag in the list; if yes, 304 is returned. Otherwise, the original content is returned. (7) vary: Response Header. The original server may have different headers according to different request sources, the most common method is vary: Accept-encoding, which is used to notify the cache mechanism that the content may look different from the encoding format of the accept-encoding-header ID when the user requests; (8) Age: the cache server can send an additional response header to specify the validity period of the response. The browser usually determines the content cache duration based on the header; if the max-age command is used in the response packet header, the cache validity period is "Max-age minus age;

V. install and configure Varnish

# Rpm package: https://repo.varnish-cache.org/rpm-IVH varnish-3.0.5-1.el6.x86_64.rpm varnish-libs-3.0.5-1.el6.x86_64.rpm varnish-docs-3.0.5-1.el6.x86_64.rpm

650) This. width = 650; "Title =" 31 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 31 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663530ZDVj.png "width =" 542 "Height =" 519 "/>

Modify varnish parameters:

# Valid parameters for/etc/sysconfig/varnish are as follows: nfiles = 131072 memlock = 82000 nprocs = "unlimited" reload_vcl = 1varnish_vcl_conf =/etc/Varnish/default. listener = 80 # modify the listener port listener of varnish = listener _ admin_listen_port = listener =/etc/Varnish/listener = 1000varnish_thread_timeout = 120varnish_storage_file =/var/lib/Varnish/listener = 1gvarnish_storage_mem_size = 128 M # memory size varnish_storage = "malloc, $ {region} "# configure backend storage as mallocvarnish_ttl = 120daemon_opts ="-A $ {varnish_listen_address }:: {varnish_listen_port}-F $ {varnish_vcl_conf}-T $ {region }: $ {varnish_admin_listen_port}-T $ {varnish_ttl}-W $ {varnish_min_threads}, $ {varnish_max_threads }, $ {varnish_thread_timeout}-u Varnish-G Varnish-S $ {varnish_secret_file}-S $ {varnish_storage }"

Configure VCL parameters:

# Modify the following content in/etc/Varnish/Default. VCL: backend default {. Host = "172.16.10.11";. Port = "80 ";}

Result Display:

650) This. width = 650; "Title =" 32 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 32 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663531vTXm.png "width =" 1042 "Height =" 101 "/>

At this time, when we access: http: // 172.16.10.77, we will find that the first response speed is slow and the subsequent response is fast.

In this case, you can use varnishstat to View cache hits.

650) This. width = 650; "Title =" 21 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 21 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663533l29M.png "width =" 844 "Height =" 486 "/>

You can also customize the cache policy:

Varnish built-in variable: the built-in variable available when the request arrives: req. URL req. request req. HTTP. header req. restarts: Number of times the request is restarted; server. IP server. port server. hostname client. IP req. the built-in variable bereq that can be used when backend requests to the backend host. URL bereq. request bereq. HTTP. header bereq. connect_timeout bereq. the built-in variable beresp available when proto obtains the response object from the backend host. status beresp. response beresp. HTTP. header beresp. TTL beresp. backend. name beresp. backend. IP beresp. backend. port built-in variables available when the cached object enters the cache (only available for vcl_hit or vcl_error, and mostly read-only) obj. status obj. response obj. TTL obj. hits obj. HTTP. the built-in variable resp available when the header responds to the client. PROTO resp. status resp. response resp. HTTP. header

Custom web. VCL with the following content:

[[Email protected] ~] # Cat/etc/Varnish/web. VCL # defines the backend host and provides health check backend web1 {. host = "172.16.10.11 ";. probe = {. url = "/index.html ";. interval = 2 S ;. window = 8 ;. threshold = 2 ;}} backend web2 {. host = "172.16.10.16 ";. probe = {. url = "/index.html ";. interval = 2 S ;. window = 8 ;. threshold = 2 ;}} Director websrv round-robin {{. backend = web1 ;}{. backend = web2 ;}# define an ACL for cache cropping ACL purgers {"127.0.0.1"; "172.16.0.0"/16;} sub VCL _ Recv {set req. backend = websrv; If (req. restarts = 0) {If (req. HTTP. x-forwarded-For) {set req. HTTP. x-forwarded-for = req. HTTP. x-forwarded-for + "," + client. IP;} else {set req. HTTP. x-forwarded-for = client. IP ;}# define if (req. URL ~ "^/Test.html $") {return (PASS);} If (req. Request = "purge") {If (! Client. ip ~ Purgers) {error 405 "method not allowed";} return (lookup);} If (req. Request! = "Get" & Req. Request! = "Head" & Req. Request! = "Put" & Req. Request! = "Post" & Req. Request! = "Trace" & Req. Request! = "Options" & Req. Request! = "Delete") {/* Non-RFC2616 or connect Which is weird. */Return (PIPE);} If (req. Request! = "Get" & Req. Request! = "Head") {/* We only deal with get and head by default */Return (PASS);} If (req. HTTP. authorization | req. HTTP. cookie) {/* Not cacheable by default */Return (PASS);} return (lookup);} sub vcl_pipe {return (PIPE );} # sub vcl_hash {# hash_data (req. URL); # If (req. HTTP. host) {# hash_data (req. HTTP. host); #} else {# hash_data (server. IP) ;#}# return (hash) ;#} sub vcl_hit {If (req. request = "purge ") {Purge; error 200" purged ";} return (deliver);} sub vcl_miss {If (req. request = "purge") {purge; Error 404 "not in cache";} return (FETCH);} sub vcl_pass {If (req. request = "purge") {error 502 "purge on a passed object";} return (PASS) ;## sub vcl_fetch {# if (beresp. TTL <= 0 S | # beresp. HTTP. set-Cookie | # beresp. HTTP. vary = "*") {#/* # * mark as "Hit-for-Pass" for the next 2 minut Es # */# Set beresp. TTL = 120 s; # Return (hit_for_pass) ;#}# return (deliver) ;#} sub vcl_deliver {set resp. HTTP. x-age = resp. HTTP. age; unset resp. HTTP. age; If (obj. hits> 0) {set resp. HTTP. x-Cache = "hit via" + "" + server. hostname;} else {set resp. HTTP. x-Cache = "Miss via" + "" + server. hostname;} return (deliver) ;}# sub vcl_deliver {## return (deliver) ;### sub vcl_error {# set obj. HTTP. Content-Type = "text/html; charset = UTF-8"; # Set obj. http. Retry-after = "5"; # synthetic {"# <? XML version = "1.0" encoding = "UTF-8"?> # <! Doctype HTML public "-// W3C // dtd xhtml 1.0 strict // en" # "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd" >#< HTML >#< head >#< title> "} + obj. status + "" + obj. response + {"</title> # 

You can use the varnishadm command to manually compile the VCL file:
650) This. width = 650; "Title =" 22 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 22 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663535envx.png "width =" 798 "Height =" 606 "/>
650) This. width = 650; "Title =" 23 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 23 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663536J99Y.png "width =" 720 "Height =" 359 "/>
650) This. width = 650; "Title =" 25 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 25 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663538VPh5.png "width =" 663 "Height =" 302 "/>
650) This. width = 650; "Title =" 26 "style =" border-right-width: 0px; border-bottom-width: 0px; border-top-width: 0px; "Border =" 0 "alt =" 26 "src =" http://img1.51cto.com/attachment/201409/25/8886857_1411663541OwTk.png "width =" 696 "Height =" 627 "/>
 
Supplement:
In the VCL automatic compilation method, edit the following in the/etc/sysconfig/varnish file:
Varnish_vcl_conf =/etc/Varnish/Web. VCL
This article from the "cold" blog, please be sure to keep this source http://guoting.blog.51cto.com/8886857/1558305
Varnish principles and configurations