Varnish Getting Started

This article describes the varnish workflow, installation, and configuration of the varnish three aspects of the content. First, we briefly introduce varnish and its workflow, understand its internal principle, then introduce the installation method of varnish, and finally introduce Varnish's configuration language vcl and how to debug VCL. Finally, a detailed annotated varnish configuration file code is attached.

This article will detail three aspects of varnish workflow, installation, and configuration. I believe that after reading this article, readers can master varnish skillfully.

Varnish Introduction

Varnish is a high-performance and open-source reverse proxy server and HTTP Accelerator, with a new software architecture, and now the hardware system closely, compared with the traditional squid, Varnish with higher performance, faster, more convenient management and many other advantages, Many large websites are beginning to try to replace squid with varnish, which promotes varnish's rapid development.

Norway's largest online newspaper, Verdens Gang (vg.no), used 3 Varnish instead of the original 12 Squid, which is a better performance than before, which is Varnish's most successful application case.

Workflow for Varnish file caching

Varnish is similar to the general server Software and is divided into master and child processes. The master process reads into the storage configuration file, invokes the appropriate storage type, then creates/reads the cache file of the appropriate size, then master initializes the structure that manages the storage space, then fork and monitor the child process. During the initialization of the main thread, the child process mmap the previously opened storage file into memory, creating and initializing the idle structure and suspending it to the storage management structure for allocation. The child process allocates several threads to work, mainly including some management threads and many worker threads.

Then, to start the real work, one of Varnish's responsible for receiving new HTTP connection threads begins to wait for the user, and if there is a new HTTP connection coming up, it is always responsible for receiving, then waking up a waiting thread and handing over the specific process to it. The Worker thread reads the URI of the HTTP request, finds an existing object, and returns and responds to the user directly if hit. If there is no hit, the requested content needs to be taken from the backend server, stored in the cache, and then replied to.

The process for allocating the cache is this: it creates a cache file of the appropriate size based on the size of the object that is being read. For ease of reading and writing, the program changes the size of each object to a multiple of the memory page that is closest to its size. Then find the free storage block of the most appropriate size from the existing free storage structure, and assign it to it. If the free block is not used up, the extra memory is made up of another free storage block, which is attached to the management structure. If the cache is full, the oldest object is freed according to the LRU mechanism.

The process of releasing the cache is this: there is a timeout thread that detects the lifetime of all objects in the cache, and if the pre-set TTL (time to Live) is not accessed, delete it and release the corresponding struct and storage memory. Note the free memory block before or after the block of storage memory is checked when released, and is merged into a larger chunk of memory if the preceding or subsequent free memory and the freed memory are contiguous.

The entire file cache management, regardless of the relationship between the file and memory, in fact, all of the object is considered in memory, if the system memory is not enough, the system will automatically swap it to swap space, without the need to varnish program to control.

Varnish Installation

Download Varnish installation Package

It is recommended to download the latest stable version (now the latest varnish version is 3.0.2), varnish to provide the source code installation package and executable program installation package, according to your custom download any installation package suitable for your platform.

Installing varnish

Source Code installation package installation

First install the Pcre library, the Pcre library is compatible with regular expressions, if not installed, you will be prompted not to find the Pcre library when installing the varnish2.0 version above. The following is the installation process for Pcre, as shown in Listing 1:

Listing 1. Pcre Library Installation Code

Tar zxvf pcre.tar.gz cd pcre/ ./configure--prefix=/usr/local/pcre/make && make install

Install varnish, whose code is shown in Listing 2:

Listing 2. Varnish Installation Code

Tar xzvf varnish-3.0.2.tar.gz CD varnish-3.0.2 export Pkg_config_path =/usr/local/pcre/lib/pkgconfig . Configure--prefix=/usr/local/varnish make do install

Executable Program installation package installation

RedHat installation varnish in a system environment, you need to install the following software: Automake, autoconf, Libtool, Ncurses-devel, Libxslt, Groff, Pcre-devel, Pkgconfig, Then install the varnish, and install the code as shown in Listing 3:

Listing 3. Varnish Installation Code

Rpm-i varnish-2.1.4-2.el5.x86_64.rpm

Start varnish

Listing 4. Varnish startup code

Varnishd-f/etc/varnish/default.vcl-s file,/var/varnish_cache,1g- t 127.0.0.1:2000-a 0.0.0.0:9082

The meanings of each parameter are as follows:

-f Specifies the configuration file location of the varnish

-s Specifies the way varnish caches are stored, commonly used in the following ways: "-S file,<dir_or_file>,<size>".

-T Address:port setting varnish telnet management address and its port

-A address:port indicates varnish's listening address to HTTP and its port

Varnish Configuration

Introduction to VCL

VCL (varnish language) is a varnish configuration language used to define varnish access policies. VCL syntax is relatively simple, similar to C and Perl. The main points are as follows:

• Blocks are delimited by curly braces, statements end with semicolons, and annotations can be added using the ' # ' notation.

• VCL uses the specified operator "=", the comparison operator "= =", the logical Operator "!,&&,!!" and other forms, it also supports regular expressions and "~" for ACL matching operations.

• VCL does not have a user-defined variable, you can set the value of the variable on backend, request, or object using the Set keyword. For example Set req.backend = Director_employeeui;

• Two strings of connections, there are no operators between them. The code is shown in Listing 5:

Listing 5. String Connection Code

Set req.http.x-hit = "hit" "It";

• \ "Character does not have a special meaning in the VCL, which is slightly different from other languages.

• VCL can use the SET keyword to set any HTTP headers, and you can use the Remove or unset keyword to remove the HTTP headers.

• VCL has a If/else judgment statement, but no loop statement.

VCL Backend

Declare and initialize a back-end object, as shown in Listing 6

Listing 6. Backend Declaration Code

Backend www { . host = "www.example.com"; . Port = "9082"; }

The use of back-end objects, as shown in Listing 7

Listing 7. Backend Code of Use

if (req.http.host ~ ^ (www.) example.com$ ") { set req.backend = www; }

Set director for VCL back end

VCL can synthesize a group of multiple backends, these groups are called Director, which can enhance the performance and elasticity, when one backend in the group hangs off, can choose another healthy backend. VCL has a variety of directors, different directors use different algorithms to choose backend, mainly have the following several:

• The Random Director

The Random director selects the Backend,.retries parameter according to the weight set (weight) to indicate the maximum number of times an attempt is made to find a backend. The weight parameter represents the weight value

• The Round-robin Director

Round-robin director selects backend in a circular fashion.

• The Client Director

Client director chooses backend according to Client.identity, and you can set the value of Client.identity to the session cookie to identify the backend.

Backend Probes

VCL can set probe to detect whether a backend is healthy, define a backend probes code as shown in Listing 8:

Listing 8. Defining backend Probes Code

Backend www { . host = "www.example.com"; . Port = "9082"; . Probe = { . url = "/test.jpg";//which URL requires varnish request . Timeout = 1 s;//wait for how long to time out . Interval = 5s//check interval . window = 5;//maintains the result of 5 sliding window . Threshold = 3;//at least three times the window was successful, declaring backend health } }

Acl

ACLS Create an Access control list for a client, and you can use ACLs to control which clients can access and which clients prohibit access. Define the ACL code as shown in Listing 9:

Listing 9. ACL Definition Code

ACL local{ "localhost"; 192.0.2.0 "/24; !" 192.0.2.23 ";//Remove the IP }

VCL built-in functions

Vcl_recv function

Used to receive and process requests. When a request arrives and is successfully received, it is called to determine how the request is processed by judging the requested data. For example, how to respond, how to respond, which backend server to use, and so on.

This function usually ends with the following keywords.

Pass: To enter the pass mode, give the request control to the Vcl_pass function.

Pipe: To enter the pipe mode, the request control to give the Vcl_pipe function.

Lookup: Indicates entering the lookup mode, handing over the request control to the lookup instruction, locating the requested object in the cache, and giving control to the function vcl_hit or function Vcl_miss based on the result of the lookup.

Error code [reason]: means to return "code" to the client and discard processing of the request. "Code" is an error ID, such as 200 and 405. "Reason" is an error message.

Vcl_pipe function

This function is called when it enters pipe mode, and is used to pass the request directly to the back-end host, returning the unchanging content to the client without changing the contents of the request and return until the connection is closed.

This function usually ends with the following keywords.

Error code [reason].

Pipe

Vcl_pass function

This function is called when it enters pass mode and is used to pass requests directly to the back-end host. The back-end host sends the reply data to the client after the data is answered, but does not make any caches and returns the latest content each time under the current connection.

This function usually ends with the following keywords.

Error code [reason].

Pass

Restart Restart the process, increase the number of boot times, and issue an error warning if the number of reboots is higher than max_restarts

Vcl_hash

This function is called when you want to add a data to the hash.

This function usually ends with the following keywords.

Hash.

Vcl_hit function

After the lookup instruction is executed, the function is called automatically when the requested content is found in the cache.

This function usually ends with the following keywords.

Deliver: Indicates that the found content is sent to the client and the control is given to the function Vcl_deliver.

Error code [reason].

Pass

Restart Restart the process, increase the number of boot times, and issue an error warning if the number of reboots is higher than max_restarts

Vcl_miss function

The method is called automatically when the requested content is not found in the cache after the lookup instruction is executed. This function can be used to determine whether the content needs to be fetched from the back-end server.

This function usually ends with the following keywords.

Fetch: Means getting the requested content from the backend and handing control over to the Vcl_fetch function.

Error code [reason].

Pass

Vcl_fetch function

The method is called after the backend host updates the cache and gets the content, and then determines whether the content is cached or returned directly to the client by judging what gets.

This function usually ends with the following keywords.

Error code [reason].

Pass

Deliver.

Esi.

Restart Restart the process, increase the number of boot times, and issue an error warning if the number of reboots is higher than max_restarts

Vcl_deliver function

This method is called before the requested content is sent to the client in the cache.

This function usually ends with the following keywords.

Error code [reason].

Deliver.

Restart Restart the process, increase the number of boot times, and issue an error warning if the number of reboots is higher than max_restarts

Vcl_error

This function is called when an error occurs.

This function usually ends with the following keywords.

Deliver.

Restart

VCL processing Process

The VCL process is shown in flowchart 1

Figure 1.VCL Processing Flow

The process of Varnish processing HTTP requests is broadly divided into the following steps.

• Receive status (VCL_RECV). That is, the request processing of the entry state, according to the VCL rules to determine whether the request should pass (Vcl_pass) or pipe (vcl_pipe), or into the lookup (local query).

• The Lookup state. After entering this state, the hash table will find the data, if found, enter the hit (vcl_hit) state, otherwise enter the Miss (Vcl_miss) state.

• Pass (vcl_pass) status. In this state, the back-end request is entered directly, which is the fetch (Vcl_fetch) state

• Fetch (Vcl_fetch) state. In the fetch state, the backend gets the request, sends the request, obtains the data, and stores it locally according to the settings.

• Deliver (Vcl_deliver) state. Send the obtained data to the client and complete the request.

VCL built-in public variables

The public variables built into the VCL can be used in different VCL functions, as described in the different stages of use.

When a request arrives, you can use the public variable as shown in table 1

table 1. Public variables available when request arrives

public variable name	meaning
Req.backend	Specify the corresponding back-end host
Server.ip	Represents the server IP
Client.ip	Represents the client IP
Req.quest	Just the type of request, such as, etc.
Req.url	Specify the requested address
Req.proto	Represents the HTTP protocol version of the client initiating the request
Req.http.header	Represents the HTTP header information in the corresponding request
Req.restarts	Indicates the number of restarts, the default maximum value is 4

Varnish in the back-end host request, but the public variables used are shown in table 2

table 2. Public variables available when back-end host requests

public variable name	meaning
Beresp.requset	Specify the type of request, such as, etc.
Beresp.url	Represents a request address
Beresp.proto	Represents the HTTP protocol version of the client initiating the request
Beresp.http.header	Represents the HTTP header information in the corresponding request
Beresp.ttl	Represents the lifetime of the cache, cache retention time (s)

After fetching content from the cache or back-end host, the common variables that can be used are shown in table 3

table 3. Public variables can be used when backend hosts get content

public variable name	meaning
Obj.status	Request status codes that return content, such as 200, 302, 504, and so on
Obj.cacheable	Whether the returned content can be cached
Obj.valid	is a valid HTTP request
Obj.response	Return request status information for content
Obj.proto	Returns the HTTP version of the content
Obj.ttl	Returns the lifetime of the content, that is, the cache time, in seconds
Obj.lastuse	Returns the last request to the current time interval, in seconds

The common variables that can be used when responding to a client are as shown in table 4

table 4. Use public variables when appropriate for clients

public variable name	meaning
Resp.status	HTTP code status returned to the client
Resp.proto	HTTP protocol version returned to the client
Resp.http.header	HTTP header message returned to the client
Resp.response	HTTP header status returned to the client

VCL debugging

VCL is a profile language, can not be used as a single-step debugging, such as C + +, when VCL operation and expected effect is not the same, it is difficult to find out which logic error, in addition to looking at the code lookup error, we can also use the built-in C language and the browser to look at the state of the returned object to find the logic

We can use the built-in C language to print the appropriate information, for example, we can print the information in the appropriate place to see if the VCL process is executed correctly. The built-in C language printing information code is shown in Listing 10:

Listing 10: Printing the information code

c{ #include <syslog.h>//first to include the header file }c c{ syslog (log_info, "VCL run here function xxx on line xxx" ); } C

After starting varnish, we can use the Tail-f/var/log/messages command to view the corresponding printing information in/var/log/messages. View the print information shown in 2:

Figure 2. Varnish printing Information

We can also set the value of certain variables to the object returned to the browser, and then view the value of the variable in the browser. Set the variable value code as shown in Listing 11:

List one by one . Varnish variable Setting code

Set Beresp.http.x-cookie-debug-req-cookie = Req.http.Cookie;

View the value of Beresp.http.x-cookie-debug-req-cookie in the Chrome browser, as shown in result 3:

Figure 3: View the value of the varnish variable in the browser

Summary

In this paper, varnish is briefly introduced, and its work flow, installation and configuration are explained in detail. Through this article, can quickly grasp the varnish.

Varnish Getting Started