Varnish practice of high performance HTTP accelerator

One, varnish introduction and Characteristics

The varnish is a high-performance, lightweight, open-source Reverse proxy server and HTTP accelerator. Varnish code is small, advanced design concepts and mature design framework is its main characteristics. Currently in the choice of caching tools varnish because of its own technical superiority has surpassed the traditional cache tool squid, more and more favored. Here are some features of varnish:
1, based on memory cache, data disappears after reboot
2, using virtual memory mode, I/O performance is good
3, supports setting the exact cache time of 0-60 seconds
4. Flexible Configuration Management
5, with powerful management functions

two, installation of varnish

< Span style= "font-family: ' Microsoft Jas Black '; >varnish installation is simple and can be installed using non-root users.
1, install the Pcre library first, and the Pcre Library is a compatible regular expression.
     >tar zxvf pcre7.9.tar.gz 
    &NBSP;>CD pcre7.9/ 
     >./configure--prefix=/zhangzya/pcre/ 
     >make && make install
2,varnish installation
     >tar XZVF varnish-3.0.3.tar.gz 
    &NBSP;>CD varnish-3.0.3 
     >export pkg_config_path=/ zhangzya /varnish       
     >make 
     >make install
      to this installation complete

Three, varnish configuration

Introduction to common functions of 1,VCL configuration files
VCL, the Varnish configuation language, is used to define the varnish access policy. The following is an introduction to VCL commonly used built-in functions.
VCL_RECV function: Used to receive and process requests.
Vcl_pipe function: Called when entering pipe mode, the request is passed directly to the backend host.
Vcl_pass function: The user passes the request directly to the back-end host, and the data is not cached.
Lookup: Finds the Request object in the cache and assigns the Vcl_hit Vcl_miss function according to the lookup results.
Vcl_hit function: After the lookup is performed, the function is automatically called in the cache to find the requested content.
Vcl_fetch function: Called after the backend host updates the cache and gets the content, determines whether the content is cached or returned directly to the client by judging what gets.
Vcl_deliver function: Called before the request content that is found in the cache occurs to the client.

Introduction to 2,VCL processing process

650) this.width=650; "height=" 520 "src=" http://media.teamshub.com/10000/tm/2016/07/19/ 6d156b25-522e-4d2c-8308-407089fb55fb/849f9d6a-e42c-47da-acaa-007e5c7b434f.png "Width=" 701 "alt=" 849f9d6a-e42c-47da-acaa-007e5c7b434f.png "/>

The process of Varnish processing HTTP requests is broadly divided into the following steps:
Receive status (VCL_RECV): That is, the request processing entry state, according to the VCL rules to determine whether the request should pass (Vcl_pass) or pipe (vcl_pipe), or into the lookup (local query).
Lookup status: After entering this state, the hash table will find the data, if found, enter the hit (vcl_hit) state, otherwise enter the Miss (Vcl_miss) state.
Pass (Vcl_pass) Status: In this state, it will go directly to the backend request, that is, into the fetch (Vcl_fetch) state
Fetch (Vcl_fetch) state: In the fetch state, the request is fetched back-end, the request is sent, the data is obtained, and the local storage is based on the settings.
Deliver (Vcl_deliver) state. Send the obtained data to the client and complete the request.

3,VCL configuration File Detailed description
After the varnish installation is complete, the default configuration file is/ECHNWEB/VARNISH/ETC/VARNISH/DEFAULT.VCL. The file content default all comments, we can have the file as a template, create a new file Varnish.vcl, the main configuration is described:

#指定后端主机的IP及端口
Backend Default {
. Host = "192.168.232.139";
. Port = "9080";
}
#开始调用vcl_recv函数
Sub Vcl_recv {
... ...
#移除匹配规则的对应请求的Cookie信息
if (req.request = = "GET" && req.url ~ "\. ( jpg|png|gif|swf|jpeg|ico|css|js|html) $ ")
{
Unset Req.http.cookie;
}
#如果请求的类型不是GET或HEAD, enter the pass mode
if (req.request! = "GET" && req.request! = "HEAD") {
return (pass);
... ...
}
Sub Vcl_fetch {
#当请求类型是GET并且请求的URL以js, CSS, etc. at the end of the cache, cache time 7 days.
if (req.request = = "GET" && req.url ~ "\. ( Js|css|mp3|jpg|png|gif|swf|jpeg|ico) $ ")
{Set beresp.ttl = 7d;}
}

Varnish corresponds to multiple web hosting applications, with load sharing and health detection mechanisms. The specific configuration is as follows:
Backend web1{
. host= "192.168.232.11";
. port= "80";
}
Backend web2{
. host= "192.168.232.12";
. port= "80";
}
#定义一个名为webserver的director, that is, the request is randomly shared by Web1 and web2 two back-end servers: The weight is used to specify the weights of two back-end servers, with high-weighted processing requests.
Director Webserver random{
{. backend=web1;. weight=5;}
{. backend=web2;. weight=8;}
}

Four, varnish start-up

1,varnish Start command is/zhangzya/varnish/sbin/varnishd, with more parameters at startup, execute
/Zhangzya/varnish/sbin/varnishd-h To obtain detailed usage of each parameter.

Startup parameters	Meaning
-A Address:port	Indicates the varnish address and port of the httpd
-B Address:port	Represents the back-end server address and port
-D	means using debug debug mode
-F	Specify the configuration file for the varnish server
-P Param=value	Specify server parameters to optimize varnish performance
-P File	Varnish process PID File storage path
-N dir	Specify the working directory of the Varnish
-T Address:port	Setting the varnish Telnet management address and port
-W Int,int,int	Set the number of worker threads in varnish, commonly used in the following ways:-W min,max-w min,max,timeout
-S kind[,storageoptions]	Specifies how the content of the varnish cache is stored, usually in the following ways:-S file, Where you specify the storage path for the cache file, "" To specify the size of the cache file

2, after starting the effect as, it should be noted that the varnish runtime will simultaneously start two processes, a master process, a child process, if the child process problems, the main process will regenerate a child process.

650) this.width=650; "Src=" http://media.teamshub.com/10000/tm/2016/07/19/6d156b25-522e-4d2c-8308-407089fb55fb/ 44e7ff91-218d-4471-bb1b-e87a0c890a23.png "width=" "alt=" 44e7ff91-218d-4471-bb1b-e87a0c890a23.png "/>

3, if the varnish starts normally, 31000 ports and 31001 ports should be in the listening state. Can be viewed through netstat.

Five, varnish run log Introduction
Varnish provides the Varnishlog and VARNISHNCSA two tools for log processing. The former is easy to view varnish cache, receive, send, whether hit and other detailed processing process, easy to varnish performance analysis and fault query; the latter facilitates daily log analysis and data mining.

1,VARNISHNCSA logs can be logged to a file
/zhangzya/varnish/bin/varnishncsa-n/echnweb/varnish1/cache-w Varnish.log &

650) this.width=650; "Src=" http://media.teamshub.com/10000/tm/2016/07/19/6d156b25-522e-4d2c-8308-407089fb55fb/ 409d3336-a8ce-4fad-bcfb-062faaeaa059.png "width=" "alt=" 409d3336-a8ce-4fad-bcfb-062faaeaa059.png "/>

2,varnishlog logs such as
/zhangzya/varnish/bin/varnishlog-n/echnweb/varnish1/cache

Six, varnish cache effect display

You can view the effect of the varnish cache by accessing the corresponding Web page in your browser. The following command line is used to view the cache hits by viewing the header of the Web page.

After the first visit, the page header information is viewed, and the status in the Red box is miss, indicating that the access is not read from the cache.
650) this.width=650, height= "src=" http://media.teamshub.com/10000/tm/2016/07/19/ 6d156b25-522e-4d2c-8308-407089fb55fb/45ef036c-08b4-4540-b668-6ed02c546fea.png "width=" 717 "alt=" 45ef036c-08b4-4540-b668-6ed02c546fea.png "/>

The second access to the same URL address, the red box status of hit, indicating cache hits, read content from the cache.
650) this.width=650; "height=" 207 "src=" http://media.teamshub.com/10000/tm/2016/07/19/ 6d156b25-522e-4d2c-8308-407089fb55fb/288665c3-7dfb-4262-b205-c3bde3c24955.png "width=" 713 "alt=" 288665c3-7dfb-4262-b205-c3bde3c24955.png "/>

Seven, varnish cache refresh

One notable advantage of varnish is the flexibility to manage cached content. The specified cache content can be controlled and purged quickly and efficiently.

1,telnet Login Management Port Clear all caches, log on after execution Ban.url/* Returns 200 indicates cleanup success.
650) this.width=650; "height=" 189 "src=" http://media.teamshub.com/10000/tm/2016/07/19/ 6d156b25-522e-4d2c-8308-407089fb55fb/c7338b47-b0a6-473b-a2ea-cac0adadeae1.png "width=" 587 "alt=" C7338b47-b0a6-473b-a2ea-cac0adadeae1.png "/>
2, clear the cache through the Linux command, clean up the cache under download under all domain names

/zhangzya/varnish/bin/varnishadm-t 192.168.232.139:31001 ban.url/download/

3. Clean up the cache of all PNG files under the 10086.com domain name

/zhangzya/varnish/bin/varnishadm-t 192.168.232.139:31001 ban Req.http.host = = "10086.com" && Req.url ~ ". png$"

Eight, optimize varnish self-parameter

< Span style= "font-family: ' Microsoft Jas Black '; >      telnet to Varnish's management port, and then executes the param.show command to see all the parameters in the varnish run. Of course, you can change the relevant parameters in this way, the following 4 key parameters are described:
thread_ Pools 4[Pools]
thread_ Pool_ min 50[threads]
thread_ Pool_ Max 5120[Threads]
thread_ Pool_ timeout 10[seconds]

Thread_pool_min: Used to set the minimum number of thread per pool. Once the pool receives the available requests, it assigns the request to the idle thread for processing.

Thread_pool_max: Represents the maximum value of the sum of the thread numbers corresponding to all pool. This value cannot be too large, set to about 90% of the system's peak, and setting up the assembly causes the process to be suspended.

Thread_pool_timeout: Indicates the time-out expiration for thread. When the thread number is greater than the thread_pool_min setpoint, thread is freed if the thread is idle longer than the Thread_pool_timeout setting.

Nine. Optimizing Linux kernel Parameters
The kernel parameter is an interface between the user and the system kernel, through which the user can dynamically update the kernel configuration while the system is running, and can adjust the proc file system to optimize the performance of Linux.

The following parameters are an official configuration:
net.ipv4.ip_local_port_range=1024 65536
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
Net.ipv4.tcp_fin_timeout=30
net.core.netdev_max_backlog=30000
Net.ipv4.tcp_no_metrics_save=1
net.core.somaxconn=262144
Net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_orphans=262144
net.ipv4.tcp_max_syn_backlog=262144
net.ipv4.tcp_synack_retries=2
net.ipv4.tcp_syn_retries=2

Varnish practice of high performance HTTP accelerator