Linux Cache Service Varnish

Varnish is a high-performance open-source HTTP accelerator, Squid Service is also, the relationship between the two like Apache and nginx,varnish younger and lighter, squid more mature and stable.
Varnish4.0 work processes such as

Description of each point
VCL_RECV: Receiving Requests
Vcl_pass, skipping cache
Vcl_hit, Cache Hit
Vcl_miss, Cache is missing
Vcl_pipe, non-HTTP
Vcl_purge, manually clear cache as expired
Vcl_synth, manual clear cache expiration sync
Vcl_deliver Send to Client
Variable scopes on each node, such as

Built-in variables:

1, Req.*:request, indicating that the request sent by the client is related to the message;
Req.http.user-agent, Req.http.Referer, ...
2, bereq.*: From varnish to be host of httpd request related;
3. beresp.*: The response message from be host to varnish is related;
4, resp.*: From the varnish response to the client-related;
5. Obj.*: The properties of cached objects stored in cache space;

Common variables:

Bereq.http.HEADERS
Bereq.request: Request method;
Bereq.url: URL of the request;
Bereq.proto: the requested protocol version;
Bereq.backend: Indicates the back-end host to invoke;
Req.http.Cookie: The value of the Cookie header in the client's request message;
Req.http.user-agent ~ "Chrome"
//-------------------------------------
Beresp.http.HEADERS
Beresp.status: The status code of the response;
Reresp.proto: protocol version;
Host name of the Beresp.backend.name:BE host;
The Beresp.ttl:BE host responds to the contents of the remaining cacheable duration;
//-------------------------------------
Obj.hits: The number of times this object was hit from the cache;
Obj.ttl: The TTL value of the object
//-------------------------------------
Server.ip
Server.hostname
//-------------------------------------
Client.ip

Varnish's expertise is reflected in cache hit Vcl_hit, and cache cleanup Vcl_purge convenient.

Simple instructions for use

1. Installation

[[email protected] ~]# yum install varnish

2. View

[[email protected] ~]# rpm -ql varnish/etc/varnish   #配置文件目录/etc/varnish/default.vcl      #配置各Child/Cache线程的缓存策略；/etc/varnish/varnish.params   # 配置varnish服务进程的工作特性，例如监听的地址和端口，缓存机制；/run/varnish.pid/usr/bin/varnishadm  #客户端脚本/usr/bin/varnishtest  #测试工具程序：#Shared Memory Log交互工具：/usr/bin/varnishhist/usr/bin/varnishlog/usr/bin/varnishncsa/usr/bin/varnishstat/usr/bin/varnishtop/usr/sbin/varnishd  #主程序/usr/sbin/varnish_reload_vcl  #编译配置文件#启动脚本/usr/lib/systemd/system/varnish.service/usr/lib/systemd/system/varnishlog.service     #持久化日志  二选一/usr/lib/systemd/system/varnishncsa.service    #持久化日志  二选一

3. Configuration files

[[email protected] varnish]# vim /etc/varnish/varnish.paramsRELOAD_VCL=1   #启动程序后能否reload配置文件VARNISH_VCL_CONF=/etc/varnish/default.vcl   #VARNISH_LISTEN_PORT=80 #默认6081 端口   如果直接当web服务器用改80# Admin interface listen address and port  管理端口地址VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1VARNISH_ADMIN_LISTEN_PORT=6082VARNISH_SECRET_FILE=/etc/varnish/secret  #密钥#VARNISH_STORAGE="malloc,256M"  #缓存VARNISH_STORAGE="file,/www/data/varnish,1g"VARNISH_USER=varnishVARNISH_GROUP=varnish

[[email protected] data]# mkdir varnish/cache[[email protected] data]# chown -R varnish.varnish varnish

4. Start

[[email protected] varnish]# systemctl start varnish[[email protected] varnish]# ss -lntup|grep varnishtcp    LISTEN     0      128       *:80                    *:*                   users:(("varnishd",pid=1765,fd=7))tcp    LISTEN     0      10     127.0.0.1:6082                  *:*                   users:(("varnishd",pid=1764,fd=6))tcp    LISTEN     0      128      :::80                   :::*                   users:(("varnishd",pid=1765,fd=8))

5. Cache Policy Background Configuration

[[email protected] varnish]# vim /etc/varnish/default.vclbackend default {    .host = "192.168.1.202";    .port = "80";}

6. Heavy Load

Loading vcl from /etc/varnish/default.vclCurrent running config name isUsing new config name reload_2018-06-06T09:17:17VCL compiled.VCL ‘reload_2018-06-06T09:17:17‘ now activeavailable       0 bootactive          0 reload_2018-06-06T09:17:17Done

7, admin administration

[[email protected] varnish]# varnishadm-hvarnishadm:invalid option--' h ' usage:varnishadm [-N ident] [-t timeout]    [-S Secretfile]-t [address]:p ORT command [...] -N is mutually exlusive with-s and-t[[email protected] varnish]# varnishadm-s/etc/varnish/secret-t 127.0.0.1:608 2200-----------------------------Varnish Cache CLI 1.0-----------------------------linux,3.10.0-693.el7.x86_64, x86_64,-sfile,-smalloc,-hcritbitvarnish-4.0.5 revision 07eff4c29type ' help ' for command list. Type ' quit ' to close CLI session.varnish> vcl.list #配置文件版本200available 0 bootactive 0 reload_2018-06-06 t09:17:17# Switch VCL Configuration every time the profile load is saved, here you can switch vcl.use boot200vcl ' boot ' now activevcl.use RELOAD_2018-06-06T09:17:17200VCL ' Reload_2018-06-06t09:17:17 ' now active# view default configuration options vcl.show-v reload_2018-06-06t09:17:17# number of threads Param.show Thread_ Pools200thread_pools Value is:2 [pools] (default) default Is:2 Minimum is:1param.show Thread_pool_  Max #min200thread_pool_max      Value is:5000 [threads] (default) default is:5000 Minimum is:100# set parameters varnish> Param.set thread_p Ools 4200# Storage View varnish> storage.list200storage devices:storage. Transient = malloc Storage.s0 = file# back end view varnish> backend.list200backend name Refs Admin Pr Obedefault (127.0.0.1,,8080) 1 probe Healthy (no probe) default (192.168.1.202,,80) 1 probe Hea Lthy (no probe) #--------------------------------------------------------------------------#非交互式 [[email  Protected] varnish]# varnishadm-s/etc/varnish/secret-t 127.0.0.1:6082 statuschild in state running

Rule writing

The default rule view is important because varnish reads the client configuration before reading the default rule

[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.listavailable       0 bootavailable       0 reload_2018-06-06T09:17:17available       0 reload_2018-06-06T10:43:58available       0 reload_2018-06-06T11:08:20available       0 reload_2018-06-06T12:52:47available       0 reload_2018-06-06T13:04:19active          0 reload_2018-06-06T13:06:08[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.show -v reload_2018-06-06T13:06:08

1. Do not check cache

sub vcl_recv {        if (req.url ~ "(?i)^/(login|admin)") {                return(pass);        }}

2. Synthetic error code

sub vcl_recv {    if (req.http.User-Agent ~ "(?i)curl"){        return(synth(403));    }}marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.200/admin.php<!DOCTYPE html>

3. Default VCL_RECV Policy

sub vcl_recv { if (req.method != "GET" && req.method != "HEAD" && req.method != "PUT" && req.method != "POST" && req.method != "TRACE" && req.method != "OPTIONS" && req.method != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.method != "GET" && req.method != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } return (hash);}

4. Specify resources to remove cookies to cache

sub vcl_recv { if (req.url ~ "(?i)\.(jpg|jpge)$" ){ unset req.http.Cookie; return(hash); }}sub vcl_backend_response { if (beresp.http.cache-control !~ "s-maxage") { if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") { unset beresp.http.Set-Cookie; set beresp.ttl = 3600s; } }} 

5. Client IP

 vcl_recv： 如果多级代理需要加 ，默认是有X-Forwarded-For if (req.http.X-Forwarded-For){ set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip; }else { set req.http.X-Forwarded-For = client.ip; }

6. Cache Pruning (practical)

#白名单设置 掩码写在外面acl purgers{# "192.168.1.104"; "127.0.0.1"/8;}sub vcl_recv { if(req.method == "PURGE"){ if (!client.ip ~ purgers){ return(synth(403)); } return(purge); }}#默认规则里面的行为 sub vcl_purge { return (synth(200, "Purged"));} 




Above three requests, first delete cache, second no hit (add cache), third hit cache

在交互模式下(用于临时清理) 这种方式很有用ban req.url ~ ^/javascript #清空所有/javascript 开头的ban req.url ~ ^/ #清空所有#配置文件清空if (req.method == "BAN") { ban("req.http.host == " + req.http.host + " && req.url == " + req.url); # Throw a synthetic page so the request won‘t go to the backend. return(synth(200, "Ban added"));}

7. Bind Multiple Backgrounds

  import directors; #默认是backend defaultbackend WEBSRV1 {. host = "192.168.1.201"; . Port = "80";} Backend Websrv2 {. host = "192.168.1.202"; . Port = "80";} Sub Vcl_init {#轮询 New Websrvs = Directors.round_robin (); Websrvs.add_backend (WEBSRV1); Websrvs.add_backend (WEBSRV2); #权重 New WEBSRVS3 = Directors.random (); Websrvs3.add_backend (websrv1,1); Websrvs3.add_backend (websrv2,2);} #可以区分出来 dynamic page with a static page sub Vcl_recv {if (req.url ~ "(? i) \.php$") {set req.backend_hint = WEBSRV1; } else {Set req.backend_hint = Websrvs.backend (); }} #---------------------------------------------------------#会话绑定sub vcl_init {New h = Directors.hash (); H.add_backend (one, 1); Backend ' one ' with weight ' 1 ' h.add_backend (1); Backend '-weight ' 1 '}sub vcl_recv {//Pick a backend based on the cookie header of the client set req.b Ackend_hint = H.backend (Req.http.cookie);}  

8. Health status Monitoring

probe www_probe { .url = "/index.html"; .timeout = 1s; .interval = 4s; .window = 5; #探测5次 成功3次算在线 .threshold = 3; .expected_response：期望的响应码，默认为200；}backend websrv1 { .host = "192.168.1.201"; .port = "80"; .probe = www_probe;}backend websrv2 { .host = "192.168.1.202"; .port = "80"; .probe = www_probe;}[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 backend.listBackend name Refs Admin Probedefault(192.168.1.201,,80) 9 probe Healthy (no probe)websrv1(192.168.1.201,,80) 5 probe Healthy 5/5websrv2(192.168.1.202,,80) 5 probe Healthy 5/5在交互模式下手动设置下线backend.set_health websrv Sickbackend.set_health websrv Healthy

Service Process Configuration

DAEMON_OPTS="-p thread_pool=3  thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"thread_pool 2 线程池 小于等于cputhread_pool_min 100 每个线程池最小几个线程 默认100thread_pool_max 5000 最大几个线程  默认5000thread_pool_timeout 300.000 空闲线程的最长时常  超过thread_pool_min个数后 空闲的超过这个时间就会被杀死thread_queue_limit 20  线程池队列长度默认20  可以增加thread_pool_add_delay 0.000 [seconds]  增加不延迟thread_pool_destroy_delay  1 second 杀死延迟一秒thread_pool_fail_delay 0.200 [seconds] 创建失败时候，间隔多少时间在重新创建client：send_timeout 600.000   发送响应包给客户端timeout_idle 5     保持连接超时空闲时常  可调增加timeout_req  2      接收客户端请求报文首部  超过2秒放弃cli_timeout    admin 管理工具请求超时时常  

backend BE_NAME {    ...    .connect_timeout = 3.5s;    .first_byte_timeout = 60s;    .between_bytes_timeout = 60s;}

Log

1. Varnishstat-varnish Cache Statistics

    -1    -1 -f FILED_NAME     -l：可用于-f选项指定的字段名称列表；    MAIN.cache_hit     MAIN.cache_miss    # varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss    # varnishstat -l -f MAIN -f MEMPOOL[[email protected] varnish]# varnishstat -1 #一次显示所有[[email protected] varnish]# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_missMAIN.cache_hit              63         0.00 Cache hitsMAIN.cache_miss             18         0.00 Cache misses

2. Varnishtop-varnish Log Entry Ranking

    -1     Instead of a continously updated display, print the statistics once and exit.    -i taglist，可以同时使用多个-i选项，也可以一个选项跟上多个标签；    -I <[taglist:]regex>    -x taglist：排除列表    -X  <[taglist:]regex>[[email protected] varnish]# varnishtop -i RespStatus  #压测 状态码

3, log files, generally do not open, first-class agent use

[[email protected] varnish]# systemctl start varnishncsa[[email protected] varnish]# tail /var/log/varnish/varnishncsa.log192.168.1.104 - - [06/Jun/2018:20:54:42 +0800] "GET http://192.168.1.200/robots.txt HTTP/1.1" 404 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"

Linux Cache Service Varnish