Analysis of Varnish cache configuration optimization in Linux

This article provides a detailed analysis of Varnish cache configuration optimization in Linux. For more information, see Varnish is a high-performance open-source HTTP accelerator, verdens Gang, Norway's largest online newspaper, uses three Varnish instances to replace the original 12 squids, providing better performance than before.

However, compared with the old squid, squid has its own advantages and disadvantages. a large number of relatively large numbers of online users only play a role in the maximum use of applications that they are familiar, it is possible that squid has the ability to make full use of the most powerful manpower.
Varnish adopts the "Visual Page Cache" technology. in terms of memory utilization, Varnish is superior to Squid, which avoids Squid from frequently exchanging files in the memory and disk, the performance is higher than Squid.

Through Varnish port management, you can use regular expressions to quickly and batch clear some caches, which is not available in Squid.
I will give a brief introduction and notes on some of varnish's insights and configuration methods.

Lab environment: Red Hat Enterprise Linux Server release 5.4 (Tikanga)
Kernel 2.6.18-164. el5
Yum install pcre-devel # install a software package in advance. Otherwise, an error is prompted.
Tar zxvf varnish-2.1.3.tar.gz
Cd varnish-2.1.3
./Configure -- prefix =/usr/local/varnish-2.1.3
Make & make install
Edit the configuration file. there are templates, but there are too many comments. you 'd better create one by yourself.
Vim/usr/local/varnish-2.1.3/etc/varnish. conf
########### Content and notes of the configuration file #################### ###
# Http request processing
#1. the status of the receive request entry. the pass or lookup local query is determined based on vcl.
# Lookup: Searches for data in the hash table. if it is found, it enters the hit status; otherwise, it enters the fetch status.
# Pass: Select the background to enter the fetch status
# Fetch: Obtain, send, and obtain data from the backend and store the data locally.
# Deliver: send data to the client and enter done.
# Done: processing is finished.
######### Configure backend servers ##############
The code is as follows:
Backend linuxidc01 {
. Host = "192.168.1.142 ";
. Port = "7070 ";
. Probe = {
. Timeout = 5S;
. Interval = 2 s;
. Window = 10;
. Threshold = 8;
}
}
Backend linuxidc02 {
. Host = "192.168.1.141 ";
. Port = "7070 ";
. Probe = {
. Timeout = 5S;
. Interval = 2 s;
. Window = 10;
. Threshold = 8;
}
}

############# Configure the backend server group and perform health check for 6 seconds. set the weight using random ########
######### The default round-robin mechanism is another method ####################
The code is as follows:
Director idc15474 random
{. Retries = 6;
{. Backend = linuxidc02;
. Weight = 2;
}
{. Backend = linuxidc01;
. Weight = 2;
}
}

######### Define the access list, allow the following addresses to clear the varnish cache #######################
The code is as follows:
Acl local {
"Localhost ";
"127.0.0.1 ";
}

######## Determine the types of backend servers and cache configurations from the url ###################### ######
The code is as follows:
Sub vcl_recv
{
If (req. http. host ~ "^ Linuxidc15474.vicp.net") # redirect matching domain names to the backend server
{Set req. backend = linuxidc15474 ;}
Else {error 404 "Unknown HostName! ";}
If (req. request = "PURGE") # Do not allow IP addresses in the non-access control list to clear the varnish cache.
{If (! Client. ip ~ Local)
{
Error 405 "Not Allowed .";
Return (lookup );
}
}
# Clear cookies with jpg files in URLs
If (req. request = "GET" & req. url ~ "\. (Jpg | png | gif | swf | jpeg | ico) $ ")
{
Unset req. http. cookie;
}
# Determine req. http. X-Forwarded-For. if the front end has multiple reverse proxies, you can obtain the client IP address.
If (req. http. x-forwarded-)
{
Set req. http. X-Forwarded-For = req. http. X-Forwarded-For "," client. ip;
}
Else {set req. http. X-Forwarded-For = client. ip ;}
# Varnish implement anti-Leech protection for images
# If (req. http. referer ~ "Http ://.*)
#{
# If (! (Req. http. referer ~ "Http: //. * vicp \. net" |
# Req. http. referer ~ "Http: //. * linuxidc15474 \. net "))
#{
# Set req. http. host = "linuxidc15474.vicp.net ";
# Set req. url = "/referer.jpg ";
#}
# Return (lookup );
#}
# Else {return (pass );}
If (req. request! = "GET "&&
Req. request! = "HEAD "&&
Req. request! = "PUT "&&
Req. request! = "POST "&&
Req. request! = "TRACE "&&
Req. request! = "OPTIONS "&&
Req. request! = "DELETE ")
{Return (pipe );}
# Direct forwarding of non-GET | HEAD requests to backend servers
If (req. request! = "GET" & req. request! = "HEAD ")
{Return (pass );}
# For GET requests, and the url contains. php and. php? And directly forward it to the backend server.
If (req. request = "GET" & req. url ~ "\. (Php) ($ | \?) ")
{Return (pass );}
# Requests with authentication and cookies are directly forwarded to backend servers
If (req. http. Authorization | req. http. Cookie)
{Return (pass );}
{
# Search for the above access requests from the cache
Return (lookup );
}
# The specified font directory is not cached
If (req. url ~ "^/Fonts /")
{Return (pass );}
}
Sub vcl_pipe
{Return (pipe );}
# In pass mode, the request is sent to the backend, and the backend returns data to the client, but does not enter the cache for processing.
Sub vcl_pass
{Return (pass );}
Sub vcl_hash
{
Set req. hash + = req. url;
If (req. http. host)
{Set req. hash + = req. http. host ;}
Else {set req. hash + = server. ip ;}
Return (hash );
}
# After lookup, if the requested cache is found in the cache, the following keywords are generally ended:
Sub vcl_hit
{
If (! Obj. cacheable)
{Return (pass );}
Return (deliver );
}
# Call when the cache is not found after lookup. the following keywords are completed and the fetch parameter is called to re-test whether the cache is added.
Sub vcl_miss
{Return (fetch );}
# Set the cache type of the varnish server to be called after obtaining data from the backend
Sub vcl_fetch
{If (! Beresp. cacheable)
{Return (pass );}
If (beresp. http. Set-Cookie)
{Return (pass );}
# The WEB server indicates that the content is not cached. The varnish server does not cache the content.
If (beresp. http. Pragma ~ "No-cache" | beresp. http. Cache-Control ~ "No-cache" | beresp. http. Cache-Control ~ "Private ")
{Return (pass );}
# Cache the get files in jpg, png, and other formats during access. the cache duration is 7 days and s is seconds.
If (req. request = "GET" & req. url ~ "\. (Js | css | mp3 | jpg | png | gif | swf | jpeg | ico) $ ")
{Set beresp. ttl = 7d ;}
# Cache the static pages including htm in get for 300 seconds
If (req. request = "GET" & req. url ~ "\/[0-9] \. htm $ ")
{Set beresp. ttl = 300 s ;}
Return (deliver );
}
#### Add the header information on the page to view cache hits ########
Sub vcl_deliver
{
Set resp. http. x-hits = obj. hits;
If (obj. hits> 0)
{Set resp. http. X-Cache = "HIT cqtel-bbs ";}
Else {set resp. http. X-Cache = "MISS cqtel-bbs ";}
}

######################## The above configuration file is varnish ######### #################
Create a user:
Groupadd www
Useradd www-g www
Cache location for creating varnish_cache
Mkdir/data/varnish_cache
Start varnish
Ulimit-SHn 8192 #### set the file descriptor. because my sub-database has poor performance, you can set it according to your own configuration.
/Usr/local/varnish-2.1.3/sbin/varnishd-u www-g www-f/usr/local/varnish-2.1.3/etc/varnish. conf-a 0.0.0.0: 80-s file,/data/varnish_cache/varnish_cache.data, 100 M-w 1024,8192, 10-t 3600-T 127.0.0.1: 3500
#### What is-u used to run-g to run-f varnish configuration file-a bound IP address and port-s varnish cache file location and size-w minimum, maximum thread and timeout-T varnish management Port, mainly used to clear the cache
# End the varnishd process
Pkill varnishd
Start varnishncsa to write Varnish access logs to log files:
/Usr/local/varnish-2.1.3/bin/varnishncsa-w/data/logs/varnish. log &
Run at every day, cut the Varnish log by day, generate a compressed file, and delete the script of the old log last month (/var/logs/cutlog. sh ):
Vim/usr/local/varnish-2.1.3/etc/varnish/cut_varnish_log.sh
Write the following script:
#! /Bin/sh
# This file run at 00:00
Date = $ (date-d "yesterday" + "% Y-% m-% d ")
Pkill-9 varnishncsa
Mv/data/logs/varnish. log/data/logs/$ {date}. log
/Usr/local/varnish-2.1.3/bin/varnishncsa-w/data/logs/varnish. log &
Mkdir-p/data/logs/varnish/
Gzip-c/data/logs/$ {date}. log>/data/logs/varnish/##date=.log.gz
Rm-f/data/logs/$ {date}. log
Rm-f/data/logs/varnish/$ (date-d "-1 month" + "% Y-% m *" ).log.gz
Scheduled task:
Crontab-e
00 00 ***/usr/local/varnish-2.1.3/etc/varnish/cut_varnish_log.sh

Optimize Linux kernel parameters
Vi/etc/sysctl. conf
Net. ipv4.tcp _ fin_timeout = 30
Net. ipv4.tcp _ keepalive_time = 300
Net. ipv4.tcp _ syncookies = 1
Net. ipv4.tcp _ tw_reuse = 1
Net. ipv4.tcp _ tw_recycle = 1
Net. ipv4.ip _ local_port_range = 5000 65000
Make the configuration take effect
/Sbin/sysctl-p

Manage ports through Varnish and clear cache in batches using regular expressions
Clear all caches
/Usr/local/varnish-2.1.3/bin/varnishadm-T 127.0.0.1: 3500 url. purge * $
Clear all caches in the image directory
/Usr/local/varnish-2.1.3/bin/varnishadm-T 127.0.0.1: 3500 url. purge/image/
127.0.0.1: 3500 indicates the address of the cache server to be cleared. www.linuxidc.com indicates the domain name to be cleared./static/image/tt. jsp indicates the list of URLs to be cleared.
/Usr/local/varnish-2.1.3/bin/varnishadm-T 127.0.0.1: 3500 purge "req. http. host ~ Www.linuxidc.com $ & req. url ~ /Static/image/tt. jsp"
++ ++
A php function to clear the Squid cache
The code is as follows:
Function purge ($ ip, $ url)
{
$ Errstr = '';
$ Errno = '';
$ Fp = fsockopen ($ ip, 80, $ errno, $ errstr, 2 );
If (! $ Fp)
{
Return false;
}
Else
{
$ Out = "PURGE $ url HTTP/1.1 \ r \ n ";
$ Out. = "Host: blog.s135.com \ r \ n ";
$ Out. = "Connection: close \ r \ n ";
Fputs ($ fp, $ out );
$ Out = fgets ($ fp, 4096 );
Fclose ($ fp );
Return true;
}
}

Purge ("192.168.0.4", "/index. php ");
?>

++ ++
Configure automatic start of Varnish upon startup
Vim/etc/rc. d/rc. local
Write the following content in the last line:
Ulimit-SHn 8192
/Usr/local/varnish-2.1.3/sbin/varnishd-u www-g www-f/usr/local/varnish-2.1.3/etc/varnish. conf-a 0.0.0.0: 80-s file,/data/varnish_cache/varnish_cache.data, 100 M-w 1024,8192, 10-t 3600-T 127.0.0.1: 3500
/Usr/local/varnish-2.1.3/bin/varnishncsa-w/data/logs/varnish. log &
Check the Varnish server connection count and hit rate:
/Usr/local/varnish-2.1.3/bin/varnishstat
The above is the varnish status,
1675 0.00 0.06 Client requests received ed: number of Client requests received by the server
179 0.00 0.01 Cache hits indicates the number of times that data is returned to the client from the Cache, that is, the hit rate.
11 0.00 0.00 Cache misses indicates the number of times that data is retrieved from the backend service application and returned to the user by skipping the pass Cache.
Use help to see which Varnish commands can be used:
/Usr/local/varnish-2.1.3/bin/varnishadm-T 127.0.0.1: 3500 help