Introduction to high-performance open-source http accelerator Varnish

Introduction to high-performance open-source http accelerator Varnish

Varnish
1. Introduction to varnish
Varnish is a high-performance open-source HTTP accelerator that is more stable than squid and has higher efficiency and consumes less resources. I believe varnish has enough capabilities to replace squid in reverse proxy and web acceleration. Verdens Gang, Norway's largest online newspaper, uses three Varnish instances to replace the original 12 squids, providing better performance than before.
 
Ii. structure and features of Varnish
Varnish is a lightweight Cache and reverse proxy software. Its advanced design concept and mature design framework are the main features of Varnish. Currently, Varnish has a small amount of code, although the functions are constantly improved, they still need to be enriched and enhanced. The following summarizes some features of Varnish:
(1) It is based on the memory cache, and the data will disappear after restart.
(2) Using the virtual memory mode, io performance is good.
(3) supports setting 0 ~ Precise cache time within 60 seconds.
(4) VCL configuration management is flexible.
(5) the maximum size of cached files on 32-bit machines is 2 GB.
(6) powerful management functions, such as top, stat, admin, and list.
(7) Clever state machine design and clear structure.
(8) use the binary heap to manage cached files and actively delete the files.

Iii. varnish System Architecture
Varnish mainly runs two processes: the Management process and the Child process (also called the Cache process ).
 
The Management process mainly implements new application configuration, compiling VCL, monitoring varnish, initializing varnish, and providing a command line interface. The Management process detects the Child process every several seconds to determine whether it is running normally. If the Child process does not receive a response within the specified duration, Management restarts the Child process.
 
The Child process contains multiple types of threads, such:
Cache-main thread: there is only one global thread used to start the cache;
Acceptor thread: receives and responds to new connection requests;
Worker thread: The child process starts a worker thread for each session. Therefore, hundreds or more worker threads may occur in high concurrency scenarios;
Expiry thread: clears expired content from the cache;
Backend poll thread: One backend server for detecting the health status of backend servers;
Epoll/kqueue threads: the number of configurable threads. The default value is 2, which is used to manage the thread pool.
 
When configuring varnish, you only need to pay attention to the cache-worker thread, and you can only configure the number of its thread pools. Other parameters are not configurable. At the same time, the number of thread pools needs to be increased only in scenarios with large traffic, and experience shows that more than two of them are not helpful for performance improvement.

Iv. Installation of Varnish
Since the system I use is RHEL5.8, the following software package needs to be installed before installing varnish:
Automake, autoconf, libtool, ncurses-devel, libxslt, groff, pcre-devel, pkgconfig
# Yum-y install *. rpm
 
Compile and install varnish
# Tar xf varnish-2.1.3.tar.gz
# Cdvarnish-2.1.3
#./Configure -- prefix =/usr/loca/varnish
 

# Make & make install

Provide configuration files for varnish
Copy the default configuration file of varnish to/etc/sysconfig/varnish.
# Cp/usr/local/varnish/etc/varnish/default. vcl/etc/sysconfig/varnish
 
Create a working directory for Varnish users and Varnish users
# Useradd-s/sbin/nologin varnish
# Mkdir/web/cache
# Chown-R varnish: varnish/web/cache

In varnish 2.0, you must specify the backend server address for varnish before starting varnish. Therefore, you must edit the varnish configuration file and specify the backend server address.
# Vim/etc/sysconfig/varnish
Add the following information:
Backend default {
. Host = "127.0.0.1 ";
. Port = "80 ";
}
 
V. Start and Stop varnish
#/Usr/local/varnish/sbin/varnishd-f/etc/sysconfig/varnish-s malloc, 200 M-T 127.0.0.1: 2000-a 0.0.0.0: 8080
-F: Specifies the varnish configuration file.
-S: Specifies the storage type and capacity of varnish.
-T: Specifies a management interface based on this document, which can be used to manage varnish without stopping varnish.
-A: Specifies the listening socket.
 
Check whether varnish is started successfully.
# Netstat-tunlp | grep varnish
Tcp 0 0 0.0.0.0: 8080 0.0.0.0: * LISTEN 18684/varnishd
Tcp 0 0 127.0.0.1: 2000 0.0.0.0: * LISTEN 18683/varnishd
Among them, 8080 is the port for Varnish listening, and 2000 is the Varnish telnet-Based Management port.

Stop Varnish and run the following command:
# Killall varnishd
 
Vi. Varnish binary Program Files
After Varnish is installed, many binary program files are generated. Common Commands include:
/Usr/local/varnish/sbin/varnishd: This Varnish startup command is the most important.
/Usr/local/varnish/bin/varnishadm: used to control and manage Varnish instances. Such as deleting a cache object
/Usr/local/varnish/bin/varnishlog: displays the Varnish log information.
/Usr/local/varnish/bin/varnishncsa: This command can also display Varnish log information, except that the log information displayed by this command is in a somewhat similar format to the logs output by apache combine.
/Usr/local/varnish/bin/varnishstat: displays Varnish cache information.
 
VII. varnishd commands
The varnishd startup command is the/usr/local/varnishd/sbin/varnisd Command, which has many parameters, you can use/usr/local/varnish/sbin/varnishd-h to view detailed information about related parameters. Common parameters include:
-A address: port: Specifies the socket for the Varnish listener.
-B address: port: Specify the backend server address and the listening port
-D: indicates the debug mode.
-F file: Specifies the Varnish configuration file.
-P file: Specifies the Varnish pid file.
-P param = value: Specifies the server parameter to optimize Varnish performance.
-N dir: Specifies the working directory of Varnish.
-S kind [, storageoptions]: Specifies the Varnish backend storage method. Common backend storage methods include:
-S malloc,-s file, <dir_or_file>, <size>. Note that in a 32-bit operating system, Varnish supports a maximum storage capacity of 2 GB.
-T: Specifies the default ttl value.
-T address: port: sets the telnet-based management address and port of Varnish.
-V: Specifies the version number of Varnish.
-W int [, int [, int]: specifies the number of Varnish worker threads. The configuration methods are as follows:
-W <fixed_count>
-W min, max
-W min, max, timeout [default:-w2, 500,300]

8. How does Varnish record data?
A notable feature of Varnish is how it records data. varnish records all the data in the memory. When the memory is used, it records the data from the beginning to overwrite the oldest records. Therefore, you can quickly record data without occupying disk space. However, once the server goes down, all the data in the memory will be released.
 
9. varnishlog command
Varnish provides many tools to view data in the memory. Here I only talk about varnishlog applications.
Run the varnishlog command on the command line. The result is as follows:
17 RxHeader c Accept-Encoding: gzip, deflate
17 RxHeader c Host: 192.168.108.202: 8080
17 RxHeader c Connection: Keep-Alive
17 VCL_call c recv
17 VCL_return c lookup
17 VCL_call c hash
17 VCL_return c hash
The first column is an arbitrary number, which is used to define the request. The same number indicates the same HTTP transmission.
The second column is information tag. All logs carry a tag to mark corresponding operations. Rx indicates that varnish receives data, and Tx indicates that varnish sends data.
The third column indicates where the data is transmitted or transmitted, from c (client) or B (backend ).
The fourth column is the recorded data.
Varnishlog also has many sub-options:
-B: Only the logs between varnish and backend server are displayed. You can use this parameter to optimize the hit rate.
-C: similar to-B, but it represents varnish and client communication.
-I tag: only information related to a tag is displayed. For example, "varnishlog-iSessionOpen" only displays new sessions. Note that the tag names in this region are case insensitive.
-I: use regular expressions to filter data. For example, "varnishlog-c-I RxHeader-I Cookie" will display all header information that is sent to the client containing Cookie words.

10. varnishstat command
The binary program of varnishstat is/usr/local/varnish/bin/varnishstat. The cache hit rate directly shows the running status and effect of varnish. A high cache hit rate shows that varnish is in good running state and the performance of web servers is also improved, an excessively low cache hit rate indicates that varnish configuration may be faulty and needs to be adjusted. Therefore, an overall understanding of varnish's hit rate and cache status is crucial for optimizing and adjusting varnish.
Run the/usr/local/varnish/bin/varnishstat command to view the cache status of Varnish. For example:
#/Usr/local/varnish/bin/varnishstat
Hitrate ratio: 1 1 1
Hitrate avg: 0.5294 0.5294 0.5294
 
18 0.00 0.01 Client connections accepted
17 0.00 0.01 Client requests received ed
9 0.00 0.00 Cache hits
8 0.00 0.00 Cache misses
8 0.00 0.00 Backend conn. success
8 0.00 0.00 Fetch with Length
11. N struct sess_mem
1. N struct objectcore
1. N struct objecthead
1. N struct smf
1. N large free smf
10. N worker threads
10 0.00 0.00 N worker threads created
1. N backends
8. N expired objects
8. n lru moved objects
10 0.00 0.00 Objects sent with write
18 0.00 0.01 Total Sessions
17 0.00 0.01 Total Requests
 
Important parameters include:
Client connections accepted: number of accepted connections
Client requests received ed: Total number of requests accepted. If persistent connections are used, this value is generally greater than the Client connections accepted value.
Cache hits: number of hits
Cache misses: the number of non-life periods.
N struct objectcore: Number of cached objects
N expired objects: indicates the number of expired cache objects.
N lru moved objects: indicates the number of cached objects that have been eliminated

Concepts of Cache Server Varnish

Concepts of Cache Server Varnish

Structural notes for Varnish Cache

Install and configure Varnish-5.8 in CentOS 2.1.5

The RedHat script uses the CentOS source to update and install Nginx, PHP 5.3, and Varnish.

Using Varnish to build Cache Server notes

Install and configure the cache service Varnish

Preparations for Varnish compilation and Installation

Configuration Optimization of Varnish cache in Linux

Varnish details: click here
Varnish: click here

This article permanently updates the link address: