Squid Study Notes

Squid Study Notes

 

 

1. Pre-installation Configuration

 

Parameters to be corrected before compilation and installation mainly include file descriptor and mbuf clusters.

1. file descriptor

View the maximum number of file descriptors:

Ulimit-n

Three steps are required to adjust the number of file descriptors:

Step 1: edit the system header file/usr/include/bits/types. h.

Define _ fd_siziefd 8192

Step 2: adjust the number of FD limits for the kernel

Echo 8192>/proc/sys/fs/file-max

Step 3: Modify the FD limit for the current shell Environment

Ulimit-hn8192

 

2. mbuf Clusters

 

3. Adjust the range of the temporary Port

The default local port used by squid to connect to the client is 3128, and the default remote port used to connect to the server is 80. When squid is connected to the source server, the system assigns a port number to the local socket, which has specific range restrictions.

Echo "1024 40000">/proc/sys/NET/IPv4/ip_local_port_range

 

2 ,. the final task of the configure script is to create makefiles and other files. In some cases, you may need to re-run the configure script. For example, if you have adjusted the kernel parameters, you must run it again. /configure to make the new settings take effect. You need to re-run the configure script before using the make command to re-compile the script. There are two methods: first, run the following command: config. status -- recheck; the second is to use the command touch config. status, which updates the timestamp of the file. Note: If you want to add or delete the./configure option, you must re-enter the complete command line. Common configure options:

-- Prefix = Prefix: the installation directory of squid. The default value is/usr/local;
-- Localstatedir = dir: the installation location of the VaR directory. The default value is $ prefix/var;
-- Sysconfdir = dir: installation location of the etc directory. The default value is $ prefix/etc;
-- Enable-dlmalloc [= lib]: Compile and link the dlmalloc package in the squid source code package;
-- Enable-gnuregex: activates the GNU Regular Expression if necessary;
-- Enable-carp: the cache array routing protocol (CARP) is used to forward the lost cache to the array or cluster of the parent cache;
-- Enable-async-io [= n_threads]: asynchronous I/O can be used to improve storage performance. The aufs module uses a large number of threads to perform disk I/O operations. The n_threads parameter changes the number of threads used by squid.

-- Enable-async-io options are equivalent to the following three options:
-- With-pthreads: links to the P thread library of the system during compilation. The aufs storage module is the only part of squid that requires threads.
-- With-aufs-threads = n_threads: specifies the number of threads used by the aufs storage mechanism. Squid automatically calculates the number of threads required based on the number of cache directories.
-- Enable-storeio = UFS, aufs, diskd, Coss, null: indicates the storage module used by squid during compilation. All available storage modules are available under/src/Fs in the source code. The UFS module is the default and the most stable, but its disadvantage is that its performance is relatively limited.

-- Enable-removal-policies = heap, LRU: the exclusion policy is used to exclude old targets when squid needs to free up space for new cache targets. LRU (least recently used) is the default value. It uses the double-stranded table structure, GDS (greed dual size) and LFU (least frequently used) use the heap structure. Note that, the implementation of this data structure is not unique.
-- Enable-ICMP: uses ICMP messages to determine the round-trip time, which is similar to the Ping program;
-- Enable-delay-pools: latency pool, used to limit bandwidth. The pool consists of a large number of client IP addresses. When requests from these clients miss the cache, their responses will be artificially delayed.
-- Enable-usreagent-Log: records the HTTP message header of the User-Agent in the client request message;
-- Enable-Referer-Log: record the HTTP message header of the Referer from the client request message;
-- Disable-WCCP: A Web Cache coordination protocol (WCCP) used to intercept and distribute HTTP requests to one or more caches. The WCCP protocol is activated by default. This option can be used to prevent WCCP code compilation.
-- Enable-SNMP: compile all SNMP-related code during compilation, including a cashed version of cmu snmp library;
-- Enable-cachemgr-hostname [= hostname]: cachemgr is a CGI program used to query and manage squid. The default value of hostname is null.
-- Enable-ARP-ACL: supports ARP, Ethernet addresses, and access control lists;
-- Enable-HTCP: HTCP is a hypertext Cache Protocol, which is similar to the internal Cache Protocol of ICP;
-- Enable-SSL: Allows squid to terminate the SSL/TLS connection, which is only valid for the acceleration requests in proxy mode;
-- With-OpenSSL [= dir]: tells squid where to find the OpenSSL library file and header file;
-- Enable-Cache-digests: the cache digest is another available alternative to ICP;

-- Enable-err-languages ages = "lang1 lang2. ..": language of the error message to be installed. The corresponding language file will be copied to $ prefix/share/errors. If this option is not specified, all available languages in the source package will be installed.

-- Enable-default-err-Language = LANG: default language used for error messages. English is the default value.

-- With-Coss-membuf-size = N: This option sets the memory buffer size of the Coss cache directory. The default value is 1 MB. Cyclic Object Storage System (CoSS) is an experimental storage mechanism of squid.
-- Enable-Poll: forces the use of the poll () function;
-- Desable-Poll: Do not use the poll () function;
-- Disable-http-violations: by default, squid allows a configuration that violates the HTTP specification. This option can completely prohibit code that violates the HTTP specification.
-- Enable-IPF-transparent: uses an IP Filter to intercept the cache.
-- Enable-PF-transparent: Use http interception on the operating systems that use the PF package filter. PF is the standard packet filter ).
-- Enable-Linux-netfilter: netfilter is the name of PF. If you want to use the HTTP Interception Function in Linux 2.4 or later versions, activate this option.
-- Disable-Ident-lookups: IDENT is a simple protocol that allows the server to find the user name of a specific TCP connection from the client.
-- Disable-Internal-DNS: the squid Source Code contains two different DNS implementations. One is built-in Lookup (using squid's own implementation of DNS ), one is external Lookup (using the C library gethostbyname () and gethostbyaddr () Functions). The former is used by default. This option activates external searches and disables built-in searches.
-- Enable-truncate: truncate () system call is a substitute for unlink. Unlink completely deletes the cache file, while truncate sets the file size to zero. Some people think that this method of detaching disk space but leaving directory items can improve performance.
-- Disable-hostname-checks: by default, squid requires the URL host name to comply with the old RFC 1034 standard. This option can be used to remove this restriction.
-- Enable-underscores: allows the host name to contain underscores, but you must still consider that the DNS resolution program may prohibit the host name from containing underscores.
-- Enable-auth = Basic, digest, NTLM: sets the HTTP authentication mechanism;

-- Enable-Basic-auth-helpers = List: compile one or more HTTP basic verification aids in the helpers/basic_auth directory;
-- Enable-NTLM-auth-helpers = List: compile one or more http ntlm verification aids in the helpers/ntlm_auth directory;

-- Enable-NTLM-Fail-Open: allows access to the NTLM verification module when an error occurs.

-- Enable-digest-auth-modules = List: compile one or more HTTP digest verification aids in the helpers/digest_auth directory;
-- Enable-external-ACL-helpers = List: compile one or more extended ACL auxiliary programs;
-- Disable-unlinked: an external auxiliary process that executes unlink () or truncate () system calls on the cache file. We think that by executing the file deletion in an external process, it can significantly improve squid performance;
-- Enable-stacktrace: If squid crashes, data tracing information is written to the cache. log file;
-- Enable-X-accelerator-vary: used when squid is configured as a proxy. It is recommended that squid search for the X-accelerator-vary header from the original backend server when responding to the request.

 

3. Squid directory structure

 

Libexec: auxiliary program, usually started by other programs;

Libexec/unlinked: delete files from the cache directory. Deleting a file is a performance bottleneck of squid. Executing a delete operation through an external process can improve the performance.

Libexec/cachemgr. cgi: CGI interface of the squid management function, which needs to be copied to the cgi-bin directory of the Web server;

Libexec/diskd: If -- enable-storeio = diskd is specified, the file is visible;

Libexec/Pinger: If -- enable-ICMP is specified, the file is visible;

Sbin: Only programs available as root;

Sbin/squid: the main program of squid;

Bin: programs available to all users;

Bin/runcache: Start the squid script;

Bin/runaccel: It is almost the same as runcache, but another command line parameter is added to tell squid where to listen for HTTP requests.

Bin/squidclient: the HTTP client program used to test squid; for example: squidclient-M get http://www.baidu.com> tmp.html

ETC: configuration file;

ETC/squid. conf: Main configuration file;

ETC/squid. conf. Default: default configuration file;

ETC/mime. conf: indicates the MIME type used by squid for data obtained from FTP and Gopher servers;

ETC/mime. conf. Default: Default MIME file;

Share: usually contains read-only data files;

Share/mib.txt: SNMP Management Information Base (MIB) file, which is required by SNMP client software (such as snmpget and MRTG;

Share/icons: storage icon;

Share/errors: error message template;

VaR: contains files that are not very important and frequently changed, and do not need to be backed up;

Var/logs: log file;

Var/cache: cache directory;

 

 

4. Squid command line options

When running the $ prefix/squid/sbin/squid main program, you can specify some options, such:

-H: View help information.
-V: View version information.
-A port: overwrite the http_port value of squid. conf (only the first port is affected ).
-U port: overwrite the icp_port value of squid. conf.
-D level: writes the debugging message to stderr. Level specifies the highest level of the message to be written to stderr. It complies with the debug_options instruction.

-S: cache. log message records system logs/etc/syslog. Messages of debug level 0 and 1 are forwarded, and messages of level 0 are recorded by Syslog level log_warning, messages of level 1 are recorded at the syslog level log_notice. All messages use the syslog Device of local4.
-F file: Specifies an optional configuration file.
-Z: initializes the cache and swap directories. This option is required when you first run squid or add a cache directory.

-N: Run squid on the foreground terminal. The warning/error message is printed directly on the screen (stderr );

-D: DNS initialization test is not performed when squid is started;

-X: Enter the complete debugging mode, that is, debug_options all 9;

-K function: the function can be reconfigure, rotate, shutdown, interrupt, kill, debug, check, and parse. Reconfigure allows the running squid program to reload the configuration file; rotate allows the log file to be stored cyclically; shutdown sends a signal to squid to close the process; interrupt immediately closes squid; kill sends force-off kill signals to squid (final means); debug converts squid to full debugging mode; check checks the status of squid process; parse parses squid. CONF file to check whether the configuration is correct.

For example:

Squid-K debug; sleep 5; squid-K debug // obtain the debugging log information for 5 seconds

 

5. Common configuration commands of squid. conf

 

= Network parameter configuration =

Http_port: the port number of the HTTP request listening client by squid. The default value is 3128;

Https_port: the port number that squid listens to the HTTPS request from the client. The default value is 443.

 

= Log and cache directory =

Cache_dir: cache_dir scheme directory size L1 L2. Set cache directory parameters, such as storage mechanism (UFS by default) and storage directory ($ prefix/var/cache by default.

Logformat: Specify the log storage format as needed;

Access_log: Specifies the directory where access. log is stored. The default value is $ prefix/var/logs. Access. log records client requests. If no record is required, you can define access_log/dev/null.

Cache_log: Specifies the directory where cache. log is stored. The default value is $ prefix/var/logs. Cache. log records debugging messages and other information of squid.

Cache_store_log: Specifies the storage directory of store. log. The default value is $ prefix/var/logs. Stroe. log records every object in and out of the cache. If you do not need to record it, You can define cache_store_log none.

Debug_options: defines the Log Level of the squid function module. The default value is debug_options all 1;

 

 

= Options for configuring the cache size =

Cache_mem: used to store in-transit objects (in-request), hot objects (Successful requests), and negative-cached objects (failed requests). The default value is 8 Mb;

Maximum_object_size_in_memory: objects exceeding this value are not cached in the memory. The default value is 8 KB;

Memory_replacement_policy: the Policy for replacing objects when the memory cahce space is insufficient. LRU is the default value;

Cache_replacement_policy: disk cache replacement policy, including LRU, gdsf, and lfuda. LRU is the default value;

Minimum_object_size: Objects lower than this value cannot be buffered. The default value is 0;

Maximum_object_size: objects that exceed this value cannot be buffered. you can determine the object size based on the Content-Length header in the response message. If the response does not contain the Content-Length header, the object is first written to the disk, check the object size. The default value is 4096kb.

Cache_swap_low: When the usage (percentage) of the cache directory exceeds this value, the old object will be deleted to ensure that the usage does not exceed this value. Therefore, when squid runs stably, the usage of the cache directory is close to this value. The default value is 90;

Cache_swap_high: The default value is 95;

 

= Adjust cache Options =

Refresh_pattern: refresh_pattern-I Regexp min percent Max, indirectly controlling disk buffering. There is a compromise between the cache hit rate and the latest data performance.

Quick_abort_min: if the client suspends the request and the remaining content of the resource is less than 16 KB, the cache continues to download by default. The default value of quick_abort_min is 16 kb;

Quick_abort_max: if the client suspends the request and the remaining content of the resource is greater than 16 KB, the cache will cancel the download by default, and the default value of quick_abort_max is 16 KB;

Read_ahead_gap: size of the buffer to be sent to the client for prefetch. The default value is 17kb;

Negative_ttl: defines the negative-cached cache time (time_to_live) of the error page (such as "404 Not Found );

Positive_dns_ttl: the time to live for successful DNS query. The default value is 6 hours;

Negative_dns_ttl: The negative caching object survival time for failed DNS queries .;

Range_offset_limit:

Request_header_max_size: specifies the maximum size of the HTTP header of the request;

Request_body_max_size: specifies the maximum value of the HTTP Request body;

Collapsed_forwarding: when multiple identical Uris are returned to the source, they are merged into one request for processing.

 

 

= Access control =

ACL: Access Control List, in the format of ACL aclname Sr/dest ip/mask.

Http_access: Filter client requests. Format: http_access allow/deny aclname.

Http_reply_access: Filter server replies;

Icp_access: the Internet cache protocal is the communication protocol between sibling caches. This command is used to configure the access control mechanism between the cache and other adjacent caches.

Reply_body_max_size: if the message body in the Response Message exceeds this value, delete it immediately;

Reply_header_max_size: the object in the response message cannot exceed this value;

 

= Timeout settings =

Request_timeout: The request times out;

Persistent_request_timeout: maximum interval between two requests in a persistent connection;

Connect_timeout: Connection timeout;

 

= Other items =

Update_headers: When the 304 not modified response code is received, squid updates the relevant header domain by default. update_headers off can block this feature;

Via: closes or enables the via header. The via header is used to record the intermediate nodes (such as gateway, proxy, and router) that the message passes through during transmission );

Header_access: header_access allow/deny header_name, which allows or disables access to a certain header domain;

Header_replace: the content of the header field rejected by header_access can be replaced by header_replace (removed by default );

Deny_info: when a request is filtered out by access rules, squid will display an error page. This configuration item can define the error page.

Forwarded_for: indicates whether the squid records the hostname or IP address in the request it forwards. It is enabled by default. For example, X-forwarded-for: 192.1.2.3;

Reload_assist_ims: convert a no-cache or reload request to an IMS request (if-modified-since );

Cachemgr_passwd: the Cache Management password;

Cache_inclutive_user: Set a valid account for running squid;

Cache_effective_group: sets a valid user group;

Visible_hostname: Set the host name;

 

= Administrator parameter configuration =

Cache_mgr: sets the Administrator mailbox. When an error occurs in squid, a message is sent to the specified mailbox, such as cache_mgr [email protected].

 

= Cache_peer option =

Cache/no_cache: cache allow/deny alcname. The specified URL resource is not cached;

 

= External support options =

Storeurl_rewrite_program:

Storeurl_rewrite_children:

Storeurl_rewrite_concurrency:

Url_rewrite_program:

Url_rewrite_children:

Url_rewrite_concurrency:

Url_rewrite_host_header:

 

= Delay pool parameter configuration =

Store_dir_select_algorithm: This algorithm selects the cache directory for the cache object. There are two algorithms: Least-load and round-robin. The former is the directory with the smallest load, and the latter is the directory with no full load.

Minimum_expiry_time :( expires-date), minimum cache time;

Vary_ignore_expire:

Server_http11: Use http/1.1 in outbound requests;

Client_persistent_connections: whether the client supports persistent connections. It is enabled by default;

Server_persistent_connections: whether the server supports persistent connections. It is enabled by default;

Ignore_ims_on_miss:

Coredump_dir:

Squid Study Notes