Application of squid in Image Storage Architecture

I have synchronized it from my independent blog and discussed it together

I haven't written any articles for nearly a month, so I'm so lazy. I'm so excited today. I wrote an article about squid, which is really great to use squid as a distributed Proxy Cache Server, I hope this article will help you.

Copyright Disclaimer: non-commercial reprinting, retaining the integrity of the original content, and signing the author Wang Guofeng and the original text link

Preface

In the previous article, I showed my position on caching: caching all resources that can be cached. It also discussed how to use the cache function of IIS to easily meet the needs of small and medium sized image caching. You can review this article about IIS Client Cache and server cache. Since the publication of the previous article, many people have reported to me that the cache function of IIS has limited load, and varnish or squid is recommended. Yes, I agree very much, so today I will discuss with you the application of the distributed Proxy Cache Server squid in the Image Storage Architecture. If you have any mistakes, please point out, thank you.

Squid Introduction

Squid is a high-performance distributed proxy cache server. It is generally used for pre-Web Cache to speed up Web access. To illustrate the running process, I will continue to use the YD style of my previous article. Once again, I would like to invite an empty teacher who sees people and flowers and cars, and ask the readers to give a round of applause:

Squid is currently the most popular proxy for empty works. To allow all empty fans to enjoy high-quality empty works more quickly, squid will work overtime every day to organize and save the latest empty works, and further optimize it, such as de-code.

One day, I was so bored that I called squid and asked him for the latest empty image (sending a download request to Squid );

Squid carefully identified my voice and found it to be an old friend, so he agreed to my request (ACL Access Control in Squid, only accept valid requests );

Squid said to me that he had an empty photo just a week ago and asked me if he had read it. I said he sent it to me and sent it to me soon (Squid cache hit, download is very fast, oh yeah !);

I received the photo and found that I had read it two days ago. So I told squid to be up-to-date. Squid looked at the inventory and found that there was no latest inventory, so let me wait for a while. He will ask if other proxy brothers are there. I agree, thank you (Squid supports distributed cluster mode, cache data between parent and child nodes and brothers can be synchronized );

He came back and told me that there were no other brothers. When I was very lost, he suddenly said that he would come to Hangzhou today, and he could send a picture to the empty man, as soon as I was happy, I told him to go quickly (when the Squid cache fails, data can only be obtained from the source server );

After a long time, squid was back. He said there were too many empty people. After a long time, he got her latest photo, after I thanked him, he quickly sent me the photo. Then he said that he must save the photo to provide it to other empty fans and other brothers (squid will save the data obtained from the source server to its own cache, if the next user initiates the same download request, squid directly sends the cached data to the user. Of course, it can also send the data to other brokers, so Lele is inferior to everyone ).

Here, I believe you have understood the working process of squid. Below I will briefly summarize the features of squid:

1. Open-source, based on the GNU General Public License, means you can use and modify squid under the same open-source protocol.
2. Supports multiple protocols. Currently, HTTP, FTP, Gopher, WAIS, SSL, and other network protocols are supported.
3. Supports distributed clusters. Squid uses TCP (HTTP) and UDP (ICP/HTCP) communication to determine the neighboring cache status.
4. Access control is supported. You can set ACL and ARL to restrict certain accesses.

Squid: http://www.squid-cache.org/

You can download squid and read relevant documents on the official website. Of course, I will also introduce the usage of squid in Chinese.

Squid configuration details

The following is a detailed configuration list of squid and its description:

# The http_port Command tells the port on which the squid listens for HTTP requests. The default port is 3128. In addition to the following format, it can also be http_port 192.168.63.50: 3128.
Http_port 3128
Icp_port 3130

# Set the cache directory. You can set multiple cache directories. The syntax is: <cache_dir> <aufs | UFS> <directory> <Mbytes size> <dir1> <dir2>
Cache_dir ufs/var/spool/squid 1000 64 1024

# The following is the Directory and file name for storing log files!
Cache_access_log/var/log/squid/access. Log
Cache_log/var/log/squid/cache. Log
Cache_store_log/var/log/squid/store. Log
Pid_filename/var/run/squid. PID

# Disable the authentication mechanism. In some versions, squid is automatically added to the proxy authentication mechanism, which is not required in general cases. Therefore, locate the lines including auth_param and add comments to them.
# Auth_param basic children 5
# Auth_param basic realm Squid proxy-caching Web Server
# Auth_param basic credentialsttl 2 hours
 
# Set squid users, user groups, and administrator accounts
Cache_inclutive_user squid
Cache_paitive_group squid
Cache_mgr youraccount@your.e.mail

# Memory-related configuration: Because my system memory is very small, I only give 8 MB! If your physical memory is large, such as 512 MB, you can increase it to 64 or 128 MB.
Cache_mem 128 MB

# Disk capacity-related configuration (Note: The following 90 and 95 are percentages). If your cache_dir disk is large, you can change 4096 to 32768 kb.
Cache_swap_low 90
Cache_swap_high 95
Maximum_object_size 4096 KB

# Configuration related to memory storage
Maximum_object_size_in_memory 8 KB

# Define the ACL (Access Control List). Syntax: ACL <ACL> <ACL Name> <ACL type> <configuration content>
# For user-defined part
 
ACL all SRC 0/0
ACL manager proto cache_object
ACL localhost SRC 127.0.0.1/32
ACL safe_ports port 80 21 443 563 70 210 280 488 591 777-1025
ACL ssl_ports 443 563
ACL connect method connect
ACL mynetwork SRC 192.168.0.0/16

# Use the previously defined ACL to define access control rules
Http_access allow manager localhost
Http_access deny Manager
Http_access deny! Safe_ports
Http_access deny connect! Ssl_ports
Http_access allow mynetwork
Http_access deny all

# Define the relationship with other proxy servers. Syntax: <cache_peer> <Host Name> <Category> Cache_peer 192.168.60.6 parent 4480 7 NO-query default

# Set the relationship with other proxy servers:
# <Cache_peer_access> <upper-layer proxy> <allow | deny> <ACL Name>
# Cache_peer_access 192.168.60.6 allow aclxxx
# Cache_peer_access 192.168.60.6 deny! Aclxxx
 
Coredump_dir/var/spool/squid

Note: The preceding configuration instructions are from here.

After the configuration is complete, we can start squid as follows:

1). Run the following command to check the correctness of the configuration file:

# Squid-K parse

Only when this step is correct can you proceed to the next step.

2) initialize the cache directory, that is, create the storage format of the cache directory.

Run the command before starting the squid service for the first time (before running squid for the first time, or whenever you add a new cache_dir, you must initialize the cache directory .)

# Squid-z

The cache directory initialization may take some time, depending on the size and quantity of the cache directory, as well as the disk drive speed. If you want to observe this process, use the-x option:

# Squid-ZX

3) Start the squid service

# Service squid start

If squid is installed in the/usr/local/squid directory

#/Usr/local/squid/sbin/squid-SD

Finally, I recommend you go to the squid authoritative guide and hope you can use squid to build a Distributed Image cache system. Squid is really great!

For the health of you and your family, please indicate the original source: http://www.itivy.com/ivy/archive/2012/3/10/squid-usage.html

In addition, I think you may be interested in the following Architecture-related articles:
Youku Architecture Study Notes
Technical preparations for websites with millions of visits
Learning the Image Storage Architecture of Facebook
Full Analysis of Facebook Image Storage Architecture Technology
Thank you for reading this article!