Lighttpd + squid + Apache to build an efficient Web Server

 

Architecture Principle

Apache is usually the preferred web server in the Open Source Field. Because of its powerful and reliable nature, Apache has a brand effect and can be applied to most applications. However, its strength is sometimes cumbersome, and the configuration file is daunting. In high concurrency, the efficiency is not very high. The lightweight Web Server Lighttpd is a rising star. Its static file response capability is much higher than that of Apache, which is said to be 2-3 times that of Apache. The high performance and ease of use of Lighttpd are enough to impress us and use it whenever possible in the field where it is competent. Lighttpd also supports PHP well and supports other languages such as Python through FastCGI.

After all, Lighttpd is a lightweight server and cannot be compared with Apache in terms of functions. Some applications are not competent. For example, Lighttpd does not support caching, and most of today's websites use programs to generate dynamic content. Without caching, even if the program is more efficient, it is difficult to meet the needs of large traffic volumes, in addition, it is meaningless for the program to keep doing the same thing. First, the web program needs to cache the data that is used repeatedly. Even this is not enough. Simply starting a web handler costs a lot, and caching the final static page is essential. This is squid's strength. It serves as a proxy and supports efficient caching. It can be used to provide reverse proxy acceleration for websites. Put squid on the front end of Apache or Lighttpd to cache the dynamic content generated by the Web server, and the Web application only needs to set the page validity time appropriately.

Even for websites dynamically generated by most of the content, there are still some static elements, such as slice, JS scripts, CSS, etc. Placing squid on the Apache or lighttp front-end will lead to performance degradation, after all, processing HTTP requests is the strength of web servers. In addition, static content that already exists in the file system is cached in Squid, which wastes memory and hard disk space. Therefore, you can consider placing Lighttpd in front of squid to form a processing chain of Lighttpd + squid + Apache. Lighttpd is at the very beginning and is used to process static content requests, the dynamic content request is forwarded to Squid through the proxy module. If the content of the request in Squid does not expire, it is directly returned to Lighttpd. New requests or expired page requests are handled by the Apache web program. After Lighttpd and squid filtering, Apache will greatly reduce the number of requests to be processed, reducing the pressure on Web applications. At the same time, this architecture allows for different processing to be distributed across multiple computers, and the Lighttpd checks each other.

In this architecture, each level can be individually optimized. For example, Lighttpd can adopt asynchronous Io mode, squid can enable memory for caching, and Apache can enable MPM, in addition, multiple machines can be used at each level to balance the load, which is highly scalable.

Instance description

The following uses several sites in the daviesliu.net and rainbud.net domains as examples to introduce the specific practices of this solution. In the daviesliu.net domain, there are several blog sites implemented using mod_python, several PHP sites, and a small mod_python program. Several PHP and Django sites may be set up in the future. The server is very weak, the CPU is celon 500, and the memory is PC 100 384 M. Therefore, the efficiency of web servers is more important. These sites are deployed on the same port of the same machine in the form of virtual hosts.

Lighttpd serves port 80, squid runs on port 3128, and Apache runs on port 81.

Lighttpd Configuration

Multiple Domain Names use the/var/www/domain/subdomain directory structure. Use the evhost module to configure document-root as follows:

Evhost. Path-pattern = var. basedir + "/% 0/% 3 /"

Ftpsearch has a Perl script and must enable CGI support. It is used for FTP site search and cache is of little significance. It is directly processed by mod_cgi of Lighttpd:

$ HTTP ["url"] = ~ "^/Cgi-bin/" {# Only allow CGI's in this directory
Dir-listing.activate = "Disable" # disable directory listings
CGI. Assign = (". pl" => "/usr/bin/Perl", ". cgi" => "/usr/bin/Perl ")
}

BBS uses phpBB, which has a low access volume. You can place it in Lighttpd (FastCGI) or Apache (mod_php) and temporarily use Lighttpd to set FastCGI processing for all. PHP page requests:

FastCGI. server = (". PHP "=> (" host "=>" 127.0.0.1 "," Port "=> 1026, "bin-path" => "/usr/bin/PHP-cgi ")))

Blog.daviesliu.net and blog.rainbud.net are blogxp programs written in mod_python. All static content has an extension, but dynamic content has no extension. Blogxp uses a python program to generate data in XML format and then convert it to an HTML page using mod_xslt. It can only be run under Apache. This site uses the typical Lighttpd + squid + Apache method for processing:

$ HTTP ["host"] = ~ "^ Blog "{
$ HTTP ["url"]! ~ "/."{
Proxy. Server = ("" => ("localhost" => ("host" => "127.0.0.1", "Port" => 3128) # port 3128 is
}
}

Share has a static page, and also uses mod_python to process requests. In/cgi:

$ HTTP ["host"] = ~ "^ Share "{
Proxy. Server = (
"/Cgi" => ("localhost" => ("host" => "127.0.0.1", "Port" => 3128 ))
)
}

Squid Configuration

Only Local access is allowed:

Http_port 3128
Http_access allow localhost
Http_access deny all

Enable reverse proxy:

Httpd_accel_host 127.0.0.1
Httpd_accel_port 81 # Apache Port
Httpd_accel_single_host on
Httpd_accel_with_proxy on # enable Cache
Httpd_accel_uses_host_header on # enable VM support

In this direction, the proxy supports all domain names on the host.

Apache configuration

Configure/etc/CONF. d/apache2 to load the mod_python, mod_xslt, and mod_php modules:

Apache2_opts = "-D Python-d xslt-D PhP5"

Root directory of all websites:

<Directory "/var/www">
AllowOverride all # Allow. htaccess to overwrite
Order allow, deny
Allow from all
</Directory>

Domain name-based VM:

<Virtualhost *: 81>
Servername blog.daviesliu.net
DocumentRoot/var/www/daviesliu.net/blog
</Virtualhost>

The evhost configuration of Lighttpd is obviously not convenient here.

The. htaccess settings under blog.daviesliu.net (for ease of development, do not restart Apache ):

Sethandler mod_python
Pythonhandler blogxp. Publisher
Pythondebug on
Pythonautoreload on

<Filesmatch "/.">
Sethandler none # static files are directly processed by Apache
</Filesmatch>

<Ifmodule mod_polict.c>
Addtype text/XSL. XSL # prevents transformation of XSL files
Addoutputfilterbytype mod_xslt text/XML
Xsltcache off
Xsltprocess on
</Ifmodule>
Header set Pragma "cache"
Header set cache-control "cache"

In blogxp. Publisher, you also need to set the type and expiration time of the returned document:

Req. content_type = "text/XML"
Req. headers_out ['expires'] = formatdate (Time. Time () + 60*5)

With this configuration, all sites can be accessed through ports 80, 3128, and 81, and port 80 is used for external access to reduce load. Port 81 Can Be Used for debugging during development without caching.

Performance Testing

Due to limited time and energy, we only use AB2 for a non-standard performance comparison test (each item is tested multiple times on average). The evaluation indicator is the number of requests per second.
The test command is used to test the Lighttpd concurrency of 10 requests scripts/prototype. JS:

AB2-N 1000-C 10 http://blog.daviesliu.net: 80/scripts/prototype. js

Static content: Prototype. js (27kb)

Con	Lighttpd (: 80)	Squid (: 3128)	Apache (: 81)
1	380	210	240
10	410	215	240
100	380	160	230

It can be seen that the Lighttpd performance is strong in static content, while squid performance is worse than that of the other two Web servers without memory cache configuration.

Dynamic page:/RSS (31 KB)

Con	Lighttpd (: 80)	Squid (: 3128)	Apache (: 81)
1	103	210	6.17
10	110	200	6.04
100	100	100	6.24

In terms of dynamic content, squid plays an obvious role, while Lighttpd is limited by Squid's efficiency, and it also needs to be much lower. If multiple squids are used for balancing, the effectiveness of Lighttpd can be realized.
In the case of a single machine with few static content, you can place squid at the beginning without Lighttpd.

 

User message: Re: Lighttpd + squid + Apache to build an efficient Web Server 1. windtear was published at p.m. on January 1, September 13, 2006.

This kind of matching is acceptable, but there are some problems in the text description.

Light can add cache support on its own, but the performance only considers cache to be a little better than squid (an average of 3000 + online actual data per second)

Squid is not very useful for processing hitratio with static optimizations above 99.99%.

It is also good for the overall structure.

Light + squid + Apache the actual online architecture during the transition period. At that time, the backend did not support compression.

In fact, each piece can be patched according to their own needs. It is not the best, but it is more suitable for manageability.

Re: Lighttpd + squid + Apache build an efficient Web Server 2. soff was published at p.m. on January 1, September 13, 2006.

Lighttpd + PhP will often cause PHP to die if the traffic is high, and then 500

Whether local or remote

Helpless, for Zeus, very strong, commercial is commercial.

Re: Lighttpd + squid + Apache to build an efficient Web Server 3. kxn was published at p.m. on November 31, September 13, 2006.

His result looks weird, as a result, his conclusion is wrong.

Squid does not boost dynamic page at all, the speed gain in his test is because his client is requesting the same page in paralell, and squid will return the same page for the concurrent requests. I also guess that he did not configure expire time for static content in his web server, squid will try to refetch the file with if-modified-Since Header for each request. that's why squid performs poor in the static test.

Re: Lighttpd + squid + Apache build an efficient Web Server 4. Davies published at p.m. on January 1, September 13, 2006.

I do not agree with this. For squid, dynamic pages and static pages are the same, as long as the HTTP header is set,

If expires is set, there is no cache effect.

If dynamic pages cannot be cached, how can acceleration be achieved?

 

Re: Lighttpd + squid + Apache build an efficient Web Server 5. kxn was published at p.m. on January 1, September 13, 2006.

Sorry, the English is not good. It misleads you. You cannot enter Chinese characters on the unit machine in the morning.

Dynamic Pages do not have any acceleration effect unless the HTTP expiration time header is correctly set. In other words, static pages also need to set the expiration time header.

Setting the expire time refers to setting the expiration time to several minutes or several hours, so that the page is completely buffered in Squid during this time.

You actually test the dynamic page to Improve the Performance. There are several possibilities. First, you test the function of using concurrent requests for the same page, and squid for concurrent requests for the same page, if the obtained result does not contain a non-Cache header, the result will be sent back to all requests at the same time, which is equivalent to a very short cache, and the test results will look much better, however, because there are not many opportunities to request the same page, there is basically no improvement. Another case is that the dynamic page program you use supports the IF-modified-Since Header, if he judges whether the modification has been made after this time, he will directly return not modified, which will speed up a lot.

In fact, squid is used to buffer static pages most of the time in actual production. dynamic pages do not have to be buffered. However, it requires a lot of cooperation in the page program to achieve better results.

Newsmth's WWW peak time is QPS, squid end is relatively easy, the bottleneck is in the back end.

Re: Lighttpd + squid + Apache build an efficient Web Server 6. Davies was published at p.m. on January 1, September 13, 2006.

Thank you for your answers!

As I wrote in this Article, each request will add the Expires header as the last 5 minutes of the current time, that is, each page is valid for 5 minutes, squid seems to determine whether to refresh the cache based on the time. If-modified-since is not supported by the server

These 5 minutes are determined based on the general page update frequency.

For a web application with a large access volume, such as newsmth's WWW, if the expiration time of the PHP page is set to 1-2 seconds, requests during this period will be responded to by the cache, even if the data is updated during this cache period, it does not affect the user's usage. The 1-2 second lag effect does not significantly affect the user experience, however, in exchange for faster server response, especially for blog parts with large traffic but not frequently updated, this may be very effective.

Of course, if the IF-modified-since interface is implemented, it will be more effective, but the workload is too large.

Re: Lighttpd + squid + Apache to build an efficient Web Server 7. kxn was published at p.m. on January 1, September 13, 2006.

It seems that I did not read your article carefully and did not notice that the expire header was mentioned in your article.

You can also set the expire header on a static page.

In this way, squid is used to buffer all data.

If the expire header is not available, squid will use if modified since to fl each request.

The expire time on the PHP page of smthwww is 5 minutes or 10 minutes. I forgot.

Re: Lighttpd + squid + Apache to build an efficient Web Server 8. scaner published at p.m. on January 1, September 13, 2006.

The general feeling is that the squid solution is sufficient if there is no very large access volume.

If Lighttpd is used, there is basically no need for Apache,

Unless it is a very special application, Lighttpd basically supports.

When a single machine is tossing so many layers, there will be no performance gains.

Re: Lighttpd + squid + Apache to build an efficient Web Server 9. Wei LiTaO was published at p.m. on July 19, September 16, 2006.

In fact, the cache function of Lighttpd is very powerful. You can refer to his NML document to solve the caching problem of dynamic content. In addition, it is of little significance to set up a squid for a single server. Of course, the squid bloom filter is very effective unless you have too many items to cache.

Re: Lighttpd + squid + Apache build an efficient Web Server 10. bianbian was published at p.m. on November 9, April 6, 2007.

Lighttpd has bugs, causing serious memory leakage. I am using nginx and testing the effect on lilybbs. Static dynamic content is the final way out. Those clicks really need to be removed.

The current architecture of lilybbs:

------ Nginx ---------

|

Squid FastCGI proxy

| (Gradual migration) |

Static file njuwebbsd

(Migrate to FastCGI step by step)

Re: Lighttpd + squid + Apache build an efficient Web Server 11. bianbian was published at p.m. on November 11, April 6, 2007.

Ft. Space Layout is not supported. For the architecture, see:

Http://bbs.nju.cn/blogcon? Userid = bbsadm &...

Re: Lighttpd + squid + Apache build an efficient Web Server 12. bianbian was published at p.m. on November 31, April 6, 2007.

In addition. I don't think it is necessary to build a single machine with three layers. You can discard Apache in this case. My regret is that nginx is very strong in other aspects, that is, memcache is not perfect, so you must get a squid.

Re: Lighttpd + squid + Apache build an efficient Web Server 13. Davies was published at a.m.

The solution in my article is only useful in special occasions.

The primary node is used for playing.

Click it to analyze the log to do it offline, or put some data separately, and use ajax to follow the new part.

Re: Lighttpd + squid + Apache build an efficient Web Server 14. Davies was published at a.m.

The first time I heard about nginx, I thought it should be something at the same level as Lighttpd. The difference would not be too big. If you want to fight for concurrency performance, it is estimated that yaws is not enough, and a simple test will be conducted in another day.

Re: Lighttpd + squid + Apache build high-efficiency Web Server 15. posted at p.m. on November 30, October 18, 2007.

Agree to bianbian

If a single machine is used, it is not necessary to make it so complicated.

In general, use squid + Apache

Use Apache expires to set the expiration time

Dynamic Content settings such as expires and last-modified

If there are multiple databases, you can use balance.

Re: Lighttpd + squid + Apache build high-efficiency Web Server 16. Ahu published on November 25, 2007 p.m.

Nginx is something similar to squid, better than it; it is similar to lighttodd in Apache.

Re: Lighttpd + squid + Apache build an efficient Web Server 17. Davies was published on November 11, November 26, 2007.

Unlike squid, nginx has only one simple memcache and cannot be cached on the hard disk.

I am using Lighttpd + mod_cache + mod_memcache to cache images. The results are very good.