Construction of a large-scale forum system environment (0.2 million IP load balancing practices)

(This article only discusses about the discuz forum system. Because the software package update speed is fast, you may have used the new version of the software package when reading this post, and the installation methods may be different, for details, see the README file of the software package)
　　
Test environment: ideal forum.
　　
The ideal forum is the most popular stock forum in China. The number of registered members has exceeded 1 million and increases steadily at a rate of 60000 people per month. The daily page access volume has exceeded 2 million and maintained a steady growth trend, an average of more than 20 thousand online users within 60 minutes, with a maximum record of more than 30 thousand. Currently, the topic is over 0.3 million, the post size is close to 10 million, the database size is 5.8 GB, and the total attachment size is about 150 GB.
　　
In the past, the ideal forum had three servers, two Web servers, and one Database server. The access had gradually experienced bottlenecks. At the pig's suggestion, the webmaster decided to add a server to the database, the other three websites are used to upgrade the operating system of the original server.
　　
Hardware details
　　
MySQL Server: dualxeon 5335/8 GB memory/73 gb sas hard drive (raid0 + 1)/centos5.1-x86_64/mysql5
　　
The three web servers are as follows:
　　
N1. dual Xeon 3.0 2 GB memory
　　
N1. dual Xeon 3.0 4 GB memory
　　
N1. dual Xeon 3.0 (dual-core) 4G memory
　　
In addition, three gb scsi hard disks are prepared for RAID 5 to store attachments. The four machines are connected through the Intranet.
　　
Pig has considered the following solutions:
　　
1. Zeus + PhP5 + eaccelerator
　　
2. Squid + apache2 + PHP + eaccelerator
　　
3. nginx + PhP (FastCGI) + eaccelerator
　　
4. nginx + apache2 + PHP + eaccelerator
　　
The first solution is perfect and stable, but the biggest problem is that Zeus is a paid software, and piracy will always be blamed.
　　
In the second solution, squid forwards requests to apache2. Many websites adopt this method and the efficiency is very high. pig's head has also been tested, but the problem is very serious, because Squid caches the file, squid must open it for every accessed file. The ideal forum has a GB attachment, and the traffic is huge, in this case, only squid is enabled, and the machine will soon refuse to respond to any request because of too many files opened. It does not seem to be suitable. It is only suitable for websites with only a few hundred MB of cached files.
　　
In the third solution, pig's test result for the third solution is that when the traffic volume is high, PHP often displays Bad Gateway. It seems that the method of executing PHP through the TCP connection FastCGI is not stable enough, pig head also tested to run php through a UNIX socket connection, which is also unstable.
　　
In contrast, pig head currently uses the fourth solution.
　　
Install apache2.
　　
(Because the server uses freebsd7, most of the software will be installed through ports)
　　
Because apache2 only needs to process PHP requests, it is basically not required for other modules, so do not install other modules, even if rewrite is not required, because rewrite will be implemented on nginx, if you are familiar with it, you can also modify the makefile to delete unnecessary parts. After optimization, Apache will process PHP requests in the most stable and efficient way.
　　
CD/usr/ports/www/apache20
　　
Make install clean
　　
Modify httpd. conf (only the parts to be modified/added are listed here)
　　
VI/usr/local/etc/apache2/httpd. conf
　　
Change keepalive on to keepalive off and add
　　
Serverlimit 2048
　　
Maxclients increased to 512
　　
Listen 127.0.0.1: 81 # because the httpd server does not need to be opened to the outside, it only needs to listen to the local port for handling PHP requests forwarded by nginx.
　　
Added support for PHP.
　　
Addtype application/X-httpd-PHP. php
　　
Addtype application/X-httpd-PHP-source. PHPs
　　
As for adding a VM, it will not be too long. Note that the VM can also listen to port 81 on the local host.
　　
Install PhP5 (install GD library and other modules in advance)
　　
CD/usr/ports/lang/PhP5
　　
Modify makefile and add the required items.
　　
There should have been such a paragraph
　　
Configure_args = \
　　
-With-layout = GNU \
　　
-With-config-file-scan-Dir =$ {prefix}/etc/PHP \
　　
-Disable-all \
　　
-Enable-libxml \
　　
-With-libxml-Dir =$ {localbase }\
　　
-Enable-reflection \
　　
-Program-Prefix = ""
　　
We want to change it
　　
Configure_args = \
　　
-With-layout = GNU \
　　
-With-config-file-scan-Dir =$ {prefix}/etc/PHP \
　　
-Disable-all \
　　
-Enable-libxml \
　　
-With-libxml-Dir =$ {localbase }\
　　
-Enable-reflection \
　　
-Program-Prefix = ""\
　　
-With-config-file-Path =/etc-enable-mbstring-enable-ftp-with-Gd-with-JPEG-Dir =/usr/local-with-PNG-Dir =/usr/local-enable-magic-quotes-with-mysql =/usr/local-with-pear-enable-sockets-with-TTF-with-FreeType-Dir =/ usr/local-enable-Gd-native-TTF-with-zlib
-Enable-sysvsem-enable-sysvshm-with-libxml-Dir =/usr/local-with-PCRE-RegEx-enable-XML
　　
Make install clean
　　
CP work/php-5.2.5/PHP. ini-Dist/etc/PHP. ini
　　
Install eaccelerator
　　
CD/usr/ports/www/eaccelerator
　　
Make install clean
　　
Add the following parts to the end of PHP. ini:
　　
Extension_dir = "/usr/local/lib/PHP/20060613 /"
　　
Extension = "eaccelerator. So"
　　
Eaccelerator. cache_dir = "/tmp/eaccelerator"
　　
Eaccelerator. shm_size = "64 ″
　　
Eaccelerator. Enable = "1 ″
　　
Eaccelerator. optimizer = "1 ″
　　
Eaccelerator. check_mtime = "1 ″
　　
Eaccelerator. DEBUG = "0 ″
　　
Eaccelerator. Filter = ""
　　
Eaccelerator. shm_max = "0 ″
　　
Eaccelerator. shm_ttl = "60 ″
　　
Eaccelerator. shm_prune_period = "60 ″
　　
Eaccelerator. shm_only = "0 ″
　　
Eaccelerator. Compress = "1 ″
　　
Eaccelerator. compress_level = "9 ″
　　
Eaccelerator. Keys = "shm_and_disk"
　　
Eaccelerator. Sessions = "shm_and_disk"
　　
Eaccelerator. content = "shm_and_disk"
　　
Create cache directory and modify permissions
　　
Mkdir/tmp/eaccelerator
　　
Chmod 777/tmp/eaccelerator
　　
Chown nobody: Nobody/tmp/eaccelerator
　　
Nginx installation and configuration
　　
CD/usr/ports/www/nginx
　　
Make install
　　
There are several modules that we need.
　　
HTTP Module
　　
Http_addition Module
　　
Http_rewrite Module
　　
Http_realip Module
　　
Http_stub_status Module
　　
Others are needed.
　　
Modify configuration file
　　
VI/usr/local/etc/nginx. conf
　　
User nobody Nobody;
　　
Worker_processes 4;
　　
# Error_log logs/error. log;
　　
# Error_log logs/error. Log notice;
　　
# Error_log logs/error. Log Info;
　　
# PID/var/log/nginx. PID;
　　
Events {
　　
Worker_connections 10240;
　　
}
　　
HTTP {
　　
Include mime. types;
　　
Default_type application/octet-stream;
　　
Limit_zone one $ binary_remote_addr 10 m;
　　
# Log_format main '$ remote_addr-$ remote_user [$ time_local] $ request'
　　
# '"$ Status" $ body_bytes_sent "$ http_referer "'
　　
# '"$ Http_user_agent" "$ http_x_forwarded_for "';
　　
Sendfile off;
　　
Tcp_nopush off;
　　
# Keepalive_timeout 0;
　　
Keepalive_timeout 10;
　　
Gzip off;
　　
Server {
　　
Listen 80;
　　
SERVER_NAME www.55188.net
Www.55188.com www1.55188.com www2.55188.com 55188.com 55188.net
Www.55188.cn 55188.cn bbs.55188.net bbs.55188.com bbs.55188.cn;
　　
Index index.html index.htm index. php;
　　
Root/home/WWW;
　　
Access_log/dev/null combined;
　　
Limit_conn one 5; # limit the number of concurrent connections of one IP address to five
　　
Error_page 404/404 .html;
　　
Error_page 403/403 .html;
　　
Location/status {
　　
Stub_status on;
　　
Access_log off;
　　
Auth_basic "nginxstatus ";
　　
Auth_basic_user_file CONF/htpasswd;
　　
}
　　
# Use the discuz6.0 rewrite rule in the root directory. If your forum is under the level-2 Directory, modify the location accordingly.
　　
Location /{
　　
Rewrite ^/archiver/(FID | tid)-[\ W \-] + \. html) $/archiver/index. php? $1 last;
　　
Rewrite ^/Forum-([0-9] +)-([0-9] +) \. html $/Forumdisplay. php? FID = $1 & page = $2 last;
　　
Rewrite ^/thread-([0-9] +)-([0-9] +)-([0-9] +) \. html $/viewthread. php? Tid = $1 & extra = Page \ % 3d $3 & page = $2 last;
　　
Rewrite ^/space-(username | UID)-(. +) \. html $/space. php? $1 = $2 last;
　　
Rewrite ^/Tag-(. +) \. html $/Tag. php? Name = $1 last;
　　
Break;
　　
Error_page 404/404 .html;
　　
Error_page 403/403 .html;
　　
}
　　
# Do anti-leech protection for attachments. If there is no correct Referer, the system will return the 403 page
　　
Location ~ * ^. + \. (GIF | JPG | PNG | SWF | FLV | RAR | zip | Doc | PDF | GZ | bz2 | JPEG | BMP | XLS) $ {
　　
Valid_referers none blocked server_names * .55188.net * .55188.com;
　　
If ($ invalid_referer ){
　　
Rewrite ^/http://www.55188.com/403.html;
　　
}
　　
}
　　
# Forward PHP requests to the local port 81 for Apache to process.
　　
Location ~ \. Php $ {
　　
Proxy_pass http: // 127.0.0.1: 81;
　　
Proxy_redirect off;
　　
Proxy_set_header host $ host;
　　
Proxy_set_header X-real-IP $ remote_addr;
　　
Proxy_set_header X-forwarded-for $ proxy_add_x_forwarded_for;
　　
Proxy_hide_header content-type;
　　
}
　　
}
　　
}
　　
Test whether all your configuration files are correct.
　　
/Usr/local/sbin/apachectl configtest
　　
/Usr/local/sbin/nginx-T
　　
If there is no problem, start the server.
　　
/Usr/local/sbin/apachectl start
　　
/Usr/local/sbin/nginx-C/usr/local/etc/nginx. conf
　　
Check the Home Page. It should be normal.
　　
Subsequent discussions,
　　
1. database.
　　
The compilation and installation of databases will not be discussed repeatedly. We only discuss the environment. Because the database in the ideal forum is large and developing rapidly, we need to compare the previous budget, the hard drive requires a SAS hard drive of 15 K rpm for RAID 0 + 1, and the operating system requires a 64-bit version, because the server requires 8 GB of memory, note that, after a 64-bit system is used, you may not be able to find the 64-bit version of some old software. This machine is dedicated to MySQL servers. If the database exceeds 10 Gb, consider mysql_cluster.
　　
2. attachment.
　　
Because there are three servers for Web, attachments should be shared through the intranet using NFS. We will not discuss how to set up NFS. If you do not understand, please pay the tuition fee to Google.
　　
3. Web.
　　
Because the hardware configurations of the three machines are inconsistent, it is necessary to consider the load balancing problem. nginx itself has the load balancing function, but if the load balancing function is enabled, each machine will cache the data requested by the client to the local machine, which increases the hard disk I/O. This is not a small overhead for the access to the ideal forum, finally, we use DNS queries to distribute traffic. With Different a Records and better-configured machines, one more a record will be allocated, and one less a record will be generated if the configuration is poor, in this way, the traffic distribution should be balanced as a whole.
　　
4. nginx concurrent connections
　　
Pig restricts the concurrent connection of each IP address to nginx, because it is always famous for large forums, not to mention people attacking you or anything. there are a lot of collections. If there is no limit, it is very likely that problems will often lead to PhP strikes.
　　
The above is just a pig's head. If there are other progresses, the pig's head will update this post. If you have any questions or different opinions, you are welcome to discuss it.
　　
Of course, there are still many crazy methods. For example, you can put all the web files (except attachments) in the memory. If MySQL is less than 5 GB, it can also be put in all the memory, however, these methods are too powerful, and the optimization effect must be good, but the risk is high.
　　
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~
　　
Optimized results
　　
Because the upgrade of two other machines is not complete, I will only post the status of one web server. Currently
　　
Active connections: 1143
　　
Server accepts handled requests
　　
1211445 1211445 6221785
　　
Reading: 67 writing: 136 waiting: 940
　　
For Apache optimization, You need to disable unnecessary modules. Because httpd requests are all processed by nginx, Apache only needs to process PHP. Currently, I have enabled the modules.
　　
Loadmodule access_module libexec/apache2/mod_access.so
　　
Loadmodule setenvif_module libexec/apache2/mod_setenvif.so
　　
Loadmodule mime_module libexec/apache2/mod_mime.so
　　
Loadmodule autoindex_module libexec/apache2/mod_autoindex.so
　　
Loadmodule negotiation_module libexec/apache2/mod_negotiation.so
　　
Loadmodule alias_module libexec/apache2/mod_alias.so
　　
Loadmodule rewrite_module libexec/apache2/mod_rewrite.so
　　
Loadmodule php5_module libexec/apache2/libphp5.so
　　
Autoindex negotiation and rewrite should all be disabled, but relevant modifications should be made. Currently, it is definitely better to run than FastCGI.