Use inotify + rsync to deploy the dual server load balancer WEB server environment

1. Overview of the server running * 216GSAS300G * 2centos5. 5ip: 192.168.1.21web2 (new) r1_e5606 * 216GSAS300G * 2centos5. 5ip: {* 2centos5. 3discuz forum 50 overview
Server
Web1 R410 E5520 * 2 16G SAS300G * 2 Centos5.5 ip: 192.168.1.21
Web2 (new) R410 E5606 * 2 16G SAS300G * 2 centos5.5 ip: 192.168.1.23
Db R410 E5504 * 2 16G SAS146G * 2 centos5.3
DiscuzThe forum has more than 5 million posts, nearly 20 thousand online users, 4 million pv/day,
Added web2 server sharing forum php processing.
I tried to run the nfs mount web1 program in web2, but the access speed was too slow and unacceptable.
Using inotify + rsync works well. the original load of web1 is about 8, and now the two servers are 3.5

Web2 installation
Configure php-fpm after installing php in web2
192.168.1.23: 9002# Listening address and port
Rsyncuser# Users who follow rsync synchronization
Website# Synchronize user groups
128# Currently, 128 hosts occupy approximately 4 GB of physical memory
192.168.1.21, 192.168.1.23# Request-filling server (web1, web2)

Web2 openIptablesPort
Iptables-a input-p tcp-m tcp-s 192.168.1.21 -- dport 9002-j ACCEPT
/Etc/init. d/iptables save

Add database server permissions
Add the same user access permissions (discuz, ucenter…) to web2 as web1 ...)
Web1 installation settings Install inotify in web1
Https://github.com/rvoicilas/inotify-tools/wiki/
Inotify-tools 3.14 is the latest version, released on the 7th of March 2010.
Wget? Http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz no-check-certificate
Tar zxvf inotify-tools-3.14.tar.gz
Cd inotify-tools-3.14
Make
Make install
Check manpage, man inotify, and man inotifywait.

Check whether inotify is supported
The kernel has been officially integrated into kernel since kernel 2.6.13, and RHEL5 is supported.
Check whether the/proc/sys/fs/inotify/directory exists to determine whether the kernel supports inotify.
Ll/proc/sys/fs/inotify
Total 0
-Rw-r -- 1 root 0 Sep 14 14:01 max_queued_events
-Rw-r -- 1 root 0 Sep 14 14:01 max_user_instances
-Rw-r -- 1 root 0 Sep 14 14:01 max_user_watches

Test inotify
/Usr/local/bin/inotifywait-mrq? Timefmt '% d/% m/% y % H: % M '? Format '% T % w % F'-e modify, delete, create, attrib/opt/lampp/htdocs/bbs
16/09/11/opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
16/09/11/opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
16/09/11/opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
....

Rsync synchronization
Method 1: log on without using ssh
Method 2: Use rsync to authenticate logon-free
Method 1

Added synchronization users in web2
Useradd-g website rsyncuser
Passwd rsyncuser

Create a public key in web1 and upload it to web2
Ssh-keygen-t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/. ssh/id_rsa ):
Enter passphrase (empty for no passphrase ):
Enter same passphrase again:
Your identification has been saved in/root/. ssh/id_rsa.
Your public key has been saved in/root/. ssh/id_rsa.pub.
The key fingerprint is:

Copy to web2
Scp-P 22 ~ /. Ssh/id_rsa.pub rsyncuser@192.168.1.23 :.

Web2 setting logon-free
Import id_rsa.pub to. ssh/authorized_keys
Cd/home/rsyncuser
Mkdir. ssh
Cat id_rsa.pub>. ssh/authorized_keys
Chown-R rsyncuser: website. ssh
Rm id_rsa.pub
You can log on to the Apsara stack console under the web1 test.
Ssh-p 22 rsyncuser@192.168.1.23

Synchronization script
Inofity exclude supports posix regular expressions, but only one can be written.
Note that the path for storing bbs on both sides should be the same
You can use rsync to synchronize data.
Vi inotify. sh
#! /Bin/sh
Yesterday = 'date + % Y % m % d'
Src =/opt/lampp/htdocs/bbs
Des = rsyncuser@192.168.1.23:/opt/lampp/htdocs/
/Usr/local/bin/inotifywait-mrq -- exclude "data/(threadcache | log | template | Sendmail\. Lock | cache) "-- timefmt '% d/% m/% y % H: % M' -- format' % T % w % f '\
-E modify, delete, create, attrib $ {src} | while read file
Do
Echo-e $ (date + % Y-% m-% d _ % H: % M: % S) "\ r"> rsynclog/inotify. $ {yesterday }. log
Rsync-av -- delete -- exclude "data/threadcache/" -- exclude "data/log/" -- exclude "data/template/" -- exclude "data/sendmail. lock "$ {src}-e"/usr/bin/ssh-p 22 "$ {des}> rsynclog/inotify. $ {yesterday }. log
Done

Create non-synchronous directories and files on web2
Cd/opt/lampp/htdocs/bbs
Mkdir data/{threadcache, log, template}
Chown-R rsyncuser: website data/{threadcache, log, template}
Chmod-R 0775 data/{threadcache, log, template}
Touch data/sendmail. lock
Chown-R rsyncuser: website data/sendmail. lock
Chmod-R 0775 data/sendmail. lock

Set permissions and run
Chmod 700./inofity. sh
./Inofity. sh &

Add to start running
Echo 'CD/opt/shell &./inofity. sh & '>/etc/rc. local

Update the cache Directory
Skip this step if you use memcache.
When file caching is used, the/opt/lampp/htdocs/bbs/data/cache directory contains cache for newly registered members and online users. it is not suitable for inotify when files are updated frequently.
Vi rsync. sh
#! /Bin/sh
# ***** Cd/opt/shell &/bin/sh./rsync. sh>/dev/null 2> & 1
Yesterday = 'date + % Y % m % d'
Src =/opt/lampp/htdocs/bbs/data/cache
Des = rsyncuser@192.168.1.23:/opt/lampp/htdocs/bbs/data/
Echo-e $ (date + % Y-% m-% d _ % H: % M: % S) "\ r"> rsynclog/rsync. $ {yesterday }. log
Rsync-av -- delete $ {src}-e "/usr/bin/ssh-p 22" $ {des2}> rsynclog/rsync. $ {yesterday}. log

Set permissions
Chmod 775./rsync. sh
Adding crontab-e to crontab
Run every minute
* *** Cd/opt/shell &/bin/sh./rsync. sh>/dev/null 2> & 1
After the first synchronization, enable php
/Opt/php/sbin/php-fpm start

Web1 enabled Server load balancer
Web1 adjustmentNginx
First, create a test. php file for testing. you can see different machine names in phpinfo or check the file header through firebug to determine whether the file is on web2.
Echo' '> Test. php
Vi/opt/nginx/conf/nginx. conf
Location ~ /Test \. php? $
{
# Fastcgi_pass 127.0.0.1: 9000;
Fastcgi_pass 192.168.1.23: 9002;
Add_header App-Server php2;
Fastcgi_index index. php;
Fcinclude GI. conf;
Break;
}
For the official launch, you only need to perform load balancing for forum. php. 90% of the load is on this file (only when the forum is enabled)
Upstream backend {
Ip_hash;
# Ensure that the shunting intranet can access the public network or there is no external access demand
Server192.168.1.23: 9002 max_fails = 3 fail_timeout = 60 s;
Server 127.0.0.1: 9000;
}
Location ~ /Forum \. php? $
{
# Fastcgi_pass unix:/tmp/php-cgi.sock;
# Fastcgi_pass 127.0.0.1: 9000;
# Fastcgi_pass 192.168.1.23: 9002;
# Add_header App-Server php2;
Fastcgi_pass backend;
Fastcgi_index index. php;
Fcinclude GI. conf;
Break;
}
Fcgi. conf
Fastcgi_param GATEWAY_INTERFACE CGI/1.1;
Fastcgi_param SERVER_SOFTWARE nginx;
Fastcgi_param QUERY_STRING $ query_string;
Fastcgi_param REQUEST_METHOD $ request_method;
Fastcgi_param CONTENT_TYPE $ content_type;
Fastcgi_param CONTENT_LENGTH $ content_length;
Fastcgi_param SCRIPT_FILENAME $ document_root $ fastcgi_script_name;
Fastcgi_param SCRIPT_NAME $ fastcgi_script_name;
Fastcgi_param REQUEST_URI $ request_uri;
Fastcgi_param DOCUMENT_URI $ document_uri;
Fastcgi_param DOCUMENT_ROOT $ document_root;
Fastcgi_param SERVER_PROTOCOL $ server_protocol;
Fastcgi_param REMOTE_ADDR $ remote_addr;
Fastcgi_param REMOTE_PORT $ remote_port;
Fastcgi_param SERVER_ADDR $ server_addr;
Fastcgi_param SERVER_PORT $ server_port;
Fastcgi_param SERVER_NAME $ server_name;
# PHP only, required if PHP was built with -- enable-force-cgi-redirect
Fastcgi_param REDIRECT_STATUS 200;

Restart nginx
Reload nginx
/Bin/kill-HUP 'cat/dev/shm/nginx. Pi'
Discuz uploads attachments, and the background is fixed in web1. operations such as reading and pasting are distributed on two machines based on the user's ip address.