Nginx combined with keepalived for high-availability load balancing

1 Planning and preparation

• Application systems that require unified access

Application System	domain/virtual directory	application Server and URL
Svn	Dev.mycompany.com/svn	Http://50.1.1.21/svn
SVN Web Management	Dev.mycompany.com/submin	Http://50.1.1.21/submin
Website	Www.mycompany.com	http://50.1.1.10; http://50.1.1.11; http://50.1.1.12
Oa	Oa.mycompany.com	http://50.1.1.13:8080; http://50.1.1.14:8080

• Access server

Use	IP
MASTER	50.1.1.3
BACKUP	50.1.1.4

Operating system: RHEL5.6X64, CONFIGURED with Yum

Two access servers common one virtual IP (VIP): 50.1.1.2

2 Installation

Two access servers are installed Nginx and keepalived respectively:

• To prepare a dependency package:

Yum-y install gcc pcre-devel zlib-devel openssl-devel

• Download

wget http://nginx.org/download/nginx-1.2.4.tar.gz wget http://www.keepalived.org/software/keepalived-1.2.7.tar.gz

• Installing Nginx

Tar zxvf nginx-1.2.4.tar.gzcd nginx-1.2.4./configure--with-http_stub_status_modulemake && make install

• Installing keepalived

Tar zxvf keepalived-1.2.7.tar.gzcd keepalived-1.2.7./configuremake make installcp/usr/local/etc/rc.d/init.d/ keepalived/etc/rc.d/init.d/cp/usr/local/etc/sysconfig/keepalived/etc/sysconfig/mkdir/etc/keepalivedcp/usr/ local/etc/keepalived/keepalived.conf/etc/keepalived/cp/usr/local/sbin/keepalived/usr/sbin/

• Join the Startup service

echo "/usr/local/nginx/sbin/nginx" >>/etc/rc.local

echo "/etc/init.d/keepalived start" >>/etc/rc.local

3 Configuration 3.1 Configuring Nginx

The Nginx configuration of the two access servers is exactly the same, mainly configuring/usr/local/nginx/conf/nginx.conf http. Where multi-domain points are implemented through virtual hosts (configured under HTTP server), different virtual directories of the same domain are implemented by different location under each server, and the backend servers are configured with upstream under HTTP. It is then referenced through Proxypass in the server or location. To implement the previously planned access method, the HTTP configuration is as follows:

HTTP {    include       mime.types;    Default_type  Application/octet-stream;    Sendfile on        ;    Upstream dev.hysec.com {        server 50.1.1.21:80;    }    Upstream www.hysec.com {      ip_hash;      Server 50.1.1.10:80;      Server 50.1.1.11:80;      Server 50.1.1.12:80;    }    Upstream oa.hysec.com {      ip_hash;      Server 50.1.1.13:8080;      Server 50.1.1.14:8080;          server {        listen      ;        server_name dev.hysec.com;        LOCATION/SVN {            proxy_pass http://dev.hysec.com;        }        location/submin {            proxy_pass http://dev.hysec.com;        }    }    server {        listen       ;        server_name  www.hysec.com;        Location/{            proxy_pass http://www.hysec.com;        }    server {        listen       ;        server_name  oa.hysec.com;        Location/{            proxy_pass http://oa.hysec.com;        }}

Verification Method:

• The URL of each application server in the preceding table is first accessed with IP
• Then use domain name and path to access the domain/virtual path of each application system in the preceding table.

3.2 Configuring Keepalived

Following the installation method above, the keepalived configuration file is/etc/keepalived/keepalived.conf. The configuration of the master, slave server is associated but different. As follows:

Master:

! Configuration File for keepalivedglobal_defs {notification_email {        [email protected]        [email protected]   }   notification_email_from [email protected]   smtp_server smtp.hysec.com   smtp_connect_timeout   router_id nginx_master}vrrp_instance vi_1 {State    Master    interface eth0    virtual_router_id    Priority 101    Advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    Virtual_ IPAddress {        50.1.1.2    }}

Backup:

! Configuration File for keepalivedglobal_defs {notification_email {        [email protected]        [email protected]   }   notification_email_from [email protected]   smtp_server smtp.hysec.com   smtp_connect_timeout   router_id nginx_backup}vrrp_instance vi_1 {State    backup    interface eth0    virtual_router_id    Priority    Advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    Virtual_ IPAddress {        50.1.1.2    }}

Verify:

• Started on the main, from the server keepalived:/etc/init.d/keepalived start
• See if the virtual Ip:ip is already bound on the primary server addr
• Stop keepalived on the primary server:/etc/init.d/keepalived stop and then see from the server if the virtual IP is already bound:
• Start keepalived on the master server to see if the master server can re-take over the virtual IP

3.3 Let keepalived monitor the status of Nginx

After the previous configuration, if the primary server keepalived stop service, from the server will automatically take over the VIP external services, once the primary server keepalived recovery, will re-take over the VIP. But this is not what we need, we need to be able to automatically switch when Nginx stops the service.

Keepalived Support Configuration Monitoring script, we can monitor the state of Nginx through the script, if the state is not normal, a series of operations, eventually still can not recover nginx kill keepalived, so that the server can take over the service.

• How to monitor the status of Nginx

The simplest way is to monitor the nginx process, the more reliable way is to check the Nginx port, the most reliable way is to check whether multiple URLs can get to the page.

• How to try to recover a service

If you find Nginx is not normal, restart it. Wait 3 seconds to verify again, and still fail to try again.

It is easy to write a monitoring script based on the above strategy. Here use Nmap to check the Nginx port to determine the status of Nginx, remember to install Nmap first. The monitoring scripts are as follows:

#!/bin/sh# Check nginx server Statusnginx=/usr/local/nginx/sbin/nginxport=80nmap localhost-p $PORT | grep "$PORT/tcp Open" #echo $?if [$?-ne 0];then    $NGINX-S stop    $NGINX    sleep 3    nmap localhost-p $PORT | grep "$PORT/tcp Open"    [$?-ne 0] &&/etc/init.d/keepalived STOPFI

Do not forget to set execution permissions on the script, otherwise it will not work.

Assuming the above script is placed in/opt/chk_nginx.sh, the following configuration is added in keepalived.conf:

Vrrp_script Chk_http_port {    script "/opt/chk_nginx.sh"    interval 2    weight 2}track_script {    Chk_http_ Port

Further, in order to avoid starting keepalived before the Nginx, you can start the/etc/init.d/keepalived in the first nginx:

Start () {    /usr/local/nginx/sbin/nginx    sleep 3    echo-n $ "Starting $prog:"    daemon keepalived ${ Keepalived_options}    retval=$?    echo    [$RETVAL-eq 0] && touch/var/lock/subsys/$prog}

4 What else can I do?

For simple repetitive labor, it is always easy for people to make mistakes, which is best given to the machine. For example, in this case, as a unified access server, you may often want to modify the Nginx configuration, nginx HTML files and so on. Also, make sure that each server in the cluster has the same configuration. The best practice is to manage the configuration Management Server, and if not, you can also use simple Linux file synchronization to resolve.

5 SSL Configuration

To generate a secret key under nginx/conf:

#生成RSA密钥openssl dsaparam-rand-genkey-out myrsa.key 1024# generate the CA key: (To enter a password you remember) OpenSSL gendsa-des3-out Cert.key Myrsa. key# Use this CA key to create the certificate, need the password created in the previous step OpenSSL req-new-x509-days 365-key cert.key-out cert.pem# Set the certificate to the root dedicated chmod

#生成免密码文件

OpenSSL rsa-in cert.key-out cert.key.unsecure

If you want to enable SSL, configure the following in Nginx:

# here is the relevant configuration of SSL for server {  listen 443;  server_name www.example.com; # your own domain name  root/home/www;  SSL on;  Ssl_certificate cert.perm;  #使用. unsecure file can not enter the password Ssl_certificate_key cert.key.unsecure during nginx startup    ;  Location/{  # ...  }}

Nginx combined with keepalived for high-availability load balancing