Nginx HA and HTTPS configuration deployment

The Nginx HA overall scheme architecture is:                                    (内网192.168.199.5)             +-----------VIP----------+                |                        |             |                        |           Master                   Backup        192.168.199.90            192.168.199.57        +----------+             +----------+        | HAProxy  |             | HAProxy  |        |nginx(SSL)|             |nginx(SSL)|        |keepalived|             |keepalived|                +----------+             +----------+             |               v        192.168.199.88/89       +----------+       | multiple |       |  NGINXs  |       +----------+             |               v     +--------+---------+     |        |         |    |        |         |    v        v         v+------+  +------+  +------+| WEB1 |  | WEB2 |  | WEB3 |+------+  +------+  +------+Each software function: * keepalived: Determine HAProxy survival, ensure ha * HAProxy: do http Load Balance * NGINX (SSL): With HAProxy placed on the same server, responsible for SSL offload * Nginx ( LB): Load balancer for app servers & Web servers Client access:      +--------+      HTTP                      :80 +----------+      | client |  --------------------------------> |          |      |        |                                    | haproxy, |      +--------+             +---------+            |  1 or 2  |     /        /     HTTPS    |  Nginx  |  HTTP  :80 | listening|    <________/    ---------> |  (SSL)  | ---------> |  ports   |                             |         |            |          |                             +---------+            +----------+ HAProxy + nginx (SSL) uses HAProxy to do HTTP load Balancer, using Nginx to do SSL offload. Test environment: * CentOS 6.4 x86_64 (Final) * Supermicro 2U4 Node * domain: L99.COMIP assignment: * lb01.l99.com 192.168.199.88 * lb01.l99. COM 192.168.199.89 * www.l99.com 192.168.199.5 (virtual IP) * 192.168.199.90 do Load Balancer (HAProxy + Nginx) installation configuration Hapro  Xyyum install libev-devel openssl-develcd /usr/local/srcwget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.24.tar.gzgit clone https://github.com/cbonte/haproxy-patches.gittar zxvf haproxy-1.4.24.tar.gz # 给haproxy 1.4.24 打 proxy协议补丁(haproxy 1.5之后才支持accpet-proxy, 由于我们要使用stud做ssl offload, 需要支持accept-proxy)cd haproxy-1.4.24patch -p1 < /usr/local/src/haproxy-patches/proxy-protocol/haproxy-1.4-proxy-protocol.patch make TARGETlinux2628 USE_EPOLL1 ARCHx86_64 && make installcp /usr/local/src/haproxy-1.4.24/haproxy /usr/sbin/cp examples/haproxy.init /etc/init.d/haproxychmod +x /etc/init.d/haproxychkconfig --add haproxychkconfig haproxy onvim /etc/haproxy/haproxy.cfgHaproxy.cfg as follows:#---------------------------------------------------------------------# Example configuration for a possible web  Application. See the# full configuration options online.## http://haproxy.1wt.eu/download/1.4/doc/configuration.txt##------------ ---------------------------------------------------------#----------------------------------------------------- ----------------# Global settings#---------------------------------------------------------------------Global # to Has these messages end up In/var/log/haproxy.log you'll # need to: # # 1) Configure Syslog to accept Network  Log events. This is do # by adding the '-r ' option to the syslogd_options in #/etc/sysconfig/syslog # 2) config Ure local2 events to go to The/var/log/haproxy.log # file. A line like the following can is added to #/etc/sysconfig/syslog # # local2.*/var/lo G/haproxy.log # # Log 127.0.0.1 local2 log 127.0.0.1 local0 log 127.0.0.1 local1 debug chroot/var/lib/haproxy pidfile/var/run/haproxy.pid maxconn 45000 # Total Max Connections. This was dependent on Ulimit user haproxy Group Haproxy daemon Nbproc 12 # depends on CPU processor cores, the test machine here is a 2 6-core Intel e5-2620 CPU, so the number of cores is a # turn on stats UNIX socket stats socket/var/lib/haproxy/stats#--------------------- ------------------------------------------------# Common defaults the ' listen ' and ' backend ' sections will# use I                    F not designated in their block#---------------------------------------------------------------------defaults mode HTTP log global balance roundrobin# Balance Leastconn op tion httplog option dontlognull option http-server-close option forwardfor Head ER x-real-ip option redispatch retries 3 timeout Http-request 10s timeout Queue          1m Timeout connect 5000ms timeout client 50000ms timeout server 50000ms ti Meout http-keep-alive 10s timeout check 10s maxconn 45000 # total Max Connections.    This was dependent on Ulimit stats enable stats Uri/stats # Real path redacted stats realm haproxy\ Statistics Stats Auth Username:password # Real credentials redacted Monitor-uri/monitor # Returns if we ' re up; Real path redactedfrontend http-in:80 reqdel x-real-ip reqadd x-forwarded-proto:\ http default_backend http-load -balancerfrontend https-in # bind 127.0.0.1:8443 accept-proxy bind 127.0.0.1:8443# reqdel x-real-ip reqadd X-F Orwarded-proto:\ HTTPS default_backend http-load-balancerbackend http-load-balancer server lb-1 192.168.199.88:80 max Conn 10000 Check Port server lb-2 192.168.199.89:80 maxconn 10000 Check Port 80Installing the Configuration Nginx (SSL)/usr/local/nginx/conf/nginx.confuser  nginx;worker_processes  12;error_log  logs/error.log crit;pid        logs/nginx.pid;worker_rlimit_nofile    30000;events {    use epoll;    worker_connections  51200;}http {    include       mime.types;    default_type  application/octet-stream;    # include common options #    include options.conf;    # include proxy settings #    include proxy.conf;    # domain config #    include l99.com/*.conf;}/usr/local/nginx/conf/l99.com/www.l99.com.confserver {Listen 443;        SSL on;        SSL_CERTIFICATE/USR/LOCAL/NGINX/CONF/L99.COM/LIFEIX-L99.CRT;        Ssl_certificate_key/usr/local/nginx/conf/l99.com/lifeix-l99.key;        SSL_CLIENT_CERTIFICATE/USR/LOCAL/NGINX/CONF/L99.COM/LIFEIX-DVROOT.CRT;        Ssl_session_timeout 5m;        Ssl_protocols SSLv2 SSLv3 TLSv1; Ssl_ciphers all:! Adh:!        Export56:rc4+rsa:+high:+medium:+low:+sslv2:+exp;    Ssl_prefer_server_ciphers on;        Default_type Text/plain;        Access_log Logs/access.www.ssl.l99.com.log Main;        Error_log Logs/error.www.ssl.l99.com.log;        server_name www.l99.com;        if ($request _uri ~ update.php) {rewrite/(. *) $ http://www.L99.com/timeline.action last;                } location/{Proxy_cache off;                Proxy_next_upstream http_502 http_504 error timeout invalid_header;                Proxy_ignore_headers Expires Cache-control;                Proxy_store off; Proxy_set_header Host $host;                Proxy_set_header X-real-ip $remote _addr;                Proxy_set_header x-forwarded-for $proxy _add_x_forwarded_for;                More_clear_headers "Cache-control";        Add_header Cache-control "NO-CACHE,MAX-AGE0";        Proxy_pass http://127.0.0.1:8443; }}Start and testservice haproxy restartservice nginx restart# 测试 HTTPSopenssl s_client -connect 192.168.199.90:443 -servername l99.com# 测试HTTPtelnet 192.168.199.90 80GET / HTTP/1.1Host: www.L99.comNginx (LB) configuration modification modified options.conf (mainly due to the need to record the source IP after using Haproxy as the agent)  log_format  main  ‘$http_x_forwarded_proto $http_x_real_ip $remote_addr $host $remote_user [$time_local] "$request" ‘                    ‘$status $body_bytes_sent "$http_referer" "$http_user_agent" ‘                    ‘$request_time $upstream_response_time $pipe "$gzip_ratio"‘;After restarting Nginx, access the cubic net log via Haproxy as follows:https 192.168.199.15 192.168.199.90 www.l99.com - [04/Oct/2013:17:02:33 +0800] "GET /skin/recharge/images/paybtn_bg.jpg HTTP/1.1" 304 0 "https://www.l99.com/Recharge_pay.action" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36" 0.007 0.006 . "-"HAProxy + keepalived/etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {   router_id LVS_DEVEL}vrrp_script chk_haproxy {   script "killall -0 haproxy"   # verify the pid existance   interval 2                    # check every 2 seconds   weight 2                      # add 2 points of prio if OK}vrrp_script chk_nginx {   script "killall -0 nginx"   # verify the pid existance   interval 2                    # check every 2 seconds   weight 2                      # add 2 points of prio if OK}vrrp_instance VI_1 {   interface eth0                # interface to monitor   state MASTER   virtual_router_id 51          # Assign one ID for this route   priority 101                  # 101 on master, 100 on backup   virtual_ipaddress {       192.168.199.5            # the virtual IP   }   track_script {       chk_haproxy       chk_nginx   }}

Nginx HA and HTTPS configuration deployment