Using Nginx+lua proxy for Hadoop HA

First, Hadoop HA of the Web page Access

After Hadoop turns on Ha, there are two master components available at the same time, where the component being used is called active, and the other as a backup is called standby, such as HDFs Namenode, YARN ResourceManager. HDFs Web pages can only be accessed through active namenode, and yarn's Web pages can only be accessed properly through active resoucemanager.

(1) HDFS ha Web Access

The normal use of Nginx Proxy_pass Proxy single Web service address is very simple (refer to the post of the simplest reverse proxy configuration), and the face of Hadoop ha such a multi-Web service address is a bit cumbersome.

(2) HDFS ha Web Proxy

Although Nginx upstream supports configuring multiple Web addresses, random Web requests are randomly forwarded to any Web address by default, and only one Web address is considered unreachable before it is blacklisted by Nginx. While both the active and standby nodes of the Hadoop ha are service-only, the Web Access is valid for at most one node at the same time, which requires Nginx to examine the web address in the upstream in more detail than to determine whether it is reachable.

Second, Nginx of the Upstream Health Check

The address validity check for upstream is called a health check. By periodically invoking the check logic to mark the web address of the upstream configuration, the unhealthy web address is temporarily blacklisted until a new Web request is forwarded to the address when the address is marked as healthy.

(1) Nginx itself to upstream health check support is not strong, do not have to check the logic of the free customization.

(2) The Open source project Nginx_upstream_check_module the Nginx upstream syntax and supports custom HTTP requests to check the health status of the Web service. But in the actual use of the process, encountered a very inconvenient place.

 upstream resourcemanagers {server  192.168 . 0.1 : 8084      192.168 . 0.2 : 8084      =30000  rise=1  Fall =3  timeout=http; Check_http_send   head/http/1.0\r\n\r\    n    2000  ;}  

Nginx_upstream_check_module uses the check command to define the basic properties of the health check, use Check_http_send to customize the HTTP request, check_http_expect_ Alive defines the expected health status of HTTP code. Using http_3xx here is the built-in matching syntax defined by the module, which represents HTTP code starting with 3. It must have been thought that this definition is not accurate to distinguish between 301, 302, 307 messages. Of course, the 3xx message should be the same type of message, it does not require such a precise distinction, but unfortunately the Hadoop2.7.2 version of the active ResourceManager and standby The ResourceManager return 302 and 307 messages respectively!

(3) The above two programs are not the perfect solution to the nginx upstream health check, the real perfect solution is openresty lua-resty-upstream-healthcheck. The openresty contains a large number of LUA libraries that can be freely expanded and customized to the functionality of Nginx. One of the Healthcheck.lua modules is used for upstream health checks.

But I hope that on the basis of Nginx only expand the function of upstream health check, instead of using openresty instead of nginx, so need to use Nginx lua-upstream-nginx-module module.

third, compile and install the extension Nginx

(1) Since Lua-upstream-nginx-module is an extension of nginx using a LUA script, the LUA interpreter must be installed. Luajit is an instant compiler for the Lua language and is more efficient.

$wgethttp//luajit.org/download/luajit-2.0.4.tar.gz$ TarZXVF luajit-2.0.4.Tar. gz$ CD Luajit-2.0.4$  Make$  Make Install$ export Luajit_lib=/usr/local/lib$ Export Luajit_inc=/usr/local/include/luajit-2.0$ RM/usr/local/lib/libluajit-5.1. so*$ CD.

The export environment variables luajit_lib and luajit_inc are used for subsequent compilation Lua-nginx-module modules. All dynamic link libraries for Libluajit are removed to ensure that subsequent compilations are statically linked, otherwise dynamic links are the default.

(2) After preparing the LUA environment, download the Nginx Lua module Lua-nginx-module, Nginx Development package Ngx_devel_kit, Nginx Upstreamlua module Lua-upstream-nginx-module, pcre Library and OpenSSL library, Nginx source code. The list of extracted files is as follows:

./lua-nginx-module-0.10. 5 . /lua-upstream-nginx-module-0.05. /nginx-1.10. 1 . /ngx_devel_kit-0.3. 0 . /openssl-openssl_1_0_1t. /pcre-8.38

Execute command to compile Nginx:

$ CD nginx-1.10.1$ ./configure--prefix=/etc/nginx--sbin-path=/usr/sbin/nginx--conf-path=/etc/nginx/nginx.conf--error-log-path=/var /log/nginx/error.log--http-log-path=/var/log/nginx/access.log--pid-path=/var/run/nginx.pid--lock-path=/var/run /nginx.lock--http-client-body-temp-path=/var/cache/nginx/client_temp--http-proxy-temp-path=/var/cache/nginx/ Proxy_temp--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_ Temp--http-scgi-temp-path=/var/cache/nginx/scgi_temp--user=root--group=root--with-http_ssl_module--with-http_ Realip_module--with-http_addition_module--with-http_sub_module--with-http_dav_module--with-http_flv_module-- With-http_mp4_module--with-http_gunzip_module--with-http_gzip_static_module--with-http_random_index_module-- With-http_secure_link_module--with-http_stub_status_module--with-http_auth_request_module--with-threads-- With-stream--with-stream_ssl_module--with-http_slice_module--with-mail--with-mail_ssl_module--with- file-aio--with-ipv6--with-http_v2_module--with-http_v2_module--with-cc-opt='-o2-g-pipe-wall-wp,-d_fortify_source=2-fexceptions-fstack-protector--param=ssp-buffer-size=4-m64-mtune= Generic'--with-pcre=. /pcre-8.38--with-openssl=. /OPENSSL-OPENSSL_1_0_1T--add-module=. /ngx_devel_kit-0.3.0--add-module=. /lua-nginx-module-0.10.5--add-module=. /lua-upstream-nginx-module-0.05$  Make&& Make Install

(3) After installation, the Nginx configuration file is/etc/nginx/nginx.conf, the executable file is/usr/sbin/nginx. To execute the nginx boot command:

$ nginx

visit http://127.0.0.1:8080 to see the Nginx homepage.

(4) Add the LUA Test link to test if the LUA module is working properly.

Location/Lua {    "Hello, world. " " ;     ' "Text/plain";                   Ngx.say (ngx.var.test);     ' ; }

Update Nginx configuration:

$ nginx-s Reload

Visit Http://127.0.0.1:8080/lua to see "Hello,world."

Four, Nginx Agent Hadoop HA

(3) Nginx agent Hadoop HA

Although the Lua-upstream-nginx-module module is installed, it is still necessary to use the Openresty Healthcheck.lua module to complete the upstream health check function.

(1) Download the latest version of the Openresty code. Execute the following command:

 Make  Make Install ls /usr/local/openresty/lualib/resty/upstream/healthcheck.lua

One of the Healthcheck.lua scripts is the Health check module we need.

(2) Configuration nginx.conf:

# upstreamupstream resourcemanagers {server192.168.0.1:8084; Server192.168.0.2:8084; KeepAlive -;} Upstream Namenodes {server192.168.0.1:50070; Server192.168.0.2:50070; KeepAlive -;} # Health Checklua_package_path"/usr/local/openresty/lualib/?. LUA;;"; lua_shared_dict Healthcheck 1m;lua_socket_log_errors off;init_worker_by_lua_block {local HC= Require"Resty.upstream.healthcheck"local OK, err=Hc.spawn_checker {SHM="Healthcheck", upstream="resourcemanagers", type="http", Http_req="get/http/1.0\r\n\r\n", Interval= -, timeout= the, Fall=3, Rise=2, Valid_statuses= {302}, concurrency=1,       }       ifNot OK ThenNgx.log (NGX. ERR,"=======> failed to spawn RM Health Checker:", err) return end local OK, err=Hc.spawn_checker {SHM="Healthcheck", upstream="Namenodes", type="http", Http_req="get/webhdfs/v1/?op=liststatus http/1.0\r\n\r\n", Interval= -, timeout= the, Fall=3, Rise=2, Valid_statuses= { $}, concurrency=1,       }       ifNot OK ThenNgx.log (NGX. ERR,"=======> failed to spawn NameNode Health Checker:", err) return end} # proxylocation/yarn/{proxy_pass http://resourcemanagers/;# some sub_filter, rewrite config}location/hdfs/{proxy_pass http://namenodes/;# Some sub_filter, rewrite config}

Update Nginx configuration:

$ nginx-s Reload

visit Http://127.0.0.1:8080/hdfs or Http://127.0.0.1:8080/yarn to see the web pages of HDFs or yarn.

V. Summary

In summary, the ability to extend Nginx with Lua is powerful and easy to customize, which is why Openresty is so powerful. Although Openresty has provided the Lua-resty-upstream-healthcheck module to complete the upstream health check function, we have expanded this feature personally on the community version of Nginx. I hope this article can help you to quickly configure the Nginx+lua environment, and easily develop their own nginx extension features.

Using Nginx+lua proxy for Hadoop HA