Use Nginx + Lua to proxy Hadoop HA and nginxhadoop

Use Nginx + Lua to proxy Hadoop HA and nginxhadoop

I,Hadoop HAOfWebPage access

 

After Hadoop enables HA, there will be two Master components to provide services at the same time. The component in use is called Active, and the other is called Standby as backup, such as HDFS NameNode and YARN ResourceManager. HDFS web pages can be accessed only through Active NameNode. Similarly, YARN web pages can be accessed only through Active ResouceManager.

 

 

(1) hdfs ha Web Access

 

Normal use of Nginx proxy_pass proxy a single Web service address is very simple (refer to the simplest reverse proxy configuration in blog), and it will be a little troublesome for multiple Web Service addresses such as Hadoop HA.

 

 

(2) hdfs ha Web Proxy

 

Although Nginx upstream supports configuring multiple Web addresses, Web requests are randomly forwarded to any Web address by default. After a web address is considered inaccessible, nginx is blacklisted. The Active and Standby nodes of Hadoop HA always serve, but at the same time, only one node's Web access is valid at most, this requires Nginx to check the Web address in upstream in more detail, rather than roughly determining whether it is reachable.

 

II,NginxOfUpstreamHealth Check

 

Health Check is used to check the IP address validity of upstream. The health check logic is called regularly to mark the Web address configured by upstream. Unhealthy Web addresses are temporarily blacklisted until they are marked as healthy, to forward new Web requests to this address.

(1) Nginx does not support upstream health checks, so it cannot customize the inspection logic.

 

(2) The nginx_upstream_check_module of the open-source project expands the Nginx upstream syntax in the form of Nginx patches and supports custom HTTP requests to check the health status of Web Services. However, in actual use, it is inconvenient.

upstream resourcemanagers {    server 192.168.0.1:8084;    server 192.168.0.2:8084;    check interval=30000 rise=1 fall=3 timeout=5000 type=http;    check_http_send "HEAD / HTTP/1.0\r\n\r\n";    check_http_expect_alive http_3xx;    keepalive 2000;}

Nginx_upstream_check_module uses the check command to define the basic attributes of health check, uses check_http_send to customize HTTP requests, and check_http_expect_alive to define the expected health status HTTP code. Http_3xx is the built-in matching syntax defined by this module, indicating the HTTP code starting with 3. We have already thought that this definition method cannot precisely distinguish 301, 302, and 307 packets. Of course, under normal circumstances, the 3xx packets should be of the same type, and such a precise distinction is not required, unfortunately, Active ResourceManager and Standby ResourceManager of Hadoop2.7.2 return messages 302 and 307 respectively!

 

(3) the above two solutions are not the perfect solution for Nginx upstream health check. The perfect solution is OpenResty's lua-resty-upstream-healthcheck. OpenResty has a large number of built-in Lua libraries, allowing you to freely expand and customize Nginx functions. The healthcheck. lua module is used for upstream health check.

However, I want to only extend the upstream health check function on the basis of Nginx, instead of replacing Nginx with OpenResty. Therefore, we need to use the Nginx lua-upstream-nginx-module.

 

Iii. Compilation and installation ExtensionNginx

 

(1) because the lua-upstream-nginx-module is extended using the Lua script, the Lua interpreter must be installed. LuaJIT is a real-time compiler for Lua, which is more efficient.

$ wget http://luajit.org/download/LuaJIT-2.0.4.tar.gz$ tar zxvf LuaJIT-2.0.4.tar.gz$ cd LuaJIT-2.0.4$ make$ make install$ export LUAJIT_LIB=/usr/local/lib$ export LUAJIT_INC=/usr/local/include/luajit-2.0$ rm /usr/local/lib/libluajit-5.1.so*$ cd ..

The export environment variables LUAJIT_LIB and LUAJIT_INC are used for subsequent compilation of the lua-nginx-module. Delete all dynamic link libraries of libluajit to ensure static links during subsequent compilation; otherwise, dynamic links are used by default.

 

(2) After the Lua environment is ready, next, download the Nginx Lua module lua-nginx-module, Nginx Development Kit ngx_devel_kit, Nginx upstreamLua module lua-upstream-nginx-module, pcre Library, openssl library, and Nginx source code. The decompressed file list is as follows:

./lua-nginx-module-0.10.5./lua-upstream-nginx-module-0.05./nginx-1.10.1./ngx_devel_kit-0.3.0./openssl-OpenSSL_1_0_1t./pcre-8.38

 

Run the following command to compile Nginx:

$ cd nginx-1.10.1$ ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=root --group=root --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-pcre=../pcre-8.38 --with-openssl=../openssl-OpenSSL_1_0_1t --add-module=../ngx_devel_kit-0.3.0 --add-module=../lua-nginx-module-0.10.5 --add-module=../lua-upstream-nginx-module-0.05$ make && make install

 

(3) After installation, the Nginx configuration file is/etc/nginx. conf and the executable file is/usr/sbin/nginx. Run the Nginx startup command:

$ nginx

Visit http: // 127.0.0.1: 8080 to see the Nginx homepage.

 

(4) Add the Lua test link to test whether the Lua module works properly.

location /lua {    set $test "hello, world.";    content_by_lua '        ngx.header.content_type = "text/plain";        ngx.say(ngx.var.test);    ';}

 

Update Nginx Configuration:

$ nginx -s reload

Visit http: // 127.0.0.1: 8080/lua to see "hello, world .".

 

IV,NginxProxyHadoop HA

 

 

(3) Nginx proxy Hadoop HA

 

Although the lua-upstream-nginx-module is installed, you still need to use the healthcheck. lua module of OpenResty to complete the upstream health check function.

 

(1) download the OpenResty code of the latest version. Run the following command:

make && make installls /usr/local/openresty/lualib/resty/upstream/healthcheck.lua

The healthcheck. lua script is the required health check module.

 

(2) Configure nginx. conf:

# upstreamupstream resourcemanagers {    server 192.168.0.1:8084;    server 192.168.0.2:8084;    keepalive 2000;}upstream namenodes {    server 192.168.0.1:50070;    server 192.168.0.2:50070;    keepalive 2000;} # health checklua_package_path "/usr/local/openresty/lualib/?.lua;;";lua_shared_dict healthcheck 1m;lua_socket_log_errors off;init_worker_by_lua_block {       local hc = require "resty.upstream.healthcheck"       local ok, err = hc.spawn_checker {           shm = "healthcheck",           upstream = "resourcemanagers ",           type = "http",           http_req = "GET / HTTP/1.0\r\n\r\n",           interval = 2000,           timeout = 5000,           fall = 3,           rise = 2,           valid_statuses = {302},           concurrency = 1,       }       if not ok then           ngx.log(ngx.ERR, "=======> failed to spawn RM health checker: ", err)           return       end       local ok, err = hc.spawn_checker {           shm = "healthcheck",           upstream = "namenodes ",           type = "http",           http_req = "GET /webhdfs/v1/?op=LISTSTATUS HTTP/1.0\r\n\r\n",           interval = 2000,           timeout = 5000,           fall = 3,           rise = 2,           valid_statuses = {200},           concurrency = 1,       }       if not ok then           ngx.log(ngx.ERR, "=======> failed to spawn NameNode health checker: ", err)           return       end} # proxylocation /yarn/ {    proxy_pass http://resourcemanagers/;    # some sub_filter, rewrite config}location /hdfs/ {    proxy_pass http://namenodes/;    # some sub_filter, rewrite config}

 

Update Nginx Configuration:

$ nginx -s reload

Visit http: // 127.0.0.1: 8080/hdfs or http: // 127.0.0.1: 8080/yarn to view the HDFS or YARN Web page.

 

V. Summary

 

In summary, the use of Lua to expand Nginx is very powerful and easy to customize, which is also the reason why OpenResty is so powerful. Although OpenResty has provided the lua-resty-upstream-healthcheck module to complete the upstream health check function, we have personally expanded this function on the Community version of Nginx. I hope this article will help you quickly configure the Nginx + Lua environment and develop your own Nginx extension functions.