Access to SparkUI based on Nginx Reverse proxy

: This article mainly introduces the access to SparkUI based on Nginx Reverse proxy. if you are interested in the PHP Tutorial, refer to it. Access to SparkUI based on Nginx Reverse proxy

Scenarios and solutions

When the spark cluster is deployed, only the Master has a public IP address and the Worker machine does not have a public IP address. to access the SparkUI of the Run on the Master machine, when you need to view the logs on the Worker machine, there will be a problem of inaccessible. At this time, a solution is to use Nginx + SSH Tunnel port redirection to direct all requests from different ports on different machines to port 80 on the Master machine, and then use Nginx for reverse proxy forwarding.

Step decomposition

1. Deploy the Spark cluster and run SparkUI on the Master node.

2. Deploy Nginx on the Master machine, modify the $ NGINX_HOME/conf/nginx. conf file, and add the reverse proxy logic of the Work and Master machines, for example:

   worker_processes  1;error_log  logs/error.log;error_log  logs/error.log  notice;error_log  logs/error.log  info;pid        logs/nginx.pid;events {    worker_connections  1024;}http {    include         mime.types;    default_type    pplication/octet-stream;    sendfile        off;    keepalive_timeout  20;    gzip  on;    proxy_intercept_errors  off;    upstream server_spark_master {        server 127.0.0.1:8080;    }    upstream server_jyvhyguguiu-slave-0 {        server 10.110.1.1:8081;    }    upstream server_jyvhyguguiu-slave-1 {        server 10.110.1.2:8081;    }    server {        listen 80;        server_name spark_master;        location / {                proxy_pass http://server_spark_master;        }    }    server {        listen 80;        server_name jyvhyguguiu-slave-0;        location / {                proxy_pass http://server_jyvhyguguiu-slave-0;        }    }    server {        listen 80;        server_name jyvhyguguiu-slave-1;        location / {                proxy_pass http://server_jyvhyguguiu-slave-1;        }    }       }

Uses the machine name ing principle of nginx for reverse proxy.

3. bind the Master machine and Worker machine to 127.0.0.1 by modifying the hosts file.

4. use the port forwarding principle supported by SSH to forward all the ports used by SparkUI to port 80 of the Nginx listening service remotely. For example, SparkUI uses port 8081 of the Worker machine, port 8080 of the Master machine, and Port 80 of the Master machine, then, all the three local ports (because the Master and Worker machines are all bound to the lo Nic) are forwarded to the remote port 80. The command is as follows:

ssh -N -f -L 8081:127.0.0.1:80 username@ipssh -N -f -L 8080:127.0.0.1:80 username@ipssh -N -f -L 80:127.0.0.1:80 username@ip

5. access port 127.0.0.1: 8080 to open the SparkUI of the remote machine and access the logs of any Worker machine.

PS: In fact, SSH port forwarding can also use the Master machine as the proxy to forward requests to the Worker machine. The network path is consistent with the reverse proxy of Nginx, however, SSH port forwarding can only be performed at layer 4, that is, the transport layer, rather than Layer 7 forwarding like Nginx (capable of identifying the machine name and forwarding according to the machine name ), therefore, the same effect cannot be achieved only through SSH port forwarding.

The above introduces the access to SparkUI based on Nginx Reverse proxy, including some content, and hope to be helpful to friends who are interested in PHP tutorials.