Spark's straggler in-depth learning (1): How to monitor the GC of remote spark in local graphics-using Java's own JVISUALVM

I. The purpose of this articleStraggler is the hotspot of research, and there are straggler problems in spark. GC problem is one of the most important factors that lead to straggler, in order to understand the straggler problem caused by GC, we need to learn GC problem first and how to monitor the GC of Spark. GC issues are more discussed, and a series of articles is recommended for learning: to become a GC expert in Java. Ii. Tools required for this articleThe tools required in this article are simple and do not need to be downloaded.     The premise of monitoring GC is: 1, already installed on the cluster spark and can normally submit the job, 2, the local installed jdk1.6 above version. The actual tools used in this article are: Java comes with JVISUALVM. Third, configuration JVISUALVM1, remote cluster: Need simple configuration and start jstatd. The method is relatively simple, a total of two steps: I. Create a new Jstatd.all.policy file in the bin directory of the Java_home file with the following contents:

Grant CodeBase "file:${java.home}/. /lib/tools.jar " {   permission java.security.AllPermission;};

II. Execution: Jstatd-j-djava.security.policy=jstatd.all.policy-j-djava.rmi.server.hostname=yourip.

Replace the Yourip in the command with the address of node where the master of Spark is located, which is also the address that JVISUALVM needs to connect to. Make sure that the RMI and connect errors are not reported.

2, the Local host: No configuration, start JVISUALVM can.

Create a new remote host in JVISUALVM with an IP address of the previously filled address, using the JSTATD connection, and the default port of 1099. You can then see the JVM virtual machine running remotely.

Iv. Monitoring

1. Download the visual GC plugin in the plugins section of the "Tools" option in JVISUALVM, and then you can see the monitor page of the virtual machine more than one column of GC monitoring.

2. Submit an application on the cluster via spark submit, which can be monitored locally and double-click to view its information.

3, monitoring to the following:

Spark's straggler in-depth learning (1): How to monitor the GC of remote spark in local graphics-using Java's own JVISUALVM