Spark's straggler in-depth learning (1): How to monitor the GC of remote spark in local graphics-using Java's own JVISUALVM

Source: Internet
Author: User

I. The purpose of this articleStraggler is the hotspot of research, and there are straggler problems in spark. GC problem is one of the most important factors that lead to straggler, in order to understand the straggler problem caused by GC, we need to learn GC problem first and how to monitor the GC of Spark. GC issues are more discussed, and a series of articles is recommended for learning: to become a GC expert in Java. Ii. Tools required for this articleThe tools required in this article are simple and do not need to be downloaded.     The premise of monitoring GC is: 1, already installed on the cluster spark and can normally submit the job, 2, the local installed jdk1.6 above version. The actual tools used in this article are: Java comes with JVISUALVM. Third, configuration JVISUALVM1, remote cluster: Need simple configuration and start jstatd. The method is relatively simple, a total of two steps: I. Create a new Jstatd.all.policy file in the bin directory of the Java_home file with the following contents:
Grant CodeBase "file:${java.home}/. /lib/tools.jar " {   permission java.security.AllPermission;};

II. Execution: Jstatd-j-djava.security.policy=jstatd.all.policy-j-djava.rmi.server.hostname=yourip.

Replace the Yourip in the command with the address of node where the master of Spark is located, which is also the address that JVISUALVM needs to connect to. Make sure that the RMI and connect errors are not reported.

2, the Local host: No configuration, start JVISUALVM can.

Create a new remote host in JVISUALVM with an IP address of the previously filled address, using the JSTATD connection, and the default port of 1099. You can then see the JVM virtual machine running remotely.

Iv. Monitoring

1. Download the visual GC plugin in the plugins section of the "Tools" option in JVISUALVM, and then you can see the monitor page of the virtual machine more than one column of GC monitoring.

2. Submit an application on the cluster via spark submit, which can be monitored locally and double-click to view its information.

3, monitoring to the following:

Spark's straggler in-depth learning (1): How to monitor the GC of remote spark in local graphics-using Java's own JVISUALVM

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.