View the JVM configuration and memory usage of the spark process

How to view the JVM configuration and generational memory usage of a running spark process is a common monitoring tool for online running jobs:

1, through the PS command query PID

Ps-ef  | grep 5661

You can position the PID according to the special characters in the command

2. Query the JVM parameter settings of the process using the Jinfo command

Jinfo 105007

Detailed JVM configuration information can be obtained.

Attaching to process ID 105007, please wait ... Debugger attached successfully. Server compiler detected. JVM version is 24.65-b04java System Properties:spark.local.dir =/diskb/sparktmp,/diskc/sparktmp,/diskd/sparktmp,/ Diske/sparktmp,/diskf/sparktmp,/diskg/sparktmpjava.runtime.name = Java (TM) SE Runtime Environmentjava.vm.version = 24.65-b04sun.boot.library.path =/usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64java.vendor.url =/http Java.oracle.com/java.vm.vendor = Oracle Corporationpath.separator =: file.encoding.pkg = Sun.iojava.vm.name = Java HotSpot (TM) 64-bit Server VMsun.os.patch.level = Unknownsun.java.launcher = Sun_standarduser.country = Cnuser.dir =/opt/ Bin/spark_dev_jobjava.vm.specification.name = Java Virtual machine Specificationjava.runtime.version = 1.7.0_67- b01java.awt.graphicsenv = Sun.awt.X11GraphicsEnvironmentSPARK_SUBMIT = Trueos.arch = Amd64java.endorsed.dirs =/usr/ Java/jdk1.7.0_67-cloudera/jre/lib/endorsedspark.executor.memory = 24gline.separator = Java.io.tmpdir =/tmpjava.vm.specification.vendor = Oracle Corporationos.name = Linuxspark.driver.memory = 15gspark.master = spark:// 10.130.2.220:7077sun.jnu.encoding = Utf-8java.library.path =:/opt/cloudera/parcels/cdh/lib/hadoop/lib/native:/usr /java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/libsun.nio.ch.buglevel = Java.class.version = 51.0java.specification.name = Java Platform API Specificationsun.management.compiler = HotSpot 64-bit Tiered Compilersspark.submit.deployMode = Clientspark.executor.extraJavaOptions =-xx:permsize=8m-xx:+printgcdetails-xx:+ Printgctimestampsos.version = 2.6.32-573.8.1.el6.x86_64user.home =/rootuser.timezone = PRCjava.awt.printerjob = sun.print.PSPrinterJobfile.encoding = Utf-8java.specification.version = 1.7spark.app.name = com.hexun.streaming.NewsTopNRealRankOffsetRisespark.eventLog.enabled = Trueuser.name = Rootjava.class.path =/opt/ cloudera/parcels/cdh/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/opt/modules/spark-1.6.1-bin-hadoop2.6/conf/:/opt/ modules/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar:/opt/modules/spark-1.6.1-bin-hadoop2.6/lib/ datanucleus-core-3.2.10.jar:/opt/modules/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/opt/ modules/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/etc/hadoop/conf/ Java.vm.specification.version = 1.7sun.arch.data.model = 64sun.java.command = Org.apache.spark.deploy.SparkSubmit-- Master spark://10.130.2.220:7077--conf spark.driver.memory=15g--conf spark.executor.extrajavaoptions=-xx:permsize =8m-xx:+printgcdetails-xx:+printgctimestamps--conf spark.ui.port=5661--class Com.hexun.streaming.NewsTopNRealRankOffsetRise--executor-memory 24g--total-executor-cores--jars/opt/bin/ sparkjars/kafka_2.10-0.8.2.1.jar,/opt/bin/sparkjars/spark-streaming-kafka_2.10-1.6.1.jar,/opt/bin/sparkjars/ Metrics-core-2.2.0.jar,/opt/bin/sparkjars/mysql-connector-java-5.1.26-bin.jar NewsTopNRealRankOffsetRise.jarjava.home =/usr/java/jdk1.7.0_67-cloudera/jreuser.language = Zhjava.spEcification.vendor = Oracle Corporationawt.toolkit = Sun.awt.X11.XToolkitspark.ui.port = 5661java.vm.info = Mixed Modejava.version = 1.7.0_67java.ext.dirs =/usr/java/jdk1.7.0_67-cloudera/jre/lib/ext:/usr/java/packages/lib/ Extsun.boot.class.path =/usr/java/jdk1.7.0_67-cloudera/jre/lib/resources.jar:/usr/java/jdk1.7.0_67-cloudera/jre /lib/rt.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/sunrsasign.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/ jsse.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/jce.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/charsets.jar:/ Usr/java/jdk1.7.0_67-cloudera/jre/lib/jfr.jar:/usr/java/jdk1.7.0_67-cloudera/jre/classesjava.vendor = Oracle Corporationfile.separator =/spark.cores.max = 24spark.eventlog.dir = hdfs://nameservice1/ Spark-logjava.vendor.url.bug = Http://bugreport.sun.com/bugreport/sun.io.unicode.encoding = UnicodeLittlesun.cpu.endian = Littlespark.jars = file:/opt/bin/sparkjars/kafka_2.10-0.8.2.1.jar,file:/opt/bin/ Sparkjars/spark-streaming-kafka_2.10-1.6.1.jar,file:/opt/bin/sparkjars/metrics-core-2.2.0.jar,file:/opt/bin/sparkjars/mysql-connector-java-5.1.26-bin.jar,file:/ Opt/bin/spark_dev_job/newstopnrealrankoffsetrise.jarsun.cpu.isalist = VM flags:-xms15g-xmx15g-xx:maxpermsize=256m

3. Use Jmap to view memory generation usage in the process

Jmap-heap 105007

You can get the details of how the Java process uses memory, including the use of memory in the new generation of older generations

Attaching to process ID 105007, please wait ... Debugger attached successfully. Server compiler detected. JVM version is 24.65-b04using thread-local object allocation. Parallel GC with + thread (s) Heap configuration:minheapfreeratio = 0 Maxheapfreeratio = maxheapsize = 1610 6127360 (15360.0MB) NewSize = 1310720 (1.25MB) maxnewsize = 17592186044415 MB oldsize = 5439 488 (5.1875MB) Newratio = 2 Survivorratio = 8 PermSize = 21757952 (20.75MB) MaxPermSize = 268435456 (256.0MB) g1heapregionsize = 0 (0.0MB) Heap usage:ps young Generationeden space:capacity = 4945084416 (4716. 0MB) used = 2674205152 (2550.320770263672MB) free = 2270879264 (2165.679229736328MB) 54.07804856369109% used From space:capacity = 217579520 (207.5MB) used = 37486624 (35.750030517578125MB) free = 180092896 (171.7499 6948242188MB) 17.22893036991717% usedto space:capacity = 206045184 (196.5MB) used = 0 (0.0MB)  Free = 206045184 (196.5MB) 0.0% Usedps old Generation capacity = 10737418240 (10240.0MB) used = 7431666880 (7087.389831542969MB) Free = 3305751360 (3152.6101684570312MB) 69.2127913236618% Usedps Perm Generation capacit y = 268435456 (256.0MB) used = 128212824 (122.27327728271484MB) free = 140222632 (133.72672271728516MB) 47.7 62998938560486% used

View the JVM configuration and memory usage of the spark process