Http://zhangrenhua.com Blog has moved
background On the premise of installing spark, the code we write needs to be debugged remotely. Read many documents online to find each one that is right and share it today.
Modify the configuration file to modify the ${spark_home}/bin/spark-class file, locate the java_opts: Parameter description:-xdebug Enable debugging features
-XRUNJDWP enables the JDWP implementation with several sub-options:
Transport=dt_socket JPDA The transfer method between front-end and Back-end. Dt_socket represents the use of socket transmissions.
The address=8888 JVM listens for requests on port 8888, which is set to a non-conflicting port.
Server=y y indicates that the JVM being started is the debugger. If n, the JVM that starts is the debugger.
Suspend=y y indicates that the JVM that is started pauses the wait until the debugger is connected to the execution. Suspend=n, the JVM does not pause the wait.
start Spark Application This article takes a sample of my own code: first package the written code on the server, and then execute the code below. If the parameters do not understand the official documents (note: Be sure to choose Yarn-client mode run): Spark-submit--class Spark. Demo--masteryarn-client/home/hadoop/apptest/spark-demo.jar/tmp/data.txt--executor-memory 1024m
Set the IP and port for remote debugging in eclipse
A
summary of the running jar files must be consistent with the source code on eclipse.
