Sqoop2 remote debugging

Sqoop2 is a CS architecture. Clients include sqoop-shell and sqoop-client. servers include sqoop-server. sqoop-Server is a Web application deployed under tomcat, consists of several servlets.

Debug sqoop-shell

Debugging sqoop-shell is to add debugging parameters when you start the sqoop client main method.

The startup method of sqoop-shell is sqoop. shclient. Find the starting point to solve the problem, find the sqoop. Sh script, and find the following section in it:

client)    #Build class path with full path to each library   for f in $CLIENT_LIB/*.jar; do     CLASSPATH="${CLASSPATH}:${BASEDIR}/$f"   done    #We need to change current directory back to original as optional user sidescript    #might be specified with relative path.   cd ${OLD_DIR}   EXEC_JAVA='java'   if [ -n "${JAVA_HOME}" ] ; then       EXEC_JAVA="${JAVA_HOME}/bin/java"    fi   ${EXEC_JAVA} -classpath ${CLASSPATH} org.apache.sqoop.shell.SqoopShell $2   ;;  *)   echo "Command is not recognized.";;

We can see that $ {exec_java}-classpath $ {classpath} Org. apache. sqoop. shell. sqoopshell $2. This line of shell script is the place where the main method of the sqoop client is started. Add:

-Xdebug -Xnoagent-Xrunjdwp:transport=dt_socket,address=8199,server=y,suspend=n

This line of code is as follows:

${EXEC_JAVA} -classpath ${CLASSPATH} -Xdebug-Xnoagent -Xrunjdwp:transport=dt_socket,address=8199,server=y,suspend=norg.apache.sqoop.shell.SqoopShell $2

Note: Put it in a line without line breaks. Server is not an IP address.

For specific meanings of parameters, refer to relevant documents. There is a series of articles on IBM dedicated to Java debugging systems, it is called "Deep Java debugging system deep Java debugging system", which is very detailed. I believe many people will fill in a blank Java knowledge.

In this way, when sqoop. Sh client is run to start sqoop shell, the output contains the following content:

Listening for transport dt_socket ataddress: 8199

In IDE, such as Eclipse, select remote debugging. In eclipse, select remote Java application. The network address and port number of the machine where the program is located. In this example, the port number is 8199.

After debugging is started on Eclipse, set the breakpoint, and then operate on sqoopshell started on the machine where sqoop is located. When it is run to the breakpoint, it stops, in eclipse, this is the same as debugging local programs.

Debug sqoop-Server

Because sqoop-Server is a Java Web application, remote debugging of sqoop-server is set to Tomcat for remote debugging. Run Tomcat's Catalina. Sh command. We can see That tomcat has provided us with the JPDA option:

Usage: catalina.sh ( commands ... )commands: debug             Start Catalinain a debugger debug -security   Debug Catalinawith a security manager jpda start        Start Catalinaunder JPDA debugger run               Start Catalinain the current window  run-security     Start in the current windowwith security manager  start             Start Catalina in a separatewindow start -security   Start in aseparate window with security manager stop              Stop Catalina,waiting up to 5 seconds for the process to end stop n            Stop Catalina,waiting up to n seconds for the process to end stop -force       Stop Catalina,wait up to 5 seconds and then use kill -KILL if still running stop n -force     Stop Catalina,wait up to n seconds and then use kill -KILL if still running version           What version oftomcat are you running?

Because the built-in Tomcat startup is controlled by the sqoop. Sh script, the startup command is as follows:

sqoop.sh server start

Check the sqoop. Sh script code and find the following parts:

   actionCmd=$2    source ${BASEDIR}/bin/sqoop-sys.sh   setup_catalina_opts     #There seems to be a bug in catalina.sh whereby catalina.sh doesn't respect    #CATALINA_OPTS when stopping the tomcat server. Consequently, we have to hackaround    #by specifying the CATALINA_OPTS properties in JAVA_OPTS variable   if [ "$actionCmd" == "stop" ]; then     export JAVA_OPTS="$JAVA_OPTS $CATALINA_OPTS"   fi     #Remove the first 2 command line arguments (server and action command(start/stop)) so we can pass    #the rest to catalina.sh script   shift   shift    $CATALINA_BIN/catalina.sh $actionCmd "[email protected]";;

From the sqoop. Sh Server start command, actioncmd is the start command. That is to say, the sqoop. Sh Server start command is the input start when the Catalina. Sh command is executed, that is, Catalina. Sh start. We want to run:

catalina.sh jpda start

Therefore, the original $ catalina_bin/Catalina. Sh $ actioncmd "[email protected]" script is directly added to JPDA. The final script is:

$ Catalina_bin/Catalina. Sh JPDA $ actioncmd "[email protected]"

When we set sqoop shell's JPDA, we add parameters at JVM startup, but from Catalina. according to the help information printed by SH, if no parameter is input, it is possible that the configuration file or script contains an environment variable:

Finally, find the following configuration in Catalina. sh:

if [ "$1" = "jpda" ] ;then  if[ -z "$JPDA_TRANSPORT" ]; then   JPDA_TRANSPORT="dt_socket"  fi  if[ -z "$JPDA_ADDRESS" ]; then   JPDA_ADDRESS="8000"  fi  if[ -z "$JPDA_SUSPEND" ]; then   JPDA_SUSPEND="n"  fi  if[ -z "$JPDA_OPTS" ]; then   JPDA_OPTS="-agentlib:jdwp=transport=$JPDA_TRANSPORT,address=$JPDA_ADDRESS,server=y,suspend=$JPDA_SUSPEND"  fi CATALINA_OPTS="$CATALINA_OPTS $JPDA_OPTS" shiftfi

This is the default configuration and can be changed by yourself.

 

After completing the preceding configuration, debug the sqoop-server project and remote Java application in eclipse, and fill in the remote JVM address and port number. In this example, the value is 8000. Click debug.

 

Sqoop2 remote debugging