Probe into Spark JobServer

A) preparatory work

Installing SBT on Linux

curl https://bintray.com/sbt/rpm/rpm | sudo tee /etc/yum.repos.d/bintray-sbt-rpm.reposudo yum install sbt

根据spark版本下载Spark-jobserver

https://github.com/spark-jobserver/spark-jobserver/releases

The version of the sample download is 0.6.2 https://github.com/spark-jobserver/spark-jobserver/archive/v0.6.2.tar.gz

Installation location for sample download:/data1/local/wqq/spark-jobserver_bak

II) deployment

Next introduce Spark-jobserver

Spark-jobserver_bak The following directory structure:

Step 1:

Copy config/local.sh.template file as local.sh, reference command: CP config/local.sh.template config/local.sh

Step 2:

Configure the local.sh file, below is an important description of the contents of the file.

Note: If you are using the Package command (server_package.sh), you only need to configure the relevant configuration of spark

# Environment and deploy file
# for use with Bin/server_deploy, bin/server_package etc.
deploy_hosts= "10.207.26.250" #使用server_deploy. SH command to deploy remote machine IP or host, use server_package.sh command without configuration

App_user=root #部署远程的机器使用用户, use the server_package.sh command without configuring
app_group=root# Deploying a remote machine using a user-owned group, using the server_package.sh command without configuring

# optional SSH Key to login to deploy server
#SSH_KEY =/path/to/keyfile.pem

Install_dir=/data1/local/spark-jobserver #远程机器安装路径, use the server_package.sh command without configuring
Log_dir=/data1/local/spark-jobserver/logs #job进程日志位置
Pidfile=spark-jobserver.pid #job进程的pid文件名称
JOBSERVER_MEMORY=1G #job进程内存大小

#以下是spark的相关配置 start

spark_version=1.6.0
max_direct_memory=512m
spark_home=/data1/local/spark-1.6.1-bin-hadoop2.3
spark_conf_dir= $SPARK _home/conf

#spark的相关配置 End

# Needed for Mesos deploys
Spark_executor_uri=/home/spark/spark-1.6.0.tar.gz
# needed for YARN running outside of the cluster
# You'll need to COPY these files from your cluster to the remote machine
# Normally these is kept on the cluster in/etc/hadoop/conf
# yarn_conf_dir=/pathtoremoteconf/conf
# hadoop_conf_dir=/pathtoremoteconf/conf
#
# Also Optional:extra JVM args for Spark-submit
# export spark_submit_opts+= "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5433"
scala_version=2.10.4 # or 2.11.6

Step 3:

Package or remote Deployment

Package uses bin/server_package.sh local; Remote deployment uses bin/server_deploy.sh local (note: If you perform an error, Remind you can't find the local.sh file, you can copy the local.sh file to the corresponding path according to the error message.

After executing the command, SBT downloads the relevant jar package for a longer time.

To package a command procedure:

The red circle in the path is the path of the package placement. After the package is successful, you need to use the tar command to extract to the directory where the machine needs to be installed, the path to the example installation is/data1/local/spark-jobserver

Remote Deployment Command procedure:

You need to enter the root password. After executing the command, you can see it in the appropriate directory on the remote machine, where the example local.sh configuration is/data1/local/spark-jobserver

Step 4:

Start

The structure under the/data1/local/spark-jobserver directory is as follows:

Need to check the configuration of local.conf and settings.sh two files for problems, you can use it without problems server_ start.sh Boot JobServer will spark-jobserver.pid this file when it is started, and the content is the ID number of the process.

The 8090 port of the host can be accessed via the browser after it is properly booted, for example: 10.207.26.250:8090.

Reference Documents & Project addresses

Https://github.com/spark-jobserver/spark-jobserver

Probe into Spark JobServer