Build Spark machine learning model with Knime 1: Development environment Building

1. Knime Analytics Installation

Download the appropriate version from the official website https://www.knime.com/downloads

Unzip the downloaded installation package on the installation path https://www.knime.com/installation-0

is the Welcome page after the Knime launch

Do I need to install knime in Knime to be mutual with spark set XXX? Extension for Apache Spark. and install the Spark Job Server on the Hadoop cluster boundary node or node capable of performing spark-submit. The architecture diagram is as follows:

2, Knime? Extension for Apache Spark installation

In Knime Analytics, click File->install knime Extensions Select Knime Big Data Extensions, click Next to install.

3. SPARK JOB Server Installation

The following steps take CentOS 6.5 + CDH 5.7 as an example

3. 1 Download Spark Job server

$ wget http://download.knime.org/store/3.5/spark-job-server-0.6.2.3-KNIME_cdh-5.7.tar.gz

3. 2 Login as root or su root

3. 3 Installation

# Linkname=spark-job-server # useradd-d/opt/${linkname}/-m-r-s/bin/false spark-job-server # su-l-C "HDFs dfs-mkdir-p/user/spark-job-server; HDFs dfs-chown-r Spark-job-server/user/spark-job-server "HDFs # CP Spark-job-server-0.6.2.3-knime_cdh-5.7.tar.gz/opt # cd/opt # TAR-XVF Spark-job-server-0.6.2.3-knime_cdh-5.7.tar.gz # ln-s spark-job-server-0.6.2.3-knime_cdh-5.7 ${linkname} # chown-r Spark-job-server:spark-job-server ${linkname} spark-job-server-0.6.2.3-knime_cdh-5.7

3. 4 Boot up

# Ln-s/opt/${linkname}/spark-job-server-init.d/etc/init.d/${linkname} #chkconfig--levels 2345 ${linkname} on

3. 5 Edit environment.conf

Set Master, for example

master = "spark://ifrebdplatform1:7077"

Set default settings for Spark contexts:context-settings

3. 6 Editing Settings settings.sh

Set Spark_home, this example is correct by default, do not change

Set Log_dir, if not the default set of directories

3. 7 Edit Log4j-server.properties According to your preference

3. 8 Starting the Spark job server

/etc/init.d/${linkname} start

3. 9 Add create Spark context node test link in Knime

Right-click the Create Spark context node and click Execute Execution

Right-click the Create Spark context node and click Spark Context to view the results

To be Continued ...

Build Spark machine learning model with Knime 1: Development environment Building