Build Spark machine learning model with Knime 1: Development environment Building

Source: Internet
Author: User
Tags hdfs dfs

1. Knime Analytics Installation

Download the appropriate version from the official website https://www.knime.com/downloads

Unzip the downloaded installation package on the installation path https://www.knime.com/installation-0

is the Welcome page after the Knime launch

Do I need to install knime in Knime to be mutual with spark set XXX? Extension for Apache Spark. and install the Spark Job Server on the Hadoop cluster boundary node or node capable of performing spark-submit. The architecture diagram is as follows:

2, Knime? Extension for Apache Spark installation

In Knime Analytics, click File->install knime Extensions Select Knime Big Data Extensions, click Next to install.

3. SPARK JOB Server Installation

The following steps take CentOS 6.5 + CDH 5.7 as an example

3. 1 Download Spark Job server

$ wget http://download.knime.org/store/3.5/spark-job-server-0.6.2.3-KNIME_cdh-5.7.tar.gz

3. 2 Login as root or su root

3. 3 Installation

# Linkname=spark-job-server

# useradd-d/opt/${linkname}/-m-r-s/bin/false spark-job-server

# su-l-C "HDFs dfs-mkdir-p/user/spark-job-server; HDFs dfs-chown-r Spark-job-server/user/spark-job-server "HDFs

# CP Spark-job-server-0.6.2.3-knime_cdh-5.7.tar.gz/opt

# cd/opt

# TAR-XVF Spark-job-server-0.6.2.3-knime_cdh-5.7.tar.gz

# ln-s spark-job-server-0.6.2.3-knime_cdh-5.7 ${linkname}

# chown-r Spark-job-server:spark-job-server ${linkname} spark-job-server-0.6.2.3-knime_cdh-5.7

3. 4 Boot up

# Ln-s/opt/${linkname}/spark-job-server-init.d/etc/init.d/${linkname}

#chkconfig--levels 2345 ${linkname} on

3. 5 Edit environment.conf

Set Master, for example

master = "spark://ifrebdplatform1:7077"

Set default settings for Spark contexts:context-settings

3. 6 Editing Settings settings.sh

Set Spark_home, this example is correct by default, do not change

Set Log_dir, if not the default set of directories

3. 7 Edit Log4j-server.properties According to your preference

3. 8 Starting the Spark job server

/etc/init.d/${linkname} start

3. 9 Add create Spark context node test link in Knime

Right-click the Create Spark context node and click Execute Execution

Right-click the Create Spark context node and click Spark Context to view the results

To be Continued ...

Build Spark machine learning model with Knime 1: Development environment Building

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.