How to install and configure Apache Samza on Linux

How to install and configure Apache Samza on Linux

Samza is a distributed stream processing framework (streaming processing). It implements real-time stream Data processing Based on Kafka message queues. (To be precise, samza uses kafka in a modular form, so it can be structured in other message queue frameworks, but the starting point and default implementation are based on kafka)

Apache Kafka is mainly used to control message sending.

Apache Hadoop YARN provides error information, isolation processors, security, and resource management.

This article describes how to install Samza on a 32-bit Ubuntu 14.04 system.

Installation preparation:

To install and configure Apache-Samza, you need the following:

JDK 1.7
Maven2
Kafka
Yarn
Zookeeper
# Apt-get install curl gem

Download and set the JDK path:

We need to install JDK and set its environment variables.

# Cd/usr/java
 
# Wget -- no-cookies -- no-check-certificate -- header "Cookie: gpw_e24 = http % 3A % 2F % 2Fwww.Oracle.com % 2F; using lelicense = accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-i586.tar.gz"
 
# Tar xzf jdk-7u79-linux-i586.tar.gz

Decompress and set the JAVA_HOME path
# Tar-zxvf jdk-7u79-linux-i586.tar.gz
# JAVA_HOME =/usr/java/jdk1.7.0 _ 79
# Export JAVA_HOME
# PATH = $ JAVA_HOME/bin: $ PATH
# Export PATH

Add the above ~ /. Bashrc and/etc/bashrc files

Install Maven2:

Next download and install maven

# Wget https://launchpad.net /~ Bneijt/+ archive/ubuntu/ppa/+ build/2139203/+ files/maven3_3.0.1-0 ~ Ppa2_all.deb

# Dpkg-I maven3_3.0.1-0 ~ Ppa2_all.deb

Check maven version

# Mvn3-version

Apache Maven 3.0.1 (r1038046; 16:28:32 + 0530)
Java version: 1.7.0 _ 79
Java home:/usr/java/jdk1.7.0 _ 79/jre
Default locale: en_IN, platform encoding: UTF-8
OS name: "linux" version: "3.8.0-29-generic" arch: "i386" Family: "unix"

Install Hello-Samza:

In the/usr/local folder

# Cd/usr/local

Copy hello-samza in,

# Git clone git: // git.apache.org/samza-hello-samza.git hello-samza

This project contains a "grid" script with the hello-samza variable. With this, you can do everything you need. You can use it to install Kafka, Yarn, and Zookeeper.

Run the following command,

# Cd/usr/local/hello-samza

Root @ dev:/usr/local/hello-samza # bin/grid install kafka

EXECUTING: install kafka
Downloading kafka_2.10-0.8.2.1.tgz...
% Total % Received % Xferd Average Speed Time Current
Dload Upload Total Spent Left Speed
15 15.4 M 15 2406 k 0 304 k 0 0:00:51 0:00:07 443 k

Root @ dev:/usr/local/hello-samza # bin/grid install yarn

EXECUTING: install yarn
Downloading hadoop-2.6.1.tar.gz...
% Total % Received % Xferd Average Speed Time Current
Dload Upload Total Spent Left Speed
77 187 M 77 145 M 0 0 239 k 0 0:13:23 0:10:22 204 k

Root @ dev:/usr/local/hello-samza # bin/grid install zookeeper

EXECUTING: install zookeeper
Downloading zookeeper-3.4.3.tar.gz...
% Total % Received % Xferd Average Speed Time Current
Dload Upload Total Spent Left Speed
8 15.4 M 8 1324 k 0 212 k 0 0:01:14 0:00:06 266 k

Now you will find that all the packages are in the "deploy" folder under the root directory of hello-samza.

Root @ dev:/usr/local/hello-samza # cd deploy
Root @ dev:/usr/local/hello-samza/deploy # ls

Kafka yarn zookeeper

Run the bin/grid bootstrap command

Root @ dev:/usr/local/hello-samza # bin/grid bootstrap

Download http://repo1.maven.org/maven2/org/fusesource/scalate/scalate-util_2.10/1.6.1/scalate-util_2.10-1.6.1.jar
: Samza-yarn_2.10: processResources
: Samza-yarn_2.10: classes
: Samza-yarn_2.10: lesscss.
....
....
BUILD SUCCESSFUL

Total time: 20 mins 32.855 secs
/Usr/local/hello-samza
EXECUTING: install zookeeper
Using previolet usly downloaded file/root/. samza/download/zookeeper-3.4.3.tar.gz
EXECUTING: install yarn
Using previolet usly downloaded file/root/. samza/download/hadoop-2.6.1.tar.gz
EXECUTING: install kafka
Using previolet usly downloaded file/root/. samza/download/kafka_2.10-0.8.2.1.tgz
EXECUTING: start zookeeper
JMX enabled by default
Using config:/usr/local/hello-samza/deploy/zookeeper/bin/../conf/zoo. cfg
Starting zookeeper... STARTED
EXECUTING: start yarn
Starting resourcemanager, logging to/usr/local/hello-samza/deploy/yarn/logs/yarn-root-resourcemanager-dev.out
Starting nodemanager, logging to/usr/local/hello-samza/deploy/yarn/logs/yarn-root-nodemanager-dev.out
EXECUTING: start kafka
 

After the preceding grid is executed, you can verify that YARN is installed and running. Visit the URL http: // localhost: 8088. The yarn ui is displayed.

Build a Samza work package:

You need to build this package. YARN uses this package to execute the grid.

Note: For example, if you build the latest version of the hello-samza project, run the following command first.

Root @ dev:/usr/local/hello-samza #./gradlew publishToMavenLocal

You can use these commands in the hello-samza project:

Root @ dev:/usr/local/hello-samza # mvn clean package
Root @ dev:/usr/local/hello-samza # mkdir-p deploy/samza
Root @ dev:/usr/local/hello-samza # tar-xvf./target/hello-samza-0.10.0-dist.tar.gz-C deploy/samza

Run the Samza task:

After completing the build Samza package, you can use the t run-job.sh script on the grid to complete some tasks.

Root @ dev:/usr/local/hello-samza # deploy/samza/bin/run-job.sh -- config-factory = org. apache. samza. config. factories. propertiesConfigFactory -- config-path = file: // $ PWD/deploy/samza/config/wikipedia-feed.properties
 

The above task will unedit the implementation feedback from Wikipedia and put the edits in a thelinuxfaq-raw topic.

After running this topic for a few minutes, you can check the last update of Kafka:

Root @ dev:/usr/local/hello-samza # deploy/kafka/bin/kafka-console-consumer.sh -- zookeeper localhost: 2181 -- topic thelinuxfaq-raw

Visit the yarn ui again (http: // localhost: 8088). You will see that Samza is running normally, rather than an error message!

Disable Samza:

Once everything is done, you can use the grid script to close all related servers.
Root @ dev:/usr/local/hello-samza # bin/grid stop all

Output example:
EXECUTING: stop all
EXECUTING: stop kafka
EXECUTING: stop yarn
Stopping resourcemanager
Stopping nodemanager
EXECUTING: stop zookeeper
JMX enabled by default
Using config:/usr/local/hello-samza/deploy/zookeeper/bin/../conf/zoo. cfg
Stopping zookeeper... STOPPED

Start Samza:

Yes. You can use the grid script to start all services,

Root @ dev:/usr/local/hello-samza # bin/grid start all

Output example:
EXECUTING: start all
EXECUTING: start zookeeper
JMX enabled by default
Using config:/usr/local/hello-samza/deploy/zookeeper/bin/../conf/zoo. cfg
Starting zookeeper... STARTED
EXECUTING: start yarn
....
EXECUTING: start kafka

This article permanently updates the link address: