Storm install stenographer

Big Data data processing is commonly used in two modes: Batch Processing and streamcompute. In the open source field, the most famous component for batch processing is hadoop mapreduce, while streamcompute is storm. Storm is a distributed, fault-tolerant real-time computing system. It is currently an incubator project (http://storm.incubator.apache.org/) of Apache /). There are already many articles about the basic concepts of storm. Here, I will write down the steps for installing storm.

Detailed steps:

1. Install zookeeper3.4.5

2. Install zeromq 2.1.4 (http://download.zeromq.org/zeromq-2.1.4.tar.gz)

To build on Unix-like systems

If you have free choice, themost comfortable OS for developing with zeromq is probably ubuntu.

1. Makesure thatLibtool,Autoconf,AutomakeAre installed.

2. CheckwhetherUUID-DevPackage,UUID/E2fsprogsRpm or equivalent on your system is installed.

   Debian/Ubuntu
   Sudo apt-Get install UUID-Dev
   
   RedHat/Fedora
   Yum install UUID-devel

   Or

   Yuminstall libuuid-devel.x86_64

3. Unpackthe .tar.gz source archive.

4. Run./Configure, FollowedMake.

5. Toinstall zeromq system-wide runSudo makeinstall.

6. Onlinux, runSudoLdconfigAfter installing zeromq.

To see configuration options, run./Configure -- Help. ReadInstallFor more details.

3. Install jzmq (https://github.com/nathanmarz/jzmq)

./autogen.sh./configuremakemake install

An error occurred while executing make:

Socket. cpp: 178: 80: Error: invalidconversion from 'const jbyte * {aka const signed Char *} 'to 'jbyte * {aka signed Char *}' [-fpermissive]

Env-> setbytearrayregion (array, 0, optvallen, (const jbyte *) optval );

Solution:

In line 2 of The./src/socket. cpp file, remove "const ".

4. install Java 6

5. Install python2.6.6

6. Install unzip

7. install storm

wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip

ConfigurationConf /:

########## These must be filled in for a storm Configuration

Storm. zookeeper. servers:

-"Host1"

# Storm. zookeeper. Port: 2181

Storm. Local. dir: "/opt/XXX/storm-0.8.1/workdir"

Nimbus. HOST: "host1"

Supervisor. Slots. ports:

-6700

-6701

-6702

-6703

#

#

###### These may optionally be filled in:

#

# List of serim serializations

# Topology. kryo. register:

#-Org. mycompany. mytype

#-Org. mycompany. mytype2: org. mycompany. mytype2serializer

#

# List of M kryo decorators

# Topology. kryo. decorators:

#-Org. mycompany. mydecorator

#

# Locations of the DRPC servers

# DRPC. servers:

#-"Server1"

#-"Server2"

Ui. Port: 18091

Start:

1. Nimbus: Run "bin/stormnimbus>/dev/null 2> & 1 &" on the storm master node to start the nimbus background program and run it in the background. log is in logs/nimbus. Log

2. Supervisor: Run "bin/stormsupervisor>/dev/null 2> & 1 &" on each storm worker node to start the supervisor background program and run it in the background. log is in logs/supervisor. Log

3. UI: Run "bin/stormui>/dev/null 2> & 1 &" on the storm master node to start the UI background program and run it in the background. After the startup, you can use http: // {nimbushost}: 18091 observe the cluster's worker resource usage and the running status of topologies. Log in logs/UI. Log

Wegui:

8. Compile and run the incubator-storm Project

1. Gitclone git: // git.apache.org/incubator-storm.git

2. Change. /examples/storm-starter/src/JVM/storm/starter/wordcounttopology. java is designed to submit topology to the cluster instead of localcluster, so that topology can be seen on webgui.

3. Compile

Mvnclean install-dskiptests = true

4. Change bin/storm and set the conf_dir path correctly. Otherwise, an error may be thrown when submitting the topology.

5. Run:

Command: Bin/stormjar storm-starter-0.9.3-incubating-SNAPSHOT-jar-with-dependencies.jarstorm.starter.WordCountTopology

Summaries in the console:

Details:

The output -- randomsentencespout () of this topology can be seen in the logs/worker-6700.log continuously generated and then input to the topology, driving the entire stream computing:

6. Kill topology:

You can run the "bin/storm list" command to view the topology status.

You can run the "bin/storm kill wordcount" command to kill the topology named wordcount.

Storm install stenographer