Big Data Platform Architecture (FLUME+KAFKA+HBASE+ELK+STORM+REDIS+MYSQL)

Last time Flume+kafka+hbase+elk:http://www.cnblogs.com/super-d2/p/5486739.html was implemented.

This time we can add storm:

storm-0.9.5 simple configuration is as follows:

Installation dependencies

wget http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gztar ZXVF jdk-8u45-linux-x64.tar.gzcd jdk-8u45-linux-/etc/profileAdd the following: Export Java_home =/home/dir/jdk1. 8 . 0_45export CLASSPATH=.: $JAVA _home/jre/lib/rt.jar: $JAVA _home/lib/dt.jar: $JAVA _home/lib/  Tools.jarexport PATH= $PATH: $JAVA _home/bin

Install Python

sudo apt-get Install python

Installing zookeeper

 wget http:// mirror.bit.edu.cn/apache/ zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz  tar -xzvf zookeeper- 3.4 . 6  .tar.gzchown -R root:root zookeeper-3.4  . 6  cd zookeeper -3.4 . 6  CP conf /zoo_sample.cfg conf/zoo.cfgsudo VI /etc/profile Add the following: Export Zookeeper_home  =/home/dir/downloads/ Zookeeper-3.4 . 6  export PATH  = $ZOOKEEPER _home/bin: $PATH 

View Code

Install storm

wget http://www.apache.org/dyn/closer.cgi/storm/apache-storm-0.9.5/apache-storm-0.9.5.tar.gz Tar zxvf apache-storm-0.9. 5 . TAR.GZCD Apache-storm-0.9. 5  /etc/Profileadds the following: Export storm_home=/home/dir/downloads/apache-storm-0.9. 5 export PATH= $STORM _home/bin: $PATH

Make environment variables effective

source /etc/profile

Modify Storm Configuration

VI conf/Storm.yaml modified as follows: Storm.zookeeper.servers:-"127.0.0.1"#     -"Server2"Storm.zookeeper.port:2181      //Zookeeper Port default is 2181Nimbus.host:"127.0.0.1"# # Storm.local.dir:"/home/dir/storm"Ui.port:8088

Start Storm

Start Zookeeper first

zkserver.sh start Nimbusstorm Nimbus>/dev/NULL 2>&1&Start Supervisorstorm Supervisor>/dev/NULL 2>&1&start the Uistorm UI>/dev/NULL 2>&1&JPS command to see29929Nimbus17754Core20714Jps30222Supervisorui can view the result localhost in the browser after startup:8088/index.html

View Code

Run WordCount Example

CD examples/storm-starter Submit task Storm jar Storm-starter-topologies-0.9. 5 . Jar storm.starter.WordCountTopology wordcounttop kill topology Command Storm kill Wordcounttop

Big Data Platform Architecture (FLUME+KAFKA+HBASE+ELK+STORM+REDIS+MYSQL)