Zookeeper,kafka,jstorm,memcached,mysql Streaming data-processing platform deployment

A Platform Environment Introduction:

1. System Information:

Project	Information
System version:	Ubuntu14.04.2 LTS \ \l
User:	*****
Password:	******
Java environment:	Openjdk-7-jre
Language:	en_US. Utf-8,en_us:en
Disk:	Each VDA is the system disk (50G) and VDB is mounted in the/storage directory for the data disk (200G). Hcloud15 the most DB, the second disk is 2000G
Host Range:	192.168.21.7~192.168.21.15,192.168.21.17,192.168.21.18 (11 units)
Host Name:	IP address, followed by hcloud07~hcloud18

2. Service Component Distribution:

Service Name	Distributing hosts
Zookeeper	Hcloud12,hcloud13,hcloud14
Kafka	Hcloud07,hcloud08
Jstorm	Hcloud09,hcloud10,hcloud11
Memcached	Hcloud13,hcloud14
Mysql	Hcloud15
Web-ui	Hcloud17

Two Platform Environment configuration:

Three Zookeeper configuration:

1. Install the main program and the required components:

Apt-get install-y Libzookeeper-java libzookeeper-mt2 Zookeeper Zookeeperd

2. Modify the configuration file:

2.1. Zookeeper Master configuration file

vim/etc/zookeeper/conf/zoo.cfg# #修改数据目录dataDir =/storage/lib/zookeeper# #所有zookeeper主机server. 1=hcloud12 : 2888:3888server.2=hcloud13:2888:3888server.3=hcloud14:2888:3888##

2.2 Prepare before starting

Mkdir-p/storage/lib/zookeepercd/storage/lib/zookeeperln-s/etc/zookeeper/conf/myid Myidvim myID # #填入zookeeper主机的编号 , the first one is 1, followed by analogy 1##

3. Test start and define auto start:

3.1 Test start

INVOKE-RC.D Zookeeper start# will normally start successfully, if it fails, look for the reason for the failure information, or view the log Tail-f/var/log/zookeeper/zookeeper.lo

3.2 Random Auto-start

Update-rc.dzookeeper defaults

Four Jstorm configuration:

1. Download Unzip main package

Cd/srvwget Http://42.121.19.155/jstorm/jstorm-2.0.4-SNAPSHOT.tar.gztar zxfjstorm-2.0.4-snapshot.tar.gz

2. Modify the configuration file

Vim conf/storm.yaml# #注意缩进, add zookeeper host IP address, and Jstorm monitor the port storm.zookeeper.servers:-"192.168.21.12"-"192.168.2  1.13 "-" 192.168.21.14 "Supervisor.slots.ports:-6800-6801-6802-6803-6804-6805-6806- 6807# #Nimbus和Supervisor节点配置文件相同

3. Test start and define auto start:

3.1 Test start

The 3.1.1 is executed on the Nimbus node:

Nohup jstorm Nimbus & View $jstorm_home/logs/nimbus.log Check for errors

3.1.2 is executed on the Supervisor node

Nohup jstorm Supervisor & View $jstorm_home/logs/supervisor.log Check for errors

3.2 Random Auto-start

3.2.1Nimbus node

vim/srv/jstorm/startnimbus.sh # #加入环境变量, and Start command Exportjstorm_home=/srv/jstorm export path= $PATH: $JSTORM _home/bin nohup $JSTORM _home/bin/jstorm nimbus > $JSTORM _home/startnimbus.log &## chmod +x/srv/jstorm/startnimbus.sh vim/etc/ rc.local# #添加以下一行, the boot will automatically execute startnimbus.sh script startup jstorm/srv/jstorm/startnimbus.sh##

3.2.2Supervisor node

vim/srv/jstorm/startsupervisor.sh # #加入环境变量, and start command export jstorm_home=/srv/jstorm exportpath= $PATH: $JSTORM _home/bin Nohup$jstorm_home/bin/jstorm Supervisor > $JSTORM _home/startsupervisor.log & chmod +x/srv/jstorm/ startsupervisor.sh vim/etc/rc.local# #添加一下一行/srv/jstorm/startsupervisor.sh##

Five Kafka Configuration:

1. Download Unzip main package

Cd/srvwget Http://www.eu.apache.org/dist//kafka/0.8.2.1/kafka_2.9.2-0.8.2.1.tgztar zxf kafka_2.9.2-0.8.2.1.tgz

2. Modify the configuration file

Cdkafka_2.9.2-0.8.2.1/vim config/server.properties# #broker. id=1 #第二个节点依次类推log. retention.hours= 48zookeeper.connect=192.168.21.12:2181,192.168.21.13:2181,192.168.21.14:2181# #填入zookeeper的主机的IP

3. Test start-up and auto-start

vim/srv/kafka_2.9.2-0.8.2.1/startkafka.sh# #原理同Jstorm一样exportKAFKA_HOME =/srv/kafka_2.9.2-0.8.2.1nohup$kafka_ home/bin/kafka-server-start.sh $KAFKA _home/config/server.properties \> $KAFKA _home/startkafka.log #chmod + x/etc/init.d/startkafka.shvim/etc/rc.loca# #添加一行/srv/kafka_2.9.2-0.8.2.1/startkafka.sh##

Six Memcached configuration:

1. Install the main program

vim/etc/memcached.conf# #将以下行注释 #-L 127.0.0.1##

2. Start-Up and auto-start

Start: invoke-rc.d memcached start random auto start: Update-rc.d memcacheddefaults

Seven MySQL configuration:

1. Install the main program

Apt-get Instlal mysql-server-5.5# #安装过程需要输入mysql密码

2. Modify the configuration file

2.1. Modify the data path of the database (mysql default data path in the/var/lib/mysql directory)

INVOKE-RC.D MySQL stopmkdir-p/storage/lib/mysqlchown-rmysql:mysql/storage/libcp-pr/var/lib/mysql/storage/lib/vi m/etc/apparmor.d/usr.sbin.mysqld# #做如下修改, note the comma at the end #/var/lib/mysql/r,#/var/lib/mysql/** rwk,/storage/lib/mysql/r,/ storage/lib/mysql/** rwk,##

2.2. Modify the main provisioning file

Vim/etc/my sql/my.cnf# #做如下修改: #datadir =/var/lib/mysqldatadir =/storage/lib/mysql##

3. Start-up and self-booting

3.1. Start

/etc/init.d/apparmor restartinvoke-rc.d MySQL Start

3.2 Self-priming

UPDATE-RC.D MySQL Defaults

Eight Web-ui Configuration for Jstorm,zookeeper:

1.jstorm-web-ui

Installing the Tomcat Deployment Web-ui program

Apt-get install tomcat7–ycd/usr/share/tomcat7/mkdir-p. jstorm# #复制jstorm主机的配置文件到/usr/share/tomcat7/.jstorm/Directory Cp-f $JSTORM _home/conf/storm.yaml ~/.jstorm/cd/srvwget Http://42.121.19.155/jstorm/jstorm-2.0.4-SNAPSHOT.tar.gztar Zxfjstorm-2.0.4-snapshot.tar.gzcd/var/lib/tomcat7/webappscp/srv/jstorm/jstorm-ui-0.9.6.2.war./MV ROOT ROOT.oldln -sjstorm-ui-2.0.4-snapshot ROOT

2.zookeeper-web-ui

2.1. Download

3.jstorm integration with Apache

3.1Apache Load AJP Module

Apache2.2 above can use AJP way, simple and convenient;

Execute the following command to view the modules that Apache has loaded:

Apachectl-t-D Dump_modules

Execute the following command to load the PROXY_AJP module:

A2enmod PROXY_AJP

You can use the View command to view the modules that are turned on:

Apachectl-t-D Dump_modules

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M01/75/B3/wKioL1ZAi7_xiTwKAAAgyH70Nas173.png "title=" 9.png " alt= "Wkiol1zai7_xitwkaaagyh70nas173.png"/>

3.2 Configure the Apache virtual directory:

vim/etc/apache2/sites-enabled/js.conf##<virtualhost*:80> ServerName localhost documentroot "/VAR/LIB/TOMC at7/webapps/"Proxyrequests Off proxypreservehost on<proxy*> Order deny,allow Den Y from all to all</proxy> proxypass/jstorm Ajp://localhost:8009/proxypassreverse /jstorm ajp://localhost:8009/</virtualhost>##

3.3 Configuring Tomcat

vim/etc/tomcat7/server.xml# Uncomment the downlink <connectorport= "8009" protocol= "ajp/1.3" redirectport= "8443"/>

3.4 Reference Links

Https://confluence.atlassian.com/jira/configuring-apache-reverse-proxy-using-the-ajp-protocol-183468775.html

http://chirag.bodar.in/configure-apache-httpd-24-with-tomcat-7x-on-ubuntu-1204-1304-and-1404

This article is from the "LIJP" blog, make sure to keep this source http://jiapeng.blog.51cto.com/6706171/1711170

Zookeeper,kafka,jstorm,memcached,mysql Streaming data-processing platform deployment