Zookeeper,kafka,jstorm,memcached,mysql Streaming data-processing platform deployment

Source: Internet
Author: User
Tags memcached zookeeper

A Platform Environment Introduction:

1. System Information:

Project Information

System version:

Ubuntu14.04.2 LTS \ \l
User: *****
Password: ******
Java environment: Openjdk-7-jre
Language: en_US. Utf-8,en_us:en

Disk:


Each VDA is the system disk (50G) and VDB is mounted in the/storage directory for the data disk (200G).

Hcloud15 the most DB, the second disk is 2000G
Host Range: 192.168.21.7~192.168.21.15,192.168.21.17,192.168.21.18 (11 units)
Host Name: IP address, followed by hcloud07~hcloud18

2. Service Component Distribution:

Service Name Distributing hosts
Zookeeper Hcloud12,hcloud13,hcloud14
Kafka Hcloud07,hcloud08
Jstorm Hcloud09,hcloud10,hcloud11
Memcached Hcloud13,hcloud14
Mysql Hcloud15
Web-ui Hcloud17

Two Platform Environment configuration:

Three Zookeeper configuration:

1. Install the main program and the required components:

Apt-get install-y Libzookeeper-java libzookeeper-mt2 Zookeeper Zookeeperd

2. Modify the configuration file:

2.1. Zookeeper Master configuration file

vim/etc/zookeeper/conf/zoo.cfg# #修改数据目录dataDir =/storage/lib/zookeeper# #所有zookeeper主机server. 1=hcloud12 : 2888:3888server.2=hcloud13:2888:3888server.3=hcloud14:2888:3888##

2.2 Prepare before starting

Mkdir-p/storage/lib/zookeepercd/storage/lib/zookeeperln-s/etc/zookeeper/conf/myid Myidvim myID # #填入zookeeper主机的编号 , the first one is 1, followed by analogy 1##

3. Test start and define auto start:

3.1 Test start

INVOKE-RC.D Zookeeper start# will normally start successfully, if it fails, look for the reason for the failure information, or view the log Tail-f/var/log/zookeeper/zookeeper.lo

3.2 Random Auto-start

Update-rc.dzookeeper defaults

Four Jstorm configuration:

    1. Download Unzip main package

Cd/srvwget Http://42.121.19.155/jstorm/jstorm-2.0.4-SNAPSHOT.tar.gztar zxfjstorm-2.0.4-snapshot.tar.gz

2. Modify the configuration file

Vim conf/storm.yaml# #注意缩进, add zookeeper host IP address, and Jstorm monitor the port storm.zookeeper.servers:-"192.168.21.12"-"192.168.2  1.13 "-" 192.168.21.14 "Supervisor.slots.ports:-6800-6801-6802-6803-6804-6805-6806- 6807# #Nimbus和Supervisor节点配置文件相同

3. Test start and define auto start:

3.1 Test start

The 3.1.1 is executed on the Nimbus node:

Nohup jstorm Nimbus & View $jstorm_home/logs/nimbus.log Check for errors

3.1.2 is executed on the Supervisor node

Nohup jstorm Supervisor & View $jstorm_home/logs/supervisor.log Check for errors

3.2 Random Auto-start

3.2.1Nimbus node

vim/srv/jstorm/startnimbus.sh # #加入环境变量, and Start command Exportjstorm_home=/srv/jstorm export path= $PATH: $JSTORM _home/bin nohup $JSTORM _home/bin/jstorm nimbus > $JSTORM _home/startnimbus.log &## chmod +x/srv/jstorm/startnimbus.sh vim/etc/ rc.local# #添加以下一行, the boot will automatically execute startnimbus.sh script startup jstorm/srv/jstorm/startnimbus.sh##

3.2.2Supervisor node

vim/srv/jstorm/startsupervisor.sh # #加入环境变量, and start command export jstorm_home=/srv/jstorm exportpath= $PATH: $JSTORM _home/bin Nohup$jstorm_home/bin/jstorm Supervisor > $JSTORM _home/startsupervisor.log & chmod +x/srv/jstorm/ startsupervisor.sh vim/etc/rc.local# #添加一下一行/srv/jstorm/startsupervisor.sh##

Five Kafka Configuration:

1. Download Unzip main package

Cd/srvwget Http://www.eu.apache.org/dist//kafka/0.8.2.1/kafka_2.9.2-0.8.2.1.tgztar zxf kafka_2.9.2-0.8.2.1.tgz

2. Modify the configuration file

Cdkafka_2.9.2-0.8.2.1/vim config/server.properties# #broker. id=1 #第二个节点依次类推log. retention.hours= 48zookeeper.connect=192.168.21.12:2181,192.168.21.13:2181,192.168.21.14:2181# #填入zookeeper的主机的IP

3. Test start-up and auto-start

vim/srv/kafka_2.9.2-0.8.2.1/startkafka.sh# #原理同Jstorm一样exportKAFKA_HOME =/srv/kafka_2.9.2-0.8.2.1nohup$kafka_ home/bin/kafka-server-start.sh $KAFKA _home/config/server.properties \> $KAFKA _home/startkafka.log #chmod + x/etc/init.d/startkafka.shvim/etc/rc.loca# #添加一行/srv/kafka_2.9.2-0.8.2.1/startkafka.sh##

Six Memcached configuration:

1. Install the main program

vim/etc/memcached.conf# #将以下行注释 #-L 127.0.0.1##

2. Start-Up and auto-start

Start: invoke-rc.d memcached start random auto start: Update-rc.d memcacheddefaults

Seven MySQL configuration:

1. Install the main program

Apt-get Instlal mysql-server-5.5# #安装过程需要输入mysql密码

2. Modify the configuration file

2.1. Modify the data path of the database (mysql default data path in the/var/lib/mysql directory)

INVOKE-RC.D MySQL stopmkdir-p/storage/lib/mysqlchown-rmysql:mysql/storage/libcp-pr/var/lib/mysql/storage/lib/vi m/etc/apparmor.d/usr.sbin.mysqld# #做如下修改, note the comma at the end #/var/lib/mysql/r,#/var/lib/mysql/** rwk,/storage/lib/mysql/r,/ storage/lib/mysql/** rwk,##

2.2. Modify the main provisioning file

Vim/etc/my sql/my.cnf# #做如下修改: #datadir =/var/lib/mysqldatadir =/storage/lib/mysql##

3. Start-up and self-booting

3.1. Start

/etc/init.d/apparmor restartinvoke-rc.d MySQL Start

3.2 Self-priming

UPDATE-RC.D MySQL Defaults

Eight Web-ui Configuration for Jstorm,zookeeper:

1.jstorm-web-ui

Installing the Tomcat Deployment Web-ui program

Apt-get install tomcat7–ycd/usr/share/tomcat7/mkdir-p. jstorm# #复制jstorm主机的配置文件到/usr/share/tomcat7/.jstorm/Directory Cp-f $JSTORM _home/conf/storm.yaml ~/.jstorm/cd/srvwget Http://42.121.19.155/jstorm/jstorm-2.0.4-SNAPSHOT.tar.gztar Zxfjstorm-2.0.4-snapshot.tar.gzcd/var/lib/tomcat7/webappscp/srv/jstorm/jstorm-ui-0.9.6.2.war./MV ROOT ROOT.oldln -sjstorm-ui-2.0.4-snapshot ROOT

2.zookeeper-web-ui

2.1. Download

3.jstorm integration with Apache

3.1Apache Load AJP Module

Apache2.2 above can use AJP way, simple and convenient;

Execute the following command to view the modules that Apache has loaded:

Apachectl-t-D Dump_modules

Execute the following command to load the PROXY_AJP module:

A2enmod PROXY_AJP

You can use the View command to view the modules that are turned on:

Apachectl-t-D Dump_modules

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M01/75/B3/wKioL1ZAi7_xiTwKAAAgyH70Nas173.png "title=" 9.png " alt= "Wkiol1zai7_xitwkaaagyh70nas173.png"/>

3.2 Configure the Apache virtual directory:

vim/etc/apache2/sites-enabled/js.conf##<virtualhost*:80> ServerName localhost documentroot "/VAR/LIB/TOMC at7/webapps/"Proxyrequests Off proxypreservehost on<proxy*> Order deny,allow Den Y from all to all</proxy> proxypass/jstorm Ajp://localhost:8009/proxypassreverse /jstorm ajp://localhost:8009/</virtualhost>##

3.3 Configuring Tomcat

vim/etc/tomcat7/server.xml# Uncomment the downlink <connectorport= "8009" protocol= "ajp/1.3" redirectport= "8443"/>

3.4 Reference Links

Https://confluence.atlassian.com/jira/configuring-apache-reverse-proxy-using-the-ajp-protocol-183468775.html

http://chirag.bodar.in/configure-apache-httpd-24-with-tomcat-7x-on-ubuntu-1204-1304-and-1404


This article is from the "LIJP" blog, make sure to keep this source http://jiapeng.blog.51cto.com/6706171/1711170

Zookeeper,kafka,jstorm,memcached,mysql Streaming data-processing platform deployment

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.