A Platform Environment Introduction:
1. System Information:
Project |
Information |
System version: |
Ubuntu14.04.2 LTS \ \l |
User: |
***** |
Password: |
****** |
Java environment: |
Openjdk-7-jre |
Language: |
en_US. Utf-8,en_us:en |
Disk:
|
Each VDA is the system disk (50G) and VDB is mounted in the/storage directory for the data disk (200G). Hcloud15 the most DB, the second disk is 2000G |
Host Range: |
192.168.21.7~192.168.21.15,192.168.21.17,192.168.21.18 (11 units) |
Host Name: |
IP address, followed by hcloud07~hcloud18 |
2. Service Component Distribution:
Service Name |
Distributing hosts |
Zookeeper |
Hcloud12,hcloud13,hcloud14 |
Kafka |
Hcloud07,hcloud08 |
Jstorm |
Hcloud09,hcloud10,hcloud11 |
Memcached |
Hcloud13,hcloud14 |
Mysql |
Hcloud15 |
Web-ui |
Hcloud17 |
Two Platform Environment configuration:
Three Zookeeper configuration:
1. Install the main program and the required components:
Apt-get install-y Libzookeeper-java libzookeeper-mt2 Zookeeper Zookeeperd
2. Modify the configuration file:
2.1. Zookeeper Master configuration file
vim/etc/zookeeper/conf/zoo.cfg# #修改数据目录dataDir =/storage/lib/zookeeper# #所有zookeeper主机server. 1=hcloud12 : 2888:3888server.2=hcloud13:2888:3888server.3=hcloud14:2888:3888##
2.2 Prepare before starting
Mkdir-p/storage/lib/zookeepercd/storage/lib/zookeeperln-s/etc/zookeeper/conf/myid Myidvim myID # #填入zookeeper主机的编号 , the first one is 1, followed by analogy 1##
3. Test start and define auto start:
3.1 Test start
INVOKE-RC.D Zookeeper start# will normally start successfully, if it fails, look for the reason for the failure information, or view the log Tail-f/var/log/zookeeper/zookeeper.lo
3.2 Random Auto-start
Update-rc.dzookeeper defaults
Four Jstorm configuration:
Download Unzip main package
Cd/srvwget Http://42.121.19.155/jstorm/jstorm-2.0.4-SNAPSHOT.tar.gztar zxfjstorm-2.0.4-snapshot.tar.gz
2. Modify the configuration file
Vim conf/storm.yaml# #注意缩进, add zookeeper host IP address, and Jstorm monitor the port storm.zookeeper.servers:-"192.168.21.12"-"192.168.2 1.13 "-" 192.168.21.14 "Supervisor.slots.ports:-6800-6801-6802-6803-6804-6805-6806- 6807# #Nimbus和Supervisor节点配置文件相同
3. Test start and define auto start:
3.1 Test start
The 3.1.1 is executed on the Nimbus node:
Nohup jstorm Nimbus & View $jstorm_home/logs/nimbus.log Check for errors
3.1.2 is executed on the Supervisor node
Nohup jstorm Supervisor & View $jstorm_home/logs/supervisor.log Check for errors
3.2 Random Auto-start
3.2.1Nimbus node
vim/srv/jstorm/startnimbus.sh # #加入环境变量, and Start command Exportjstorm_home=/srv/jstorm export path= $PATH: $JSTORM _home/bin nohup $JSTORM _home/bin/jstorm nimbus > $JSTORM _home/startnimbus.log &## chmod +x/srv/jstorm/startnimbus.sh vim/etc/ rc.local# #添加以下一行, the boot will automatically execute startnimbus.sh script startup jstorm/srv/jstorm/startnimbus.sh##
3.2.2Supervisor node
vim/srv/jstorm/startsupervisor.sh # #加入环境变量, and start command export jstorm_home=/srv/jstorm exportpath= $PATH: $JSTORM _home/bin Nohup$jstorm_home/bin/jstorm Supervisor > $JSTORM _home/startsupervisor.log & chmod +x/srv/jstorm/ startsupervisor.sh vim/etc/rc.local# #添加一下一行/srv/jstorm/startsupervisor.sh##
Five Kafka Configuration:
1. Download Unzip main package
Cd/srvwget Http://www.eu.apache.org/dist//kafka/0.8.2.1/kafka_2.9.2-0.8.2.1.tgztar zxf kafka_2.9.2-0.8.2.1.tgz
2. Modify the configuration file
Cdkafka_2.9.2-0.8.2.1/vim config/server.properties# #broker. id=1 #第二个节点依次类推log. retention.hours= 48zookeeper.connect=192.168.21.12:2181,192.168.21.13:2181,192.168.21.14:2181# #填入zookeeper的主机的IP
3. Test start-up and auto-start
vim/srv/kafka_2.9.2-0.8.2.1/startkafka.sh# #原理同Jstorm一样exportKAFKA_HOME =/srv/kafka_2.9.2-0.8.2.1nohup$kafka_ home/bin/kafka-server-start.sh $KAFKA _home/config/server.properties \> $KAFKA _home/startkafka.log #chmod + x/etc/init.d/startkafka.shvim/etc/rc.loca# #添加一行/srv/kafka_2.9.2-0.8.2.1/startkafka.sh##
Six Memcached configuration:
1. Install the main program
vim/etc/memcached.conf# #将以下行注释 #-L 127.0.0.1##
2. Start-Up and auto-start
Start: invoke-rc.d memcached start random auto start: Update-rc.d memcacheddefaults
Seven MySQL configuration:
1. Install the main program
Apt-get Instlal mysql-server-5.5# #安装过程需要输入mysql密码
2. Modify the configuration file
2.1. Modify the data path of the database (mysql default data path in the/var/lib/mysql directory)
INVOKE-RC.D MySQL stopmkdir-p/storage/lib/mysqlchown-rmysql:mysql/storage/libcp-pr/var/lib/mysql/storage/lib/vi m/etc/apparmor.d/usr.sbin.mysqld# #做如下修改, note the comma at the end #/var/lib/mysql/r,#/var/lib/mysql/** rwk,/storage/lib/mysql/r,/ storage/lib/mysql/** rwk,##
2.2. Modify the main provisioning file
Vim/etc/my sql/my.cnf# #做如下修改: #datadir =/var/lib/mysqldatadir =/storage/lib/mysql##
3. Start-up and self-booting
3.1. Start
/etc/init.d/apparmor restartinvoke-rc.d MySQL Start
3.2 Self-priming
UPDATE-RC.D MySQL Defaults
Eight Web-ui Configuration for Jstorm,zookeeper:
1.jstorm-web-ui
Installing the Tomcat Deployment Web-ui program
Apt-get install tomcat7–ycd/usr/share/tomcat7/mkdir-p. jstorm# #复制jstorm主机的配置文件到/usr/share/tomcat7/.jstorm/Directory Cp-f $JSTORM _home/conf/storm.yaml ~/.jstorm/cd/srvwget Http://42.121.19.155/jstorm/jstorm-2.0.4-SNAPSHOT.tar.gztar Zxfjstorm-2.0.4-snapshot.tar.gzcd/var/lib/tomcat7/webappscp/srv/jstorm/jstorm-ui-0.9.6.2.war./MV ROOT ROOT.oldln -sjstorm-ui-2.0.4-snapshot ROOT
2.zookeeper-web-ui
2.1. Download
3.jstorm integration with Apache
3.1Apache Load AJP Module
Apache2.2 above can use AJP way, simple and convenient;
Execute the following command to view the modules that Apache has loaded:
Apachectl-t-D Dump_modules
Execute the following command to load the PROXY_AJP module:
A2enmod PROXY_AJP
You can use the View command to view the modules that are turned on:
Apachectl-t-D Dump_modules
650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M01/75/B3/wKioL1ZAi7_xiTwKAAAgyH70Nas173.png "title=" 9.png " alt= "Wkiol1zai7_xitwkaaagyh70nas173.png"/>
3.2 Configure the Apache virtual directory:
vim/etc/apache2/sites-enabled/js.conf##<virtualhost*:80> ServerName localhost documentroot "/VAR/LIB/TOMC at7/webapps/"Proxyrequests Off proxypreservehost on<proxy*> Order deny,allow Den Y from all to all</proxy> proxypass/jstorm Ajp://localhost:8009/proxypassreverse /jstorm ajp://localhost:8009/</virtualhost>##
3.3 Configuring Tomcat
vim/etc/tomcat7/server.xml# Uncomment the downlink <connectorport= "8009" protocol= "ajp/1.3" redirectport= "8443"/>
3.4 Reference Links
Https://confluence.atlassian.com/jira/configuring-apache-reverse-proxy-using-the-ajp-protocol-183468775.html
http://chirag.bodar.in/configure-apache-httpd-24-with-tomcat-7x-on-ubuntu-1204-1304-and-1404
This article is from the "LIJP" blog, make sure to keep this source http://jiapeng.blog.51cto.com/6706171/1711170
Zookeeper,kafka,jstorm,memcached,mysql Streaming data-processing platform deployment