The road to mathematics-distributed computing-storm (3)

5 , installation Zookeeper

This blog all content is original, if reproduced please indicate source http://blog.csdn.net/myhaspl/

[Email PROTECTED]:~/JZMQ-MASTER$CD.

[Email protected]:~ $wget http://apache.fayea.com/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz

[Email protected]:~ $tar-XZVF zookeeper-3.3.6.tar.gz

$CD zookeeper-3.3.6

[Email protected]:~/zookeeper-3.3.6$ cp./conf/zoo_sample.cfg./conf/zoo.cfg

[Email protected]:~/zookeeper-3.3.6$bin/zkserver.sh start

JMX enabled by default

[Email protected]:~/zookeeper-3.3.6$ sudobin/zkserver.sh Start

JMX enabled by default

Using config:/home/myhaspl/zookeeper-3.3.6/bin/. /conf/zoo.cfg

Starting zookeeper ... STARTED

[Email Protected]:~/zookeeper-3.3.6$bin/zkcli.sh-server 127.0.0.1:2181

Connecting to 127.0.0.1:2181

2014-12-17 16:22:17,187-info [Main:[email protected]]-clientenvironment:zookeeper.version=3.3.6-1366786, built on 07 /29/2012 06:22 GMT

2014-12-17 16:22:17,207-info [Main:[email protected]]-Clientenvironment:host.name=aaayun

2014-12-17 16:22:17,208-info [Main:[email protected]]-clientenvironment:java.version=1.7.0_71

2014-12-17 16:22:17,209-info [Main:[email protected]]-clientenvironment:java.vendor=oracle Corporation

2014-12-17 16:22:17,210-info [Main:[email protected]]-CLIENTENVIRONMENT:JAVA.HOME=/USR/LIB/JAVA/JRE

2014-12-17 16:22:17,212-info [Main:[email protected]]-clientenvironment:java.class.path=/home/myhaspl/ zookeeper-3.3.6/bin/. /build/classes:/home/myhaspl/zookeeper-3.3.6/bin/. /build/lib/*.jar:/home/myhaspl/zookeeper-3.3.6/bin/. /zookeeper-3.3.6.jar:/home/myhaspl/zookeeper-3.3.6/bin/. /lib/log4j-1.2.15.jar:/home/myhaspl/zookeeper-3.3.6/bin/. /lib/jline-0.9.94.jar:/home/myhaspl/zookeeper-3.3.6/bin/. /src/java/lib/*.jar:/home/myhaspl/zookeeper-3.3.6/bin/. /conf:.:/ Usr/lib/java/lib:/usr/lib/java/jre/lib

2014-12-17 16:22:17,213-info [Main:[email protected]]-clientenvironment:java.library.path=/usr/java/packages/lib/ Amd64:/usr/lib64:/lib64:/lib:/usr/lib

2014-12-17 16:22:17,276-info [Main:[email protected]]-clientenvironment:java.io.tmpdir=/tmp

2014-12-17 16:22:17,277-info [Main:[email protected]]-clientenvironment:java.compiler=<na>

2014-12-17 16:22:17,278-info [Main:[email protected]]-Clientenvironment:os.name=linux

2014-12-17 16:22:17,278-info [Main:[email protected]]-CLIENTENVIRONMENT:OS.ARCH=AMD64

2014-12-17 16:22:17,279-info [Main:[email protected]]-clientenvironment:os.version=3.2.0-53-generic

2014-12-17 16:22:17,279-info [Main:[email protected]]-CLIENTENVIRONMENT:USER.NAME=MYHASPL

2014-12-17 16:22:17,280-info [Main:[email protected]]-CLIENTENVIRONMENT:USER.HOME=/HOME/MYHASPL

2014-12-17 16:22:17,281-info [Main:[email protected]]-clientenvironment:user.dir=/home/myhaspl/zookeeper-3.3.6

2014-12-17 16:22:17,286-info [Main:[email protected]]-initiating clientconnection, connectstring=127.0.0.1:2181 sess Ionti[email PROTECTED]6C0DC01

2014-12-17 16:22:17,408-info [Main-sendthread (): [email protected]]-Opening socketconnection to server/127.0.0.1:2181

Welcome to zookeeper!

2014-12-17 16:22:17,796-info [Main-sendthread (localhost:2181): [email protected]]-SocketConnection established to localhost/127.0.0.1:2181, initiating session

JLine support is enabled

2014-12-17 16:22:18,204-info [Main-sendthread (localhost:2181): [email protected]]-sessionestablishment complete on Server localhost/127.0.0.1:2181, SessionID =0x14a575462220000, negotiated timeout = 30000

Watcher::

Watchedevent state:syncconnected Type:Nonepath:null

[zk:127.0.0.1:2181 (CONNECTED) 0] LS/

[Zookeeper]

[Zk:127.0.0.1:2181 (CONNECTED) 1] Quit

6. Storm Installation

[Email protected]:~/zookeeper-3.3.6$ CD.

[Email protected]:~$ wget http://mirrors.cnnic.cn/apache/storm/apache-storm-0.9.3/apache-storm-0.9.3.tar.gz

[Email protected]:~$ tar-xzvf apache-storm-0.9.3.tar.gz

Start Storm

[Email protected]:~/apache-storm-0.9.3$ sudo./bin/storm nimbus&

[1] 25871

[Email protected]:~/apache-storm-0.9.3$ sudo./bin/storm supervisor&

[2] 25882

Edit Parameter configuration

[email protected]: ~/apache-storm-0.9.3$

[Email protected]:~/apache-storm-0.9.3$ cdconf

[Email Protected]:~/apache-storm-0.9.3/conf$ls

The default configuration is as follows:

# Licensed to the Apache softwarefoundation (ASF) under one

# or more contributor licenseagreements. See the NOTICE file

# Distributed with the Additionalinformation

# regarding copyright ownership. The ASF licenses this file

# to you under the Apache License, Version2.0 (the

# "License"); Usethis file except in compliance

# with the License. Obtain a copy of the License at

#

#http://www.apache.org/licenses/license-2.0

#

# unless required by applicable law Oragreed to writing, software

# Distributed under the License isdistributed on a "as is" BASIS,

# without warranties or CONDITIONS of anykind, either express or implied.

# See the License for the specific languagegoverning permissions and

# Limitations under the License.

########### These all has the default Valuesas shown

########### Additional Configuration Goesinto Storm.yaml

Java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib"

# # # storm.* Configs is generalconfigurations

# The local dir is a where jars is kept

Storm.local.dir: "Storm-local"

Storm.zookeeper.servers:

-"localhost"

storm.zookeeper.port:2181

Storm.zookeeper.root: "/storm"

storm.zookeeper.session.timeout:20000

storm.zookeeper.connection.timeout:15000

Storm.zookeeper.retry.times:5

storm.zookeeper.retry.interval:1000

storm.zookeeper.retry.intervalceiling.millis:30000

Storm.cluster.mode: "Distributed" # can be distributed or local

Storm.local.mode.zmq:false

Storm.thrift.transport: "Backtype.storm.security.auth.SimpleTransportPlugin"

Storm.messaging.transport: "Backtype.storm.messaging.netty.Context"

# # # nimbus.* Configs is for the master

Nimbus.host: "localhost"

nimbus.thrift.port:6627

nimbus.thrift.max_buffer_size:1048576

Nimbus.childopts: "-xmx1024m"

Nimbus.task.timeout.secs:30

Nimbus.supervisor.timeout.secs:60

Nimbus.monitor.freq.secs:10

nimbus.cleanup.inbox.freq.secs:600

nimbus.inbox.jar.expiration.secs:3600

nimbus.task.launch.secs:120

Nimbus.reassign:true

nimbus.file.copy.expiration.secs:600

Nimbus.topology.validator: "Backtype.storm.nimbus.DefaultTopologyValidator"

# # # ui.* Configs is for the master

ui.port:8080

Ui.childopts: "-xmx768m"

logviewer.port:8000

Logviewer.childopts: "-xmx128m"

Logviewer.appender.name: "A1"

drpc.port:3772

Drpc.worker.threads:64

drpc.queue.size:128

drpc.invocations.port:3773

drpc.request.timeout.secs:600

Drpc.childopts: "-xmx768m"

Transactional.zookeeper.root: "/transactional"

Transactional.zookeeper.servers:null

Transactional.zookeeper.port:null

# # # supervisor.* configs is for nodesupervisors

# Define The amount of workers that can berun on this machine. Each worker was assigned a port to use for communication

Supervisor.slots.ports:

-6700

-6701

-6702

-6703

Supervisor.childopts: "-xmx256m"

#how Long supervisor'll wait to ensurethat a worker process is started

supervisor.worker.start.timeout.secs:120

#how long between heartbeats Untilsupervisor considers that worker dead and tries to restart it

Supervisor.worker.timeout.secs:30

#how frequently the supervisor checks onthe status of the processes it ' s monitoring and restarts if necessary

Supervisor.monitor.frequency.secs:3

#how frequently the Supervisor Heartbeatsto the cluster state (for Nimbus)

Supervisor.heartbeat.frequency.secs:5

Supervisor.enable:true

# # # worker.* Configs is for task workers

Worker.childopts: "-xmx768m"

Worker.heartbeat.frequency.secs:1

Task.heartbeat.frequency.secs:3

Task.refresh.poll.secs:10

Zmq.threads:1

zmq.linger.millis:5000

zmq.hwm:0

Storm.messaging.netty.server_worker_threads:1

Storm.messaging.netty.client_worker_threads:1

STORM.MESSAGING.NETTY.BUFFER_SIZE:5242880#5MB Buffer

Storm.messaging.netty.max_retries:30

storm.messaging.netty.max_wait_ms:1000

storm.messaging.netty.min_wait_ms:100

# # # topology.* configs is for specificexecuting storms

Topology.enable.message.timeouts:true

Topology.debug:false

Topology.optimize:true

Topology.workers:1

Topology.acker.executors:null

Topology.tasks:null

# Maximum amount of time a message has tocomplete before it ' s considered failed

Topology.message.timeout.secs:30

Topology.skip.missing.kryo.registrations:false

Topology.max.task.parallelism:null

Topology.max.spout.pending:null

Topology.state.synchronization.timeout.secs:60

topology.stats.sample.rate:0.05

Topology.builtin.metrics.bucket.size.secs:60

Topology.fall.back.on.java.serialization:true

Topology.worker.childopts:null

Topology.executor.receive.buffer.size:1024#batched

Topology.executor.send.buffer.size:1024#individual messages

Topology.receiver.buffer.size:8 # Settingit Too high causes a lot of problems (heartbeat thread gets starved, THROUGHPUTP Lummets)

topology.transfer.buffer.size:1024 #batched

Topology.tick.tuple.freq.secs:null

Topology.worker.shared.thread.pool.size:4

Topology.disruptor.wait.strategy: "Com.lmax.disruptor.BlockingWaitStrategy"

Topology.spout.wait.strategy: "Backtype.storm.spout.SleepSpoutWaitStrategy"

Topology.sleep.spout.wait.strategy.time.ms:1

Topology.error.throttle.interval.secs:10

Topology.max.error.report.per.interval:5

Topology.kryo.factory: "Backtype.storm.serialization.DefaultKryoFactory"

Topology.tuple.serializer: "Backtype.storm.serialization.types.ListDelegateSerializer"

topology.trident.batch.emit.interval.millis:500

Dev.zookeeper.path: "/tmp/dev-storm-zookeeper"

The storm release version has a Conf/storm.yaml file under the Extract directory to configure Storm. The default configuration can be viewed in Defaults.yaml. The configuration option in Conf/storm.yaml overrides the default configuration in Defaults.yaml.

Start the background UI

[Email protected]:~/apache-storm-0.9.3$ sudo ./bin/stormui&

[3] 25893

The road to mathematics-distributed computing-storm (3)