Storm Document (8)----configuration file description

Reprint Please specify source: http://blog.csdn.net/beitiandijun/article/details/41547569

Source Address: http://storm.apache.org/documentation/Configuration.html

Storm is a rich configure option to adjust the behavior of Nibus, supervisor, and runtime topologies. Some configuration options are system configuration, such as the topology base configuration, modifying the options for one topology may affect all topologies, and some options only involve the configuration of each topology itself, which can be modified as needed.

Each configuration option has its default value in the Defaults.yaml file in the Storm code library. You can override the default configuration by defining the Classpath Storm.yaml file under Nimbus and supervisor. Finally, you can define the topology-specific configuration so that you can submit the configuration file with the Stormsubmitter class when you commit the topology. However, the topology-specific configuration can only overwrite configuration options with the prefix "Topology".

Starting with Storm 0.7.0, you can cover every bolt or each spout's own unique configuration options. These configurations are as follows:

1, "Topology.debug"

2, "topology.max.spout.pending"

3, "Topology.max.task.parallelism"

4, "Topology.kryo.register": this option and other options work a little differently, because the sequence is available for all components in topology. More information can be viewed in serialization.

The Java API allows you to develop specific configuration options for your components in two ways:

1. Internal mode: Overwrite getcomponentconfiguration in any spout or bolts and return to component-specific configuration map.

2. External mode: The Setspout method and the Setbolt method in the Topologybuilder class return objects with Addconfiguration and Addconfigurations methods, which can be used to override configuration options for the component.

The recommended order for configuring option values is: Defaults.yaml < Storm.yaml < topology specific configuration <internal component specific Configura tion < external component specificconfiguration

Information:

L Config: A list of all configuration options and a Help class to create a specific topology configuration

L Defaults.yaml: Default value for all configurations

Configuring storm Clusters: Explains how to create and configure a storm cluster

L Run topologies on a production cluster: Lists the configurations that are useful for topologies running on the cluster

L Local mode: Lists the configurations that are useful for using local mode

Below is the Defaults.yaml content:

# Licensed to the Apache software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed and this work for additional information# regarding copyright ownership. The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License");  You are not a use of this file except in compliance# with the License. Obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## unless required by applicable law O  R agreed to writing, software# distributed under the License are distributed on a "as is" basis,# without warranties OR CONDITIONS of any KIND, either express or implied.# see the License for the specific language governing permissions and# Limitations under the license.########### these all has default values as shown########### Additional configuration goes into Storm.yamljava.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib" # # # storm.* Configs is General configurations# The local dir is wherE Jars is keptstorm.local.dir: "Storm-local" storm.zookeeper.servers:-"localhost" storm.zookeeper.port: 2181storm.zookeeper.root: "/storm" Storm.zookeeper.session.timeout:20000storm.zookeeper.connection.timeout: 15000storm.zookeeper.retry.times:5storm.zookeeper.retry.interval: 1000storm.zookeeper.retry.intervalceiling.millis:30000storm.zookeeper.auth.user: Nullstorm.zookeeper.auth.password:nullstorm.cluster.mode: "Distributed" # can be distributed or Localstorm.local.mode.zmq:falsestorm.thrift.transport: "Backtype.storm.security.auth.SimpleTransportPlugin" Storm.principal.tolocal: "Backtype.storm.security.auth.DefaultPrincipalToLocal" Storm.group.mapping.service: " Backtype.storm.security.auth.ShellBasedGroupsMapping "Storm.messaging.transport:" Backtype.storm.messaging.netty.Context "Storm.nimbus.retry.times:5storm.nimbus.retry.interval.millis: 2000storm.nimbus.retry.intervalceiling.millis:60000storm.auth.simple-white-list.users: [] Storm.auth.simple-acl.users: []storm.auth.simple-acl.users.commands: []storm.auth.simple-acl.admins: []storm.meta.serialization.delegate:] Backtype.storm.serialization.DefaultSerializationDelegate "# # nimbus.* Configs is for the masternimbus.host:" localhost "nimbus.thrift.port:6627nimbus.thrift.threads:64nimbus.thrift.max_buffer_size:1048576nimbus.childopts : "-xmx1024m" Nimbus.task.timeout.secs:30nimbus.supervisor.timeout.secs:60nimbus.monitor.freq.secs: 10nimbus.cleanup.inbox.freq.secs:600nimbus.inbox.jar.expiration.secs:3600nimbus.task.launch.secs: 120nimbus.reassign:truenimbus.file.copy.expiration.secs:600nimbus.topology.validator: " Backtype.storm.nimbus.DefaultTopologyValidator "nimbus.credential.renewers.freq.secs:600### ui.* configs is for the Masterui.port:8080ui.childopts: "-xmx768m" Ui.actions.enabled:trueui.filter:nullui.filter.params:nullui.users: Nullui.header.buffer.bytes:4096ui.http.creds.plugin: Backtype.storm.security.auth.defaulthttpcredentialspluginlogviewer.port:8000logviewer.childopts: "-Xmx128m" LoGviewer.cleanup.age.mins:10080logviewer.appender.name: "A1" Logs.users:nulldrpc.port:3772drpc.worker.threads: 64drpc.max_buffer_size:1048576drpc.queue.size:128drpc.invocations.port:3773drpc.invocations.threads: 64drpc.request.timeout.secs:600drpc.childopts: "-xmx768m" Drpc.http.port:3774drpc.https.port:- 1drpc.https.keystore.password: "" Drpc.https.keystore.type: "JKS" Drpc.http.creds.plugin: Backtype.storm.security.auth.DefaultHttpCredentialsPlugindrpc.authorizer.acl.filename: "Drpc-auth-acl.yaml" Drpc.authorizer.acl.strict:falsetransactional.zookeeper.root: "/transactional" Transactional.zookeeper.servers:  nulltransactional.zookeeper.port:null### supervisor.* configs is for node supervisors# Define the amount of workers Can is run on the This machine. Each worker was assigned a port to use for Communicationsupervisor.slots.ports:-6700-6701-6702-6703supe Rvisor.childopts: "-xmx256m" supervisor.run.worker.as.user:false#how long supervisor would wait to ensure thaT a worker process is startedsupervisor.worker.start.timeout.secs:120#how long between heartbeats until supervisor Consid  ERs that worker dead and tries to restart Itsupervisor.worker.timeout.secs:30#how frequently the supervisor checks on the Status of the processes it ' s monitoring and restarts if necessarysupervisor.monitor.frequency.secs:3#how frequently the Supervisor heartbeats to the cluster state (for Nimbus) supervisor.heartbeat.frequency.secs:5supervisor.enable: Truesupervisor.supervisors: []supervisor.supervisors.commands: []### worker.* configs is for task Workersworker.childopts: "-xmx768m" worker.gc.childopts: "" worker.heartbeat.frequency.secs:1# control how many worker Receiver threads We need per Workertopology.worker.receiver.thread.count:1task.heartbeat.frequency.secs: 3TASK.REFRESH.POLL.SECS:10TASK.CREDENTIALS.POLL.SECS:30ZMQ.THREADS:1ZMQ.LINGER.MILLIS:5000ZMQ.HWM: 0storm.messaging.netty.server_worker_threads:1storm.messaging.netty.client_worker_threads:1storm.messaging.netty.buffer_size:5242880 #5MB buffer# Since nimbus.task.launch.secs and Supervisor.worker.start.timeout.secs is workers should also wait at least this long before giving up on Connec Ting to the other Worker.storm.messaging.netty.max_retries:300storm.messaging.netty.max_wait_ms: 1000storm.messaging.netty.min_wait_ms:100# If The Netty messaging layer is busy (Netty internal buffer not writable), the Netty client would try to batch message as + as possible up to the size of storm.messaging.netty.transfer.batch.size byt ES, otherwise it'll try to flush message as soon as possible to reduce latency.storm.messaging.netty.transfer.batch.size : 262144# We Check with this interval, whether the Netty channel is writable and try to write pending messages if it I  s.storm.messaging.netty.flush.check.interval.ms:10# by default, the Netty SASL authentication are set to False. Users can override and set it true for a specific topology.storm.messaging.netty.authentication:false# default number of seconds Group mapping service would cache user Groupstorm.group.mapping.service.cache.duratio n.secs:120### topology.* configs is for specific executing stormstopology.enable.message.timeouts:truetopology.debug : falsetopology.workers:1topology.acker.executors:nulltopology.tasks:null# maximum amount of time a message have to comp Lete before it ' s considered failedtopology.message.timeout.secs:30topology.multilang.serializer: " Backtype.storm.multilang.JsonSerializer "Topology.skip.missing.kryo.registrations: Falsetopology.max.task.parallelism:nulltopology.max.spout.pending: Nulltopology.state.synchronization.timeout.secs:60topology.stats.sample.rate: 0.05topology.builtin.metrics.bucket.size.secs:60topology.fall.back.on.java.serialization: truetopology.worker.childopts:nulltopology.executor.receive.buffer.size:1024 # batchedtopology.executor.send.buffer.size:1024 #individual Messagestopology.receiver.buffer.size:8 # Setting it too High causes a lot of Problems (heartbeat thread gets starved, throughput plummets) topology.transfer.buffer.size:1024 # Batchedtopology.tick . Tuple.freq.secs:nulltopology.worker.shared.thread.pool.size:4topology.disruptor.wait.strategy: " Com.lmax.disruptor.BlockingWaitStrategy "Topology.spout.wait.strategy:" Backtype.storm.spout.SleepSpoutWaitStrategy "topology.sleep.spout.wait.strategy.time.ms: 1topology.error.throttle.interval.secs:10topology.max.error.report.per.interval:5topology.kryo.factory: " Backtype.storm.serialization.DefaultKryoFactory "Topology.tuple.serializer:" Backtype.storm.serialization.types.ListDelegateSerializer "Topology.trident.batch.emit.interval.millis: 500topology.testing.always.try.serialize:falsetopology.classpath:nulltopology.environment: Nulldev.zookeeper.path: "/tmp/dev-storm-zookeeper"

Storm Document (8)----configuration file description