Flume官方文檔翻譯——Flume 1.7.0 User Guide （unreleased version）中一些知識點

標籤：反射   ret   scheme   nat   font   avr   sch   指定   csdn   

Flume官方文檔翻譯——Flume 1.7.0 User Guide （unreleased version）（一） Flume官方文檔翻譯——Flume 1.7.0 User Guide （unreleased version）（二）Flume Properties

Property Name	Default	Description
flume.called.from.service	–	If this property is specified then the Flume agent will continue polling for the config file even if the config file is not found at the expected location. Otherwise, the Flume agent will terminate if the config doesn’t exist at the expected location. No property value is needed when setting this property (eg, just specifying -Dflume.called.from.service is enough) 如果這個屬性被指定了，那麼Flume agent會輪詢配置文檔即使在指定路徑找不到該文檔。此外，FLume agent將會結束如果配置文檔不在指定位置上。這個屬性不需要設定值（例如，只是指定-Dflume.called.from.service就足夠了）

Property: flume.called.from.service

Flume periodically polls, every 30 seconds, for changes to the specified config file. A Flume agent loads a new configuration from the config file if either an existing file is polled for the first time, or if an existing file’s modification date has changed since the last time it was polled. Renaming or moving a file does not change its modification time. When a Flume agent polls a non-existent file then one of two things happens: 1. When the agent polls a non-existent config file for the first time, then the agent behaves according to the flume.called.from.service property. If the property is set, then the agent will continue polling (always at the same period – every 30 seconds). If the property is not set, then the agent immediately terminates. ...OR... 2. When the agent polls a non-existent config file and this is not the first time the file is polled, then the agent makes no config changes for this polling period. The agent continues polling rather than terminating.

Flume每30秒周期輪詢配置文檔是否改變。如果一個文檔是第一次被輪詢到或者在上次輪詢後修改時間被改變了，那麼Flume agent會載入新的配置文檔。重新命名和移動一個文檔不會改變文檔的修改時間。當一個Flume agent輪詢一個不存在的文檔會出現以下兩種情況的一種：1. 當在指定目錄下輪詢不到設定檔時，agent會根據flume.called.from.service property這個屬性來決定他的行為。如果這個屬性設定了，那麼會以30秒為周期地進行輪詢；如果沒設定，那麼找不到就立即停止。2. 如果agent在載入過設定檔後在指定路徑輪詢不到檔案的話，那麼將不會做任何改變，然後繼續輪詢。

Log4J Appender（Log4J 日誌儲存空間）

Appends Log4j events to a flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.8.0-SNAPSHOT.jar). Required properties are in bold.

將Log4j events 添加到一個flume agent的avro source。一個用戶端想要使用這個appender必須要有 flume-ng-sdk在類路徑下（例如flume-ng-sdk-1.8.0-SNAPSHOT.jar）。必須要的屬性用黑體加粗。

Property Name	Default	Description
Hostname	–	The hostname on which a remote Flume agent is running with an avro source. 運行avro source的遠程Flumeagent的主機名稱
Port	–	The port at which the remote Flume agent’s avro source is listening. 遠程Flume agent的avro source所監聽的連接埠
UnsafeMode	false	If true, the appender will not throw exceptions on failure to send the events. 如果設為真，appender將不會在發送events失敗時拋出異常。
AvroReflectionEnabled	false	Use Avro Reflection to serialize Log4j events. (Do not use when users log strings) 使用 Avro反射來序列化 Log4j events。
AvroSchemaUrl	–	A URL from which the Avro schema can be retrieved. 一個用來恢複資料的URL，該URL是從 Avro schema來的。

Sample log4j.properties file:

#...log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname = example.comlog4j.appender.flume.Port = 41414log4j.appender.flume.UnsafeMode = true # configure a class‘s logger to output to the flume appenderlog4j.logger.org.example.MyClass = DEBUG,flume#...

 

By default each event is converted to a string by calling toString(), or by using the Log4j layout, if specified.

If the event is an instance of org.apache.avro.generic.GenericRecord, org.apache.avro.specific.SpecificRecord, or if the property AvroReflectionEnabled  is set to true then the event will be serialized using Avro serialization.

Serializing every event with its Avro schema is inefficient, so it is good practice to provide a schema URL from which the schema can be retrieved by the downstream sink, typically the HDFS sink. If AvroSchemaUrl is not specified, then the schema will be included as a Flume header.

Sample log4j.properties file configured to use Avro serialization:

每個events預設都可以通過toString()來轉換成字串，或者有指定的話可用Log4j layout。

如果events是一個org.apache.avro.generic.GenericRecord, org.apache.avro.specific.SpecificRecord類的執行個體，或者它的屬性AvroReflectionEnabled的值為true，那麼會使用Avro serialization進行序列化。

對每個event和它的Avro schema進行序列化是低效的，所以，一個好的實踐是提供一個可以從下流的sink中恢複的schemaURL，通常是HDFS sink。如果沒有指定AvroSchemaUrl的話，schema會被納入到Flume haeder。

一個使用Avro serialization的log4j屬性文檔的例子如下：

#...log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname = example.comlog4j.appender.flume.Port = 41414log4j.appender.flume.AvroReflectionEnabled = truelog4j.appender.flume.AvroSchemaUrl = hdfs://namenode/path/to/schema.avsc # configure a class‘s logger to output to the flume appenderlog4j.logger.org.example.MyClass = DEBUG,flume#...

 

Load Balancing Log4J Appender

Appends Log4j events to a list of flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.8.0-SNAPSHOT.jar). This appender supports a round-robin and random scheme for performing the load balancing. It also supports a configurable backoff timeout so that down agents are removed temporarily from the set of hosts .Required properties are in bold.

將Log4j events 添加到一個flume agent的avro source。一個用戶端想要使用這個appender必須要有 flume-ng-sdk在類路徑下（例如flume-ng-sdk-1.8.0-SNAPSHOT.jar）。這個日誌儲存空間支援一個迴圈和隨機計劃來執行負載平衡。它也支援一個可配置的後移逾時以便將掛掉的agent從主機中移除。黑體字標註的屬性是必須要的。

Property Name	Default	Description
Hosts	–	A space-separated list of host:port at which Flume (through an AvroSource) is listening for events 列出監聽events的主機列表，每個host：port用空格隔開。
Selector	ROUND_ROBIN	Selection mechanism. Must be either ROUND_ROBIN, RANDOM or custom FQDN to class that inherits from LoadBalancingSelector. 選擇機制。必須從ROUND_ROBIN,RANDOM或者繼承LoadBalancingSelector的自訂FQDN類。
MaxBackoff	–	A long value representing the maximum amount of time in milliseconds the Load balancing client will backoff from a node that has failed to consume an event. Defaults to no backoff 這個值代表以毫秒為單位的退避逾時最大值，也就是當一個節點在消費event時失效了，等待逾時時間再進行重發event。預設是沒有退避的
UnsafeMode	false	If true, the appender will not throw exceptions on failure to send the events. 如果設為真，appender將不會在發送events失敗時拋出異常。
AvroReflectionEnabled	false	Use Avro Reflection to serialize Log4j events. 使用 Avro反射來序列化 Log4j events。
AvroSchemaUrl	–	A URL from which the Avro schema can be retrieved. 一個用來恢複資料的URL，該URL是從 Avro schema來的。

Sample log4j.properties file configured using defaults:

#...log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppenderlog4j.appender.out2.Hosts = localhost:25430 localhost:25431 # configure a class‘s logger to output to the flume appenderlog4j.logger.org.example.MyClass = DEBUG,flume#...Sample log4j.properties file configured using RANDOM load balancing:#...log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppenderlog4j.appender.out2.Hosts = localhost:25430 localhost:25431log4j.appender.out2.Selector = RANDOM # configure a class‘s logger to output to the flume appenderlog4j.logger.org.example.MyClass = DEBUG,flume#...Sample log4j.properties file configured using backoff:#...log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppenderlog4j.appender.out2.Hosts = localhost:25430 localhost:25431 localhost:25432log4j.appender.out2.Selector = ROUND_ROBINlog4j.appender.out2.MaxBackoff = 30000 # configure a class‘s logger to output to the flume appenderlog4j.logger.org.example.MyClass = DEBUG,flume#...

 

Security（安全性）

The HDFS sink, HBase sink, Thrift source, Thrift sink and Kite Dataset sink all support Kerberos authentication. Please refer to the corresponding sections for configuring the Kerberos-related options.

Flume agent will authenticate to the kerberos KDC as a single principal, which will be used by different components that require kerberos authentication. The principal and keytab configured for Thrift source, Thrift sink, HDFS sink, HBase sink and DataSet sink should be the same, otherwise the component will fail to start.

HDFS sink、HBase sink、Thrift source、Thrift sink和Kite Dataset sink支援Kerberos認證。請參考配置Kerberos相關選項的章節。

當agent中的不同組件需要kerberos驗證，Flume agent會作為kerberos KDC驗證的主體。Thrift source, Thrift sink, HDFS sink, HBase sink and DataSet sink的密鑰和主體都應該相同，否則組件無法啟動。

Flume官方文檔翻譯——Flume 1.7.0 User Guide （unreleased version）中一些知識點