10.channel

Source: Internet
Author: User

Label:

!!! 1.Memory Channel Memory ChannelsThe event is stored in memory in a queue of the specified size. ideal for scenarios where high throughput is required but failure is a loss of data. Property Description:!type – type, must be "memory" capacity Maximum number of events stored in the channel transactioncapacity The maximum number of events in each transaction keep-alive 3 time-out for Add or remove operations Bytecapacitybufferpercentage 20 Defines the percent of buffer between bytecapacity and the estimated total size of all event The s in the channel, to account for data in headers. See below. bytecapacity See description Maximum Total bytes of memory allowed as a sum of the events in the This channel. The implementation only counts the Event body, which are the reason for providing the bytecapacitybufferpercentage Configur ation parameter as well. Defaults to a computed value equal to 80% of the maximum memory available to the JVM (i.e. 80% of The-xmx value passed on the command line). Note If you have multiple memory channels on a single JVM, and they happen to hold the same physical events (i.e. if You is using a replicating channel selector from a single source) then those event sizes may be double-counted for Channe L bytecapacity purposes. Setting This value to 0 would cause this value to fall back to a hard internal limit of about GB. Case Study: See the Getting Started case2.JDBC Channelevents are persisted in a reliable database. The embedded Derby database is currently supported. This can be used if recoverability is very important. !!! 3.File Channelperformance will be low, but even if program error data is not lostProperty Description:!type – type, must be "file" checkpointdir ~/.flume/file-channel/checkpoint where the checkpoint file is stored usedualcheckpoints false Backup the checkpoint. If the is set to true, the Backupcheckpointdir must be set Backupcheckpointdir – The directory where the checkpoint is backed. This directory must isn't being the same as the data directories or the checkpoint directory datadirs ~/.flume/file-channel/data A comma-delimited list of directories to hold log files. Using multiple directories on separate disks can improve file channel efficiency. transactioncapacity 10000 The maximum size of transaction supported by the channelcheckpointinterval 30000 Amount of Time (in Millis) between checkpoints maxfilesize 2146435071 maximum size of a log file minimumrequiredspace 524288000 Minimum Required free Space (in bytes). To avoid data corruption, File Channel stops accepting take/put requests if free space drops below this value /c3>capacity 1000000 Maximum capacity of the channelkeep-alive 3 Amount of Time (in SEC) to wait for a put operation use-log-replay-v1 false expert:use old replay logic Use-fast-replay false Expert:replay without using queue checkpointonclose True Controls If a checkpoint is created when the channel is closed. Creating a checkpoint on close speeds up subsequent startup of the file channel by avoiding replay. Encryption.activekey – Key name used to encrypt new data Encryption.cipherprovider – Cipher provider type, supported Types:aesctrnopadding Encryption.keyprovider – Key provider type, supported Types:jceksfile encryption.keyProvider.keyStoreFile – Path to the KeyStore file encrpytion.keyProvider.keyStorePasswordFile – Path to the keystore password file Encryption.keyProvider.keys – List of all keys (e.g. history of the Activekey setting) encyption.keyprovider.keys.*.passwordfile – Path to the optional key password file !!! 4.Spillable Memory Channel--overflow channelsevents are stored in memory queues and disks. The Memory queue acts as the primary storage, while the disk is stored as overflow content. memory storage is managed through the embedded File channel. When the memory queue is full, subsequent events are stored in the file channel. This channel is suitable for efficient throughput during normal operation for the memory channel, while the file channel for high endurance during peak times. Improves system tolerance by reducing throughput efficiency. If the agent crashes, only events stored in the file system can be restored. This channel is in the pilot phase and is not recommended for use in production environments. Property Description:!type – type, must be "spillablememory" memorycapacity 10000 The maximum value of the event stored in memory, set this value to 0 if you want to disable the memory buffer. overflowcapacity 100000000 The maximum number of events that can be stored on the disk. Set to 0 to disable disk storage. Overflowtimeout 3 The number of seconds to wait before enabling disk overflow when memory fills up. Bytecapacitybufferpercentage defines the percent of buffer between bytecapacity and the estimated total Size of all events in the channel, to account for data in headers. See below. Bytecapacity See description Maximum bytes of memory allowed as a sum of all events in the memory queue. The implementation only counts the Event body, which are the reason for providing the bytecapacitybufferpercentage Configur ation parameter as well. Defaults to a computed value equal to 80% of the maximum memory available to the JVM (i.e. 80% of The-xmx value passed on the command line). Note If you have multiple memory channels on a single JVM, and they happen to hold the same physical events (i.e. if You is using a replicating channel selector from a single source) then those event sizes may be double-counted for Channe L bytecapacity purposes. Setting This value to 0 would cause this value to fall back to a hard internal limit of about GB. avgeventsize estimated average size of events, in bytes, going into the channel <file Channel properties> see file Channel all file channel property with the exception of ' keep-alive ' and ' capacity ' can be used. The keep-alive of file channel is managed by Spillable Memory Channel. Use ' overflowcapacity ' to set the File channel ' s capacity. 5. Custom Channelscustom channels need to implement channel interfaces themselves. The custom Channle class and its dependent classes must be placed in the class-loaded directory before Flume starts. parameter Description:Type-The full path name of the Channle class implemented by itself----------------------------------------------after-school practiceFlume getting logs from multiple netcatadding host information to events via interceptorsMultiplexing with selectordifferent IP to fan out, into different channel and sink to sendStore to the local file system stored in HDFS storage into loggeroverall structure See figure

10.channel

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

Tags Index: