Flume adopts zookeeper management configuration

Flume supports the configuration of agents through zookeeper, but this is an experimental feature. The configuration file must be uploaded to the zookeeper first. The following agent is in the structure of the Zookeeper node tree:

-/flume
 |-/a1 [agent configuration file]
 | |/a2 [agent profile]

classes that process the configuration file:

Org.apache.flume.node.PollingZooKeeperConfigurationProvider: If zookeeper specified path has changed, retrieve the configuration file from zookeeper.
Org.apache.flume.node.StaticZooKeeperConfigurationProvider: After starting flume, the configuration file is not reloaded, even if the zookeeper configuration file is changed.
Org.apache.flume.agent.embedded.MemoryConfigurationProvider: Reads the configuration file from the store. The incoming data format is map.
Org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Timed drive reads the configuration file.

Org.apache.flume.node.AbstractZooKeeperConfigurationProvider Create zookeeper Client:

  Protected Curatorframework createclient () {return
    curatorframeworkfactory.newclient (zkconnstring,
        new Exponentialbackoffretry (1000, 1));
  

Flume uses curator as the zookeeper client, curator is a zookeeper client of Netflix's open source, and zookeeper has higher levels of abstraction than the native clients provided by curator Simplifies the development of zookeeper clients.

Curator's maven configuration:

<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactid>apache-curator </artifactId>
    <version>2.9.0</version>
</dependency>

Zookeeper also has an original eco-client, MAVEN configuration:

<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper< /artifactid>
    <version>3.4.6</version>
</dependency>

Using the original eco-client, upload profile flume-zookeeper.properties to zookeeper cluster:

	@Test public void Uploadfiletozk () throws Keeperexception, interruptedexception {String Propfilepath = "D:\\flume-zoo

		Keeper.properties ";
		Zookeeper ZK = null;
				try {ZK = new zookeeper ("10.0.1.85:2181,10.0.1.86:2181,10.0.1.87:2181", 300000, New Watcher () {//monitor all triggered events public void process (Watchedevent event) {System.out.println ("has triggered the + event.gettype () +" event.)
				");
		}
			});
		catch (IOException e) {//TODO auto-generated catch block E.printstacktrace ();
		} if (Zk.exists ("/flume", true) = = null) {zk.create ("/flume", NULL, Ids.open_acl_unsafe, createmode.persistent);
		} InputStream is = null;
		Bytearrayoutputstream bytestream = null;
		byte[] data = null;
			try {is = new FileInputStream (Propfilepath);
			ByteStream = new Bytearrayoutputstream ();
			int ch;
			while (ch = is.read ())!=-1) {bytestream.write (CH);
			data = Bytestream.tobytearray ();
		System.out.println (new String (data));
	catch (FileNotFoundException e) {		E.printstacktrace ();
		catch (IOException e) {e.printstacktrace ();
				Finally {try {bytestream.close ();
			Is.close ();
			catch (IOException e) {e.printstacktrace ();
		}//Create a directory node Stat Stat = zk.exists ("/flume/a1", true);
		if (stat = null) {zk.create ("/flume/a1", data, Ids.open_acl_unsafe, createmode.persistent);
			else {zk.delete ("/flume/a1", Stat.getversion ());
		Zk.create ("/FLUME/A1", data, Ids.open_acl_unsafe, createmode.persistent);
		@Test public void Get () throws Keeperexception, interruptedexception {zookeeper ZK = null;
				try {ZK = new zookeeper ("10.0.1.85:2181,10.0.1.86:2181,10.0.1.87:2181", 300000, New Watcher () {//monitor all triggered events public void process (Watchedevent event) {System.out.println ("has triggered the + event.gettype () +" event.)
				");
		}
			});
		catch (IOException e) {e.printstacktrace ();
	} System.out.println (New String (Zk.getdata ("/flume/a1", True, null)); }

Flume-zookeeper.properties Configuration file content:

A1.channels.c1.type = Memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity =
      
A1.sources.r1.channels = C1
A1.sources.r1.type = AVRO a1.sources.r1.bind = 0.0.0.0 a1.sources.r1.port
= 41414 

A1.sinks.k1.channel = C1
a1.sinks.k1.type = file_roll a1.sinks.k1.sink.directory
=
/logs/ A1.sinks.k1.sink.rollInterval = 0

a1.channels = C1
a1.sources = R1 a1.sinks
= K1

Avro gets the data and writes it to the file via memory channel.
When the configuration file is uploaded to zookeeper, start the flume with the following command:

$ bin/flume-ng agent–conf conf-z 10.0.1.85:2181,10.0.1.86:2181,10.0.1.87:2181-p/flume–name a1-dflume.root.logger=in Fo,console

Parameter name	Default Value	Description
Z	–	The Zookeeper connection string Hostname:port list is separated by commas.
P	/flume	The root path of the agent configuration file.

If there is an update to the configuration in zookeeper, Flume will reload the configuration via the Refreshconfiguration method of the Pollingzookeeperconfigurationprovider class:

private void Refreshconfiguration () throws IOException {
    logger.info ("Refreshing configuration from Zookeeper");
    byte[] data = null;
    Childdata childdata = Agentnodecache.getcurrentdata ();
    if (childdata!= null) {
      data = Childdata.getdata ();
    }
    Flumeconfiguration = configfrombytes (data);
    Eventbus.post (GetConfiguration ());
  }