[Apache flume series] flume-ng failover and load balance tests and precautions

I haven't written a blog for a long time. We have recently studied storm, flume, and Kafka. Today, I will write down the scenarios and conclusions for testing flume failover and load balance;

The test environment contains five configuration files, that is, five agents.

A main configuration file, that is, the configuration file (flume-sink.properties) for configuring the Failover and load balance relationships, which is in the following scenarios

It will change, so I will not list it here, and it will be stated in specific scenarios;

The other four configuration files are similar:

#Name the compents on this agenta2.sources = r1a2.sinks = k1a2.channels = c1#Describe/configure the sourcea2.sources.r1.type = avroa2.sources.r1.channels = c1a2.sources.r1.bind = 192.168.220.159a2.sources.r1.port = 44411#Describe the sinka2.sinks.k1.type = loggera2.sinks.k1.channel = c1#Use a channel which buffers events in memorya2.channels.c1.type = memorya2.channels.c1.capacity = 1000a2.channels.c1.transactionCapacity = 100

You only need to modify the agent name.

Scenario 1: --- idea (flume failover and load balance can be applied simultaneously, with both load and fault tolerance considerations, making the environment more secure)

Configuration file (flume-sink.properties ):

#Name the compents on this agenta1.sources = r1a1.sinks = k1 k2 k3a1.channels = c1#Describe the sinkgroupsa1.sinkgroups = g1 g2a1.sinkgroups.g1.sinks = k1 k2a1.sinkgroups.g1.processor.type = failovera1.sinkgroups.g1.processor.priority.k1 = 10a1.sinkgroups.g1.processor.priority.k2 = 5a1.sinkgroups.g1.processor.maxpenalty = 10000a1.sinkgroups.g2.sinks = k1 k3a1.sinkgroups.g2.processor.type = load_balancea1.sinkgroups.g2.processor.backoff = truea1.sinkgroups.g2.processor.selector = round_robin#Describe/config the sourcea1.sources.r1.type = syslogtcpa1.sources.r1.port = 5140a1.sources.r1.host = localhosta1.sources.r1.channels = c1#Describe the sinka1.sinks.k1.type = avroa1.sinks.k1.channel = c1a1.sinks.k1.hostname = 192.168.220.159a1.sinks.k1.port = 44411a1.sinks.k2.type = avroa1.sinks.k2.channel = c1a1.sinks.k2.hostname = 192.168.220.159a1.sinks.k2.port = 44422a1.sinks.k4.type = avroa1.sinks.k4.channel = c1

Test results:

It is found that K1, K2, and K3 can receive data, but here K2 has the lowest priority and should not have obtained data unless K1 fails.

I tested the test before this test, but I forgot to configure the K3 statement and reported many errors to me. I thought sinkgroups sinks could not be shared,

Then I ran to the community and sent an email asking if sinkgroups sinks could not be shared. As a result, a buddy told me that. Now I know that my friends and I are both wrong. Haha

Community Email: http://mail-archives.apache.org/mod_mbox/flume-user/201408.mbox/browser

In fact, I have not configured one thing above. You can check the configuration file above. K1 is fault tolerant, but K3 is not. Therefore, we need to add the following K3 Fault Tolerance node K4.

This is what @ tells me. I understand it, haha;

#Describe the sinkgroupsa1.sinkgroups = g1 g2 g3a1.sinkgroups.g1.sinks = k1 k2a1.sinkgroups.g1.processor.type = failovera1.sinkgroups.g1.processor.priority.k1 = 10a1.sinkgroups.g1.processor.priority.k2 = 5a1.sinkgroups.g1.processor.maxpenalty = 10000a1.sinkgroups.g2.sinks = k1 k3a1.sinkgroups.g2.processor.type = load_balancea1.sinkgroups.g2.processor.backoff = truea1.sinkgroups.g2.processor.selector = round_robina1.sinkgroups.g3.sinks = k3 k4a1.sinkgroups.g3.processor.type = failovera1.sinkgroups.g3.processor.priority.k3 = 10a1.sinkgroups.g3.processor.priority.k4 = 5a1.sinkgroups.g3.processor.maxpenalty = 10000

Add G3 and K4, and then send the data;

Good. Four of them are actually received.

Next, let's cut down K1 and see what the result is;

As a result, K2, K3, and K4 receive data, which is a little strange. K4 is fault-tolerant and should not receive data. K2 receives the data;

Start K1 again. Then let's take a look at K3.

The results are similar to the above; I can only say that the test results are a bit strange; haha

Scenario 2: --- idea (Failover and load balance separate different sinks)

Configuration file (flume-sink.properties ):

#Name the compents on this agenta1.sources = r1a1.sinks = k1 k2 k3 k4a1.channels = c1#Describe the sinkgroupsa1.sinkgroups = g1 g2a1.sinkgroups.g1.sinks = k1 k2a1.sinkgroups.g1.processor.type = failovera1.sinkgroups.g1.processor.priority.k1 = 10a1.sinkgroups.g1.processor.priority.k2 = 5a1.sinkgroups.g1.processor.maxpenalty = 10000a1.sinkgroups.g2.sinks = k3 k4a1.sinkgroups.g2.processor.type = load_balancea1.sinkgroups.g2.processor.backoff = truea1.sinkgroups.g2.processor.selector = round_robin#Describe/config the sourcea1.sources.r1.type = syslogtcpa1.sources.r1.port = 5140a1.sources.r1.host = localhosta1.sources.r1.channels = c1#Describe the sinka1.sinks.k1.type = avroa1.sinks.k1.channel = c1a1.sinks.k1.hostname = 192.168.220.159a1.sinks.k1.port = 44411a1.sinks.k2.type = avroa1.sinks.k2.channel = c1a1.sinks.k2.hostname = 192.168.220.159a1.sinks.k2.port = 44422a1.sinks.k3.type = avroa1.sinks.k3.channel = c1a1.sinks.k3.hostname = 192.168.220.159a1.sinks.k3.port = 44433a1.sinks.k4.type = avroa1.sinks.k4.channel = c1a1.sinks.k4.hostname = 192.168.220.159a1.sinks.k4.port = 44444#Use a channel which butters events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100

Test results:

Here we can see that K1, K3, and K4 can receive data, which is still normal. This is equivalent to the fault tolerance of K2, where K1, K3, and K4 share the data delivered by Server Load balancer.

Next we will disconnect K1 and test it to see what the situation is.

The main agent reports an error because K1 is disconnected.

Let's look at the reception.

We can see that K2, K3, and K4 are loaded here. After K1 fails, K2, K3, and K4 share data together.

The test result is normal. K1 and K4 can receive data. After K3 is disconnected, K1 and K4 are loaded.

Scenario 3: --- idea (use the failover configuration separately, multiple high-level sinks and one low-level sink)

Configuration file (flume-sink.properties ):

#Name the compents on this agenta1.sources = r1a1.sinks = k1 k2 k3 k4a1.channels = c1#Describe the sinkgroupsa1.sinkgroups = g1 g2a1.sinkgroups.g1.sinks = k1 k2 k3 k4a1.sinkgroups.g1.processor.type = failovera1.sinkgroups.g1.processor.priority.k1 = 10a1.sinkgroups.g1.processor.priority.k3 = 10a1.sinkgroups.g1.processor.priority.k4 = 10a1.sinkgroups.g1.processor.priority.k2 = 5a1.sinkgroups.g1.processor.maxpenalty = 10000#Describe/config the sourcea1.sources.r1.type = syslogtcpa1.sources.r1.port = 5140a1.sources.r1.host = localhosta1.sources.r1.channels = c1#Describe the sinka1.sinks.k1.type = avroa1.sinks.k1.channel = c1a1.sinks.k1.hostname = 192.168.220.159a1.sinks.k1.port = 44411a1.sinks.k2.type = avroa1.sinks.k2.channel = c1a1.sinks.k2.hostname = 192.168.220.159a1.sinks.k2.port = 44422a1.sinks.k3.type = avroa1.sinks.k3.channel = c1a1.sinks.k3.hostname = 192.168.220.159a1.sinks.k3.port = 44433a1.sinks.k4.type = avroa1.sinks.k4.channel = c1a1.sinks.k4.hostname = 192.168.220.159a1.sinks.k4.port = 44444#Use a channel which butters events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100

Let's send the data.

Here we can see that K3 receives data, but no matter how much data is sent, it is received by K3. The original idea was to check whether multiple sinks of the same level would carry the load. The test results failed, but not;

Next, let's take a look at how to cut K3;

If K3 breaks down the result K2 receives the data, K2 has the lowest level; if so, let's take a look at K2.

Result 1 and K4 cannot receive data. Let's take a look at the log

org.apache.flume.EventDeliveryException: All sinks failed to process, nothing left to failover to        at org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:191)        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)        at java.lang.Thread.run(Thread.java:745)2014-07-08 12:12:26,574 (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO - org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:206)] Rpc sink k2: Building RpcClient with hostname: 192.168.220.159, port: 444222014-07-08 12:12:26,575 (SinkRunner-PollingRunner-FailoverSinkProcessor) [INFO - org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:126)] Attempting to create Avro Rpc client.2014-07-08 12:12:26,577 (SinkRunner-PollingRunner-FailoverSinkProcessor) [WARN - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:620)] Using default maxIOWorkers2014-07-08 12:12:26,595 (SinkRunner-PollingRunner-FailoverSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.

All sinks failed to process. The test above shows that the sink level cannot be the same when failover is configured; if multiple sinks of the same level are configured, only one sink is effective;

I sent an email to the community. The email address is: Workshop.

Each sink needs to have a different priority. If multiple sinks have same priority, only one of them will be used.

This was answered by a foreign guy called cloudera. This means that when I configure multiple sinks, the sink level must be different;

Test 4: -- idea: test what the above guy said, haha

Profile (flume-sink.properties)

#Name the compents on this agenta1.sources = r1a1.sinks = k1 k2 k3 k4a1.channels = c1#Describe the sinkgroupsa1.sinkgroups = g1 g2a1.sinkgroups.g1.sinks = k1 k2 k3 k4a1.sinkgroups.g1.processor.type = failovera1.sinkgroups.g1.processor.priority.k1 = 10a1.sinkgroups.g1.processor.priority.k3 = 9a1.sinkgroups.g1.processor.priority.k4 = 9a1.sinkgroups.g1.processor.priority.k2 = 5a1.sinkgroups.g1.processor.maxpenalty = 10000#Describe/config the sourcea1.sources.r1.type = syslogtcpa1.sources.r1.port = 5140a1.sources.r1.host = localhosta1.sources.r1.channels = c1#Describe the sinka1.sinks.k1.type = avroa1.sinks.k1.channel = c1a1.sinks.k1.hostname = 192.168.220.159a1.sinks.k1.port = 44411a1.sinks.k2.type = avroa1.sinks.k2.channel = c1a1.sinks.k2.hostname = 192.168.220.159a1.sinks.k2.port = 44422a1.sinks.k3.type = avroa1.sinks.k3.channel = c1a1.sinks.k3.hostname = 192.168.220.159a1.sinks.k3.port = 44433a1.sinks.k4.type = avroa1.sinks.k4.channel = c1a1.sinks.k4.hostname = 192.168.220.159a1.sinks.k4.port = 44444#Use a channel which butters events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100

Let's test the fourth scenario. The test result is normal.

Messages are received by high-level sinks. Turn off, followed by receiving;

For reprint, enter the source address:

Http://blog.csdn.net/weijonathan/article/details/38557959

Http://www.51studyit.com/html/notes/20140814/1055.html