Flume Write Kafka topic overlay problem fix

Structure:

Nginx-flume->kafka->flume->kafka (because involved in the cross-room problem, between the two Kafka added a flume, egg pain. )

Phenomenon:

In the second layer, write Kafka topic and read Kafka topic same, manually set sink topic does not take effect

To open the debug log:

SOURCE instantiation:

APR 19:24:03,146 INFO [conf-file-poller-0] (org.apache.flume.source.defaultsourcefactory.create:41)-Creating Instance of source Kafka1, type org.apache.flume.source.kafka.KafkaSource21 Apr 19:24:03,146 DEBUG [ CONF-FILE-POLLER-0] (org.apache.flume.source.defaultsourcefactory.getclass:61)-Source Type Org.apache.flume.source.kafka.KafkaSource is a custom type21 Apr 19:24:03,152 INFO [conf-file-poller-0] (Org.apache . flume.source.kafka.kafkasourceutil.getkafkaproperties:37)-context={parameters:{topic=bigdata_api_ele_me_ Access, batchdurationmillis=5000, Groupid=nginx, Zookeeperconnect=xxx, Channels=bigdata_api_ele_me_access-channel4 , batchsize=2000, Type=org.apache.flume.source.kafka.kafkasource}}

Sink instantiation:

21 apr 2015 19:24:03,185 info  [conf-file-poller-0]  ( ORG.APACHE.FLUME.SINK.DEFAULTSINKFACTORY.CREATE:42)   - creating instance of sink :  web-sink2, type: org.apache.flume.sink.kafka.kafkasink21 apr 2015 19:24:03,185  DEBUG [conf-file-poller-0]  (org.apache.flume.sink.defaultsinkfactory.getclass:63)   -  Sink type org.apache.flume.sink.kafka.KafkaSink is a custom type21  apr 2015 19:24:03,190 debug [conf-file-poller-0]  ( org.apache.flume.sink.kafka.kafkasink.configure:220)   - using batch size: 200021  Apr 2015 19:24:03,190 INFO  [conf-file-poller-0]  ( org.apache.flume.sink.kafka.kafkasink.configure:229)   - using the static topic:  nginx-access this may be over-ridden by event headers21 apr 2015 19:24:03,191 info  [conf-file-poller-0]  ( ORG.APACHE.FLUME.SINK.KAFKA.KAFKASINKUTIL.GETKAFKAPROPERTIES:34)   - context={ parameters:{ Topic=nginx-access, brokerlist=1xxx, requiredacks=1, batchsize=2000, type= Org.apache.flume.sink.kafka.kafkasink, channel=bigdata_api_ele_me_access-channel4} }21 apr  2015 19:24:03,191 DEBUG [conf-file-poller-0]  ( org.apache.flume.sink.kafka.kafkasink.configure:236)   - kafka producer properties:  {metadata.broker.list=192.168.101.43:9092,192.168.101.44:9092,192.168.101.45:9092,  request.required.acks=1, key.serializer.class=kafka.serializer.stringencoder, serializer.class= Kafka.serializer.DefaultEncoder}

You can see that when you create the sink and source instances, the configuration context topic is set, but you see the following paragraph in the log:

Using The static topic:nginx-access This is over-ridden by event headers

Analysis Kafkasink Source:

In the Org.apache.flume.sink.kafka.KafkaSink.process method:

  public static final String KEY_HDR =  "KEY";  public  static final string topic_hdr =  "TOPIC";  ...         if  ((Eventtopic = headers.get (TOPIC_HDR))  == null)  {           eventTopic = topic;     The value of the     } //eventtopic is taken from the header, if not in the header, the configured topic     is used     ...        eventkey = headers.get ( KEY_HDR);        ...         Keyedmessage<string, byte[]> data = new keyedmessage<string, byte[]>            (eventtopic, eventkey, eventbody);      &nBsp;  messagelist.add (data); 

The value of topic in Configure:

    topic = context.getstring ( KAFKASINKCONSTANTS.TOPIC,      KAFKASINKCONSTANTS.DEFAULT_TOPIC);  // Get topic with flume configuration, if not set topic by default Default-flume-topic processing     if  (Topic.equals ( kafkasinkconstants.default_topic))  {      logger.warn ("The Property   ' topic '  is not set.  '  +         ' Using  the default topic name:  " +         Kafkasinkconstants.default_topic);    } else {       logger.info ("using the static topic: "  + topic +          " this may be over-ridden by event headers");  //This tip may be overwritten by the header     } 

Source of Header:

1) The data in Kafka is not the concept of header

2) message Header/body concept in flume

In this structure, the data is entered into flume by Kafkasource, the header information is added, and then flowed into the Kafkasink

The addition of headers in Kafkasource is handled in the Org.apache.flume.source.kafka.KafkaSource.process method:

        if  (Iterstatus)  {           // get next message           messageandmetadata<byte[], byte[]> messageandmetadata = it.next ();           kafkaMessage =  Messageandmetadata.message ();          kafkakey =  Messageandmetadata.key ();           // add headers  to event  (Topic, timestamp, and key)            headers = new HashMap<String, String> ();           headers.put (kafkasourceconstants.timestamp,              &Nbsp;    string.valueof (System.currenttimemillis ()));           headers.put (Kafkasourceconstants.topic, topic);

Because the header is not needed in the Kafka, comment out these pieces of code in Org.apache.flume.sink.kafka.KafkaSink.process:

/* if ((Eventtopic = Headers.get (TOPIC_HDR)) = = null) {eventtopic = TOPIC; } */eventtopic = topic; Add this paragraph, otherwise there will be a NPE error

This article is from the "Food and Light Blog" blog, please make sure to keep this source http://caiguangguang.blog.51cto.com/1652935/1638342

Flume Write Kafka topic overlay problem fix