Log4j2 sending messages to Kafka

title: 自定义log4j2发送日志到Kafka

Picture description (max. 50 words)

The 
  Tags:log4j2,kafka to provide the company's big data platform with logs for each project group, while making the project groups unaware of the changes. Did a survey only to find LOG4J2 default has the support to send the log to the Kafka function, under the surprise hurriedly looked under log4j to its realization source! found that the default implementation is synchronous blocking, if the Kafka service once hung up will block the normal service log printing, for this I made some changes based on the reference source. log4j Log Workflow Log4j2 for log4j in the performance of a significant improvement, this official has been clearly described and tested, so do not repeat. In order to use it more skillfully, it is necessary to understand its internal workflow. This is a class diagram of the official website log4j applications using the LOG4J 2 API would request a Logger with a specific name from the Logmanager. The Logmanager would locate the appropriate loggercontext and then obtain the Logger from it. If The Logger must be created it'll be associated with the loggerconfig that contains either a) the same name as the Log GER, B) The name of a parent package, or C) the root loggerconfig. Loggerconfig objects is created from Logger declarations in the configuration. The loggerconfig is associated with the appenders that actually deliver the logevents. The official website has explained the relationship between them, there is no longer a specific description of the function and function of each class, Today's focus is on the Appender class, as he will decide where to export the logs. 

Appender
The ability to selectively enable or disable logging requests based on their logger are only part of the picture. LOG4J allows logging requests to print to multiple destinations. In log4j speak, a output destination is called an Appender. Currently, appenders exist for the console, files, remote socket servers, Apache Flume, JMS, remote UNIX Syslog daemons, a nd various database APIs. See the sections on Appenders for more details on the various types available. More than one Appender can is attached to a Logger.

核心配置

Picture description (max. 50 words)

Picture description (max. 50 words)

is log4j2 send logs to Kafka core class, in fact, the main Kafkaappender, the other several classes are connected Kafka services.

Kafkaappender Core Configuration
@Plugin (name = "Kafka", category = "Core", ElementType = "Appender", PrintObject = True)
Public final class Kafkaappender extends Abstractappender {
/**

• */
  Private static final Long Serialversionuid = 1l;br/> @PluginFactory
  @PluginElement ("Layout") final layout<? Extends serializable> layout,
  @PluginElement ("Filter") Final filter filter,
  @Required (message = "No name provided for Kafkaappender") @PluginAttribute ("name") Final String name,
  @PluginAttribute (value = "Ignoreexceptions", Defaultboolean = True) Final Boolean ignoreexceptions,
  @Required (message = "No topic provided for Kafkaappender") @PluginAttribute ("topic") Final String topic,
  @PluginElement ("Properties") Final property[] Properties) {
  Final Kafkamanager Kafkamanager = new Kafkamanager (name, topic, properties);
  return new Kafkaappender (name, layout, filter, ignoreexceptions, Kafkamanager);
  }
  Private final Kafkamanager manager;
  Private Kafkaappender (final String name, final layout<? extends Serializable> Layout, final Filter filter, Final boo Lean Ignoreexceptions, Final Kafkamanager manager) {
  Super (name, filter, layout, ignoreexceptions);
  This.manager = Manager;br/>}
  @Override
  if (Event.getloggername (). StartsWith ("Org.apache.kafka")) {
  Logger.warn ("Recursive logging from [{}] for Appender [{}].", Event.getloggername (), GetName ());
  } else {
  try {
  if (getlayout () = null) {
  Manager.send (GetLayout (). Tobytearray (event));
  } else {
  Manager.send (Event.getmessage (). Getformattedmessage (). GetBytes (Standardcharsets.utf_8));
  }
  } catch (Final Exception e) {
  Logger.error ("Unable to write to Kafka [{}] for Appender [{}].", Manager.getname (), GetName (), E);
  throw new Appenderloggingexception ("Unable to write to Kafka in Appender:" + e.getmessage (), e); br/>}
  }
  }
  @Override
  Super.start ();
  Manager.startup ();
  }
  @Override
  Super.stop ();
  Manager.release ();
  }
  Log4j2.xml Simple Configuration
  <?xml version= "1.0" encoding= "UTF-8"?>
  ...
  <Appenders>
  <kafka name= "Kafka" topic= "Log-test" >
  <patternlayout pattern= "%date%message"/>
  <property name= "Bootstrap.servers" >localhost:9092</Property>
  </Kafka>
  </Appenders>

  <Loggers>
  <root level= "DEBUG" >
  <appenderref ref= "Kafka"/>
  </Root>
  <logger name= "Org.apache.kafka" level= "INFO"/> <!--avoid recursive logging--
  </Loggers>
  Which @Plugin the Name property corresponding to the XML configuration file inside the Kafka tag, of course, this can also be customized. At the same time, you need to change the Name property of the @Plugin to Mykafka. The following configuration:

<mykafka name= "Kafka" topic= "Log-test" >
Custom Configuration

有时候我们会用到的属性由于默认的 KafkaAppender 不一定支持，所以需要一定程度的改写。但是改写也比较方便，只需要从构造器的 Properties kafkaProps 属性中取值即可。为了满足项目要求，我这边定义了platform和serviceName两个属性。通过 KafkaAppender 的源码可知，他发送消息采取的是同步阻塞的方式。经过测试，一旦kafka服务挂掉，那么将会影响项目服务正常的日志输出，而这不是我希望看到的，所以我对他做了一定的程度的修改。feature：：

Kafka Service has been normal
This is an ideal situation where messages will be sent to Kafka broker
Kafka service hangs up and returns to normal after a certain period of time
When the Kafka service is suspended, all subsequent messages will be exported to the Concurrentlinkedqueue queue. At the same time, the message of this queue will be consumed continuously and output to the local file. When the heartbeat detects that Kafka broker is back to normal, the contents of the local file will be read and sent to the Kafka broker. Note that at this time, there will be a large number of messages are instantiated as Producerrecord objects, heap memory occupancy rate is very high, so I used a thread blocked a bit!
Kafka Service has been hanging
All messages will be exported to a local file.

Log4j2 sending messages to Kafka