Business System-Kafka-storm [log localization]-1. Print the log file to the local

Prerequisites:

1: You may need to understand the logback log system.

2: You may need a preliminary understanding of Kafka.

3: Before viewing the code, please carefully refer to the business diagram of the system

Because Kafka itself comes with the "hadoop" interface, if you need to migrate files in Kafka directly to HDFS, please refer to another blog post of this ID:

Business System-Kafka-storm [log localization]-2: directly transfer logs to HDFS through Kafka

1: System Design Diagram of a formal environment system:

With the Kafka cluster, under two identical topics, Kafka-storm, he Kafka-hadoop, and two consumers are used to distribute two pipelines for the same data:

1. Real-Time Channel

Second: offline Channel

During the log localization process, in the early stage, due to log cleansing, the filtering work was put in the storm cluster, that is, the logs stored in the local Locia. It is the data we have cleaned in the storm cluster.

That is:

As shown in:

In Kafka, the following code is usually used for processing:

Here we have targeted two types of logs with two consumers for processing

package com.mixbox.kafka.consumer;public class logSave {public static void main(String[] args) throws Exception {Consumer_Thread visitlog = new Consumer_Thread(KafkaProperties.visit);visitlog.start();Consumer_Thread orderlog = new Consumer_Thread(KafkaProperties.order);orderlog.start();}}

Here, we store different data to different files based on different original fields.

Package COM. mixbox. kafka. consumer; import Java. io. unsupportedencodingexception; import Java. util. hashmap; import Java. util. list; import Java. util. map; import Java. util. properties; import Org. slf4j. logger; import Org. slf4j. loggerfactory; import Kafka. consumer. consumerconfig; import Kafka. consumer. consumeriterator; import Kafka. consumer. kafkastream; import Kafka. java API. consumer. consumerconnector; import Kafka. message. messageandmetadata;/*** @ author Yin Shuai */public class consumer_thread extends thread {// In fact, we will follow the name of the passed topic, machine // Private logger _ log_order = loggerfactory. getlogger ("order"); // Private logger _ log_visit = loggerfactory. getlogger ("visit"); Private logger _ log = NULL; private final consumerconnector _ consumer; private final string _ topic; Public consumer_thread (string topic) {_ consumer = Kafka. consumer. consumer. createjavaconsumerconnector (createconsumerconfig (); this. _ TOPIC = topic; _ log = loggerfactory. getlogger (_ TOPIC); system. err. println ("log name" + _ TOPIC);} Private Static consumerconfig createconsumerconfig () {properties props = new properties (); props. put ("zookeeper. connect ", kafkaproperties. zkconnect); // here, our group ID is logsaveprops. put ("group. ID ", kafkaproperties. logsave); props. put ("zookeeper. session. timeout. ms "," 100000 "); props. put ("zookeeper. sync. time. ms "," 200 "); props. put ("auto. commit. interval. ms "," 1000 "); return New consumerconfig (props);} public void run () {Map <string, integer> topiccountmap = new hashmap <string, integer> (); topiccountmap. put (_ TOPIC, new INTEGER (1); Map <string, list <kafkastream <byte [], byte []> consumermap = _ consumer. createmessagestreams (topiccountmap); For (kafkastream <byte [], byte []> kafkastream: consumermap. get (_ TOPIC) {consumeriterator <byte [], byte []> iterator = kafkastream. iterator (); While (iterator. hasnext () {messageandmetadata <byte [], byte []> next = iterator. next (); try {// here we split a consumer to process the visit log logfile (next); system. out. println ("message:" + new string (next. message (), "UTF-8");} catch (unsupportedencodingexception e) {e. printstacktrace () ;}}} private void logfile (messageandmetadata <byte [], byte []> next) throws unsupportedencodingexception {_ log.info (new string (next. message (), "UTF-8 "));}}

A simple small tips:

Logback. xml reminds you that the configuration file here is too rough. If necessary, please fill in it by yourself.

<? XML version = "1.0" encoding = "UTF-8"?> <Configuration> <jmxconfigurator/> <! -- Console output log --> <appender name = "stdout" class = "ch. QoS. logback. Core. leleappender"> <! -- Filter trace and debug logs --> <! -- <Filter class = "ch. QoS. logback. Classic. Filter. thresholdfilter"> --> <! -- <Level> info </level> --> <! -- </Filter> --> <! -- Roll back by day, if you need to roll back by hour, set to {yyyy-MM-dd_HH} --> <rollingpolicy class = "ch. qoS. logback. core. rolling. timebasedrollingpolicy "> <filenamepattern> F:/opt/log/test. % d {yyyy-mm-dd }. log </filenamepattern> <! -- If you roll back on a daily basis, the maximum storage time is one day, and all data earlier than one day will be cleared. --> </rollingpolicy> <! -- Log output format --> <layout class = "ch. qoS. logback. classic. patternlayout "> <pattern> % d {yyyy-mm-dd hh: mm: Ss. SSS} [% thread] %-5 level % logger {36}-% MSG % n </pattern> </layout> </appender> <! -- Record the rolling log of the log file --> <appender name = "error" class = "ch. qoS. logback. core. rolling. rollingfileappender "> <File> E:/logs/error. log </File> <filter class = "ch. qoS. logback. classic. filter. levelfilter "> <level> error </level> <onmatch> Accept </onmatch> <onmismatch> deny </onmismatch> </filter> <! -- Define a log file generated every day --> <rollingpolicy class = "ch. qoS. logback. core. rolling. timebasedrollingpolicy "> <filenamepattern> E:/logs/Yuanshi-% d {yyyy-mm-dd }. log </filenamepattern> <maxhistory> 10 </maxhistory> </rollingpolicy> <! -- Log style --> <layout class = "ch. qoS. logback. classic. patternlayout "> <pattern> % d {yyyy-mm-dd hh: mm: Ss. SSS} [% thread] %-5 level % logger {36}-% MSG % n </pattern> </layout> </appender> <! -- Record the rolling log of the log file --> <appender name = "file" class = "ch. qoS. logback. core. rolling. rollingfileappender "> <File> E: \ logs \ file. log </File> <filter class = "ch. qoS. logback. classic. filter. levelfilter "> <level> info </level> <onmatch> Accept </onmatch> <onmismatch> deny </onmismatch> </filter> <! -- Define a log file generated every day --> <rollingpolicy class = "ch. qoS. logback. core. rolling. timebasedrollingpolicy "> <filenamepattern> E:/logs/venality-% d {yyyy-mm-dd }. log </filenamepattern> <maxhistory> 10 </maxhistory> </rollingpolicy> <! -- Log style --> <layout class = "ch. qoS. logback. classic. patternlayout "> <pattern> % d {yyyy-mm-dd hh: mm: Ss. SSS} [% thread] %-5 level % logger {36}-% MSG % n </pattern> </layout> </appender> <appender name = "visit" Class = "ch. qoS. logback. core. rolling. rollingfileappender "> <File> E: \ logs \ visitlog \ visit. log </File> <encoder> <pattern> % MSG % n </pattern> </encoder> <filter class = "ch. qoS. logback. classic. filter. thresholdfilter "> <level> info </level> </filter> <rollingpolicy class =" ch. qoS. logback. core. rolling. timebasedrollingpolicy "> <filenamepattern> E: \ logs \ visit. log. % d {yyyy-mm-dd} </filenamepattern> </rollingpolicy> </appender> <logger name = "visit" additivity = "false" level = "info"> <appender-ref = "visit"/> </logger> <appender name = "order" class = "ch. qoS. logback. core. rolling. rollingfileappender "> <File> E: \ logs \ orderlog \ order. log </File> <encoder> <pattern> % MSG % n </pattern> </encoder> <filter class = "ch. qoS. logback. classic. filter. thresholdfilter "> <level> info </level> </filter> <rollingpolicy class =" ch. qoS. logback. core. rolling. timebasedrollingpolicy "> <filenamepattern> E: \ logs \ order. log. % d {yyyy-mm-dd} </filenamepattern> </rollingpolicy> </appender> <logger name = "order" additivity = "false" level = "info"> <appender-ref = "order"/> </logger> <root level = "debug"> <appender-ref = "file"/> </root> </Configuration>

Business System-Kafka-storm [log localization]-1. Print the log file to the local