Use log4j to write the program log in real time Kafka

The first part constructs the Kafka environment

Install Kafka

Download: http://kafka.apache.org/downloads.html

Tar zxf kafka-<version>.tgz
CD kafka-<version>

Start Zookeeper

You need to configure config/zookeeper.properties before starting zookeeper:

Next, start zookeeper.

Bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka Server

You need to configure Config/server.properties before starting Kafka server. The main configuration of the following items, the content is not said, the comments are very detailed:

Then start Kafka Server:

Bin/kafka-server-start.sh config/server.properties

Create topic

bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test

View the created topic

bin/kafka-topics.sh--list--zookeeper localhost:2181

To start the console producer, send a message to Kafka

bin/kafka-console-producer.sh--broker-list localhost:9092--topic Test this are a message this is
another Message

Start the console consumer, consume the message just sent

bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic test--from-beginning This was
a message
is another message

Delete Topic

bin/kafka-topics.sh--delete--zookeeper localhost:2181--topic test

Note: This action is only valid when Delete.topic.enable=true

Configuring the Kafka cluster (on a single machine)

First copy the Server.properties file for multiple copies (this demonstrates the 4-node Kafka cluster, so you'll need to copy 3 profiles):

CP config/server.properties config/server1.properties
CP config/server.properties config/server2.properties
CP config/server.properties Config/server3.properties

Modify the following contents of the Server1.properties:

Broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1

In the same vein, modify the contents of Server2.properties and server3.properties and keep the zookeeper.connect properties of all profiles pointing to the zookeeper address localhost:2181 running on this computer. Note that since these Kafka nodes will all run on the same machine, it is necessary to ensure that these values are different and are processed in an additive manner. For example on the server2.properties:

broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2

After you have configured the server3.properties, start the nodes in turn:

bin/kafka-server-start.sh config/server1.properties &
bin/kafka-server-start.sh config/server2.properties &
bin/kafka-server-start.sh Config/server3.properties &

Topic & Partition

Topic can logically be considered a queue, each consumption must specify its topic, and can be simply understood as having to indicate which queue to put the message in. To enable Kafka throughput to be linearly enhanced, the topic is physically divided into one or more partition, and each partition is physically corresponding to a folder that stores all messages and index files for this partition.

Now create a backup factor of 3 on the Kafka cluster, with a partition number of 4 topic:

bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 3--partitions 4--topic Kafka

Note: The larger the backup factor Replication-factor, the more fault-tolerant cluster is, that is, when the cluster down, the likelihood of data recovery is greater. The contents of all the partitions together constitute a piece of data, the greater the number of partitions partions, the more the topic message is dispersed, the more evenly distributed message distribution in the cluster.

Then use the kafka-topics.sh--describe parameter to see topic for Kafka Details:

The first line of output is a summary of all partitions, and each subsequent row is a description of the partition. You can see the message topic for Kafka, Partioncount=4,replicationfactor=3 is exactly the number of partitions and backup factors we specified when we created it.

In addition: Leader is the node responsible for all the reads and writes of the partition; replicas refers to all nodes of the partition (whether it is alive or not), and the ISR is a subset of replicas that represents the node that holds the partition information and is currently alive.

Take partition:0. For this partition, the leader of the partition is SERVER0, distributed on the three nodes with ID 0,1,2, and the three nodes are alive.

Let's look at the log of the Kafka cluster:

Where kafka-logs-0 represents Server0 's log, kafka-logs-1 represents Server1 's log, and so on.

From the configuration above, it is known that the node ID 0,1,2,3 corresponds to Server0, Server1, Server2, Server3. The partition:0 in the example above is distributed over three nodes with IDs 0, 1, 2, so you can see kafka-0 this folder on Server0, Server1, server2 three nodes. This kafka-0 on behalf of topic as Kafka Partion0. Part II kafka+log4j Project Integration

Let's take a look at the MAVEN Project structure chart:

The jar package introduced by Pom.xml:

<dependencies> <dependency> <groupId>junit</groupId> &LT;ARTIFACTID&GT;JUNIT&L t;/artifactid> <version>4.12</version> </dependency> <dependency> < Groupid>org.apache.kafka</groupid> <artifactId>kafka-clients</artifactId> <version >0.10.2.0</version> </dependency> <dependency> <groupid>org.apache.kafka</gr Oupid> <artifactId>kafka_2.10</artifactId> <version>0.10.2.0</version> < /dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactid>k afka-log4j-appender</artifactid> <version>0.10.2.0</version> </dependency> &LT;DEP
        Endency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>18.0</version>