Window environment to build Zookeeper,kafka cluster

To demonstrate the effect of the cluster, a virtual machine (window 7) is prepared, and a single IP multi-node zookeeper cluster is built in the virtual machine (the same is true for multiple IP nodes), and Kafka is installed in both native (Win 7) and virtual machines.

Pre-preparation instructions:

1. Three zookeeper servers, the local installation of one as Server1, virtual machine installation two (single IP)

2. Three Kafka servers, this machine installs one as Server1, the virtual machine installs two.

Note : Of course you can install three servers directly on the virtual machine for Server1, Server2, Server3, respectively.

• Virtual machines and native network environments

1. Adjust the virtual machine's network mode to bridge mode, and turn off the firewall function of the virtual machine;

2. Turn off the host's firewall as well.

3. Ping each other to test for normal connection.

• Download Kafka

Download the Kafka installation package from Kafka official website Http://kafka.apache.org/downloads. (To download binary downloads This type, do not download the source files, easy to use)

• Download Zookeeper

Download the zookeeper installation package from Zookeeper official website http://zookeeper.apache.org/releases.html.

• Download JDK

Download from official website http://www.java.com/download/(recommended to download Oracle's official Java), configure environment variables.

Zookeeper cluster environment to build native configuration modify the zoo.cfg file configuration of the native Sever1:

#存储内存中数据库快照的位置, if you do not set a parameter, the update transaction log will be stored in the default location. #dataDir=/tmp/zookeeperdatadir=d:/bigdata/zookeeper-3.4.10/data# log file location Datalogdir=d:/ bigdata/zookeeper-3.4.10/zlog# Listening port clientport=2181#集群服务器配置server. 1=192.168.1.130:9000:7000server. 2=192.168.1.103:9001:7001server. 3=192.168.1.103:9002:7002

Format: Server. A = B:c:d

A: is a number that indicates the first server

B: Server IP Address

C: is a port number that is used to exchange information for the cluster members, indicating that the server is exchanging information with the leader server in the cluster.

D: The port used exclusively for election leader when the leader is hung out

The complete configuration file is as follows:

# of milliseconds of each tickticktime=2000# The number of ticks that the initial # synchronization phase can takeinitlimit=10# The number of ticks that can pass between # Sending a request and getting an acknowledgementsynclimit=5# The directory where the snapshot is stored.# DoNot use/tmp forStorage,/tmp Here is just # example sakes. #dataDir=/tmp/Zookeeperdatadir=d:/bigdata/zookeeper-3.4.10/Datadatalogdir=d:/bigdata/zookeeper-3.4.10/zlog# the port at which the clients would connectclientport=2181# The maximum number of client connections.# increase This ifYou need to handle more Clients#maxclientcnxns=60# # is sure to read the maintenance sections of the # Administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperadmin.html#sc_maintenance# # of snapshots to retain in Datadir#autopurge.snapretaincount=3# Purge Task interval in hours# Set to"0"To disable Auto purge feature#autopurge.purgeinterval=1server.1=192.168.1.130:9000:7000server.2=192.168.1.101:9001:7001server.3=192.168.1.101:9002:7002

View CodeCreateServerID

Create a new myID file with no suffix in the datadir directory in the zoo.cfg configuration file, and write 1.

Virtual machine configuration

Copy the native zookeeper installation package to the virtual machine, and then create a new two directory in the disk, Server2,server3 the directory.

Modifyserver2 zoo.cfg configuration file:

#存储内存中数据库快照的位置, if you do not set a parameter, the update transaction log will be stored in the default location. #dataDir=/tmp/zookeeperdatadir=e:/bigdata/server2/zookeeper-3.4.10/data# log file location Datalogdir =e:/bigdata/server2/zookeeper-3.4.10/zlog# listening port clientport=2182#集群服务器配置server. 1=192.168.1.130:9000:7000server. 2=192.168.1.103:9001:7001server. 3=192.168.1.103:9002:7002

CreateSeverid

Create a new myID file with no suffix in the DataDir directory in the zoo.cfg configuration file, and write 2.

Similarly configure zookeeper in Server3 so that the configuration of the zookeeper cluster is complete.

Below we launch zookeeper cluster: one to start each zookeeper server:
Successively enter the bin directory of each zookeeper server to execute Zkserver.cmd, the earlier zookeeper will print the other Zookeeper server does not start the error message, until the last boot when there is no exception information. Normal Interface:

Zookeeper cluster environment Construction encountered problems summary:

1. Zookeeper start times cannot open channel to X at election address Error contacting service. It is probably not running.

Workaround:

A. Shutting down all server firewalls

B. Modify the IP of the server to 0.0.0.0

For example:server.1=0.0.0.0:9000:7000

Kafka Cluster Environment Construction:

After downloading Kafka, unzip to a directory and modify the Server.properties configuration.

server.properties Modification Instructions:

Broker.id=0? Unique identifier in the Kafka cluster, and can only be a positive integer port=9091? The service listens on the port Host.name=192.168.1.130? The broker-bound host name (IP) if not set will bind all the interfaces. Advertised.host.name=192.168.1.130? The broker service notifies consumers and producers that in other words, consumers and producers are communicating through this host (IP). If not set, Host.name is used by default. Num.network.threads=2? The maximum number of threads that the broker processes messages, typically the number of cores of the CPU num.io.threads=8? The number of threads that broker handles IO is typically twice times num.network.threads socket.send.buffer.bytes=1048576? The buffer that the socket sends. Socket tuning Parameters SO_SNDBUFFsocket.receive.buffer.bytes=1048576? The socket receives the buffer socket's tuning parameter SO_RCVBUFsocket.request.max.bytes=104857600? The maximum number of socket requests to prevent serveroom. Log.dirs=\logs? Kafka data storage address, multiple address words separated by commas. Multiple directories distributed on different disks can improve read and write performance num.partitions=2? The default number of partitions per tipic, which can be re-enacted when creating topic log.retention.hours=168? Retention time for data files Log.retention.minutes is also a reason. Log.segment.bytes=536870912? The size of the largest file in topic-1 means no file size limit log.segment.bytes and log.retention.minutes any one of the requirements will be removed when the file is created topic can be re-enacted. If not, select the default value log.retention.check.interval.ms=60000? File size Check the cycle time, whether to punish the policy set in Log.cleanup.policy log.cleaner.enable=false? Whether to turn on log cleanup zookeeper.connect=192.168.1.130:num1,192.168.1.130:num2,192.168.1.130: num3? Above our zookeeper cluster zookeeper.connection.timeout.ms=1000000? Incoming group link time Timeout

Native configuration Modify this machine server.properties configuration for Server1:

#在kafka这个集群中的唯一标识, and can only be a positive integer broker.id=0# Kafka The address of the cluster broker.list= 192.168.0.130:9092,192.168.0.103:9093,192.168.0.103:9094#listener and Portport=9092#broker bound host name ( IP) If you do not set the interface to bind all. Host.name=192.168.1.130#kafka数据的存放地址, multiple addresses are separated by commas. Multiple directories distributed on different disks can improve read and write performance # A Comma separated list of directories under which to store log files#log.dirs=/tmp/kafka-L Ogslog.dirs=d:/bigdata/kafka_2.11-1.1.0/kafka-logs# The number of default partitions per Tipic, you can re-enact topic when creating num.partitions= 3 for all Kafka znodes.zookeeper.connect= 192.168.1.130:2181,192.168.1.103:2182,192.168.1.103:2183

Virtual machine configuration

Copy the native Kafka installation package to the virtual machine and store it in the Server2,server3 directory.

Note: If the machine is copied to the virtual machine before Kafka has been used, generated topics, direct copy to build the cluster, start Kafka will report the following error:

Cause: In the topic created in a single point environment, the broker.id of the Kafka server is 0, the borker.id is modified when the cluster is built, and the topic previously created cannot be found.

Workaround: Delete the original topic information.

Modify Kafka Configuration in the Server2 directory:

  #在kafka这个集群中的唯一标识, and can only be a positive integer broker.id  =1# Kafka the address of the cluster broker.list  =192.168.0.130:9092,192.168.0.103:9093,192.168.0.103:9094  #listener and portport  =9093 #broker bound host name (IP) If you do not set the interface to bind all. Host.name  =192.168.1.103 #kafka数据的存放地址, multiple addresses are separated by commas. Multiple directories distributed on different disks can improve read and write performance # A Comma separated list of directories under which to store log files#log.dirs  =/TMP/KAFKA-
   
    logslog.dirs  =d:/bigdata/server2/kafka_2.11-1.1.0/kafka- 
    logs# the number of default partitions per Tipic, which can be re-enacted when topic is created num.partitions  = 3 
    # root directory  
    for  
     all Kafka Znodes.zookeeper.connect  = 192.168.1.130:2181,192.168.1.103:2182,192.168.1.103:2183 
   

Similarly modify the Kafka configuration in the Server3, after configuration, the Kafka integration environment has been configured to complete, let us verify that the integration environment is OK.

Verify that the cluster environment starts the zookeeper server in turn

Go to Zookeeper's Bin directory and execute Zkserver.cmd

Start the Kafka server in turn

Enter Kafka install directory D:\bigData\kafka_2.11-1.1.0, press shift+ right mouse button, select "Open Command Window Here", open command line, enter:. \bin\windows\ on the command line. Kafka-server-start.bat. \config\server.properties Enter. Normal Start screen

On the hostKafka Server Creation topic Main parameter Description:partitions number of partitions

(1). Partitions: Number of partitions, controlling how many logs the topic will fragment into. You can display the specified, if not specified, the number of num.partitions configurations that are used in the broker (server.properties)
(2). Although increasing the number of partitions can provide the throughput of a Kafka cluster, an excessive number of partitions or too many partitions on a single server increases the risk of unavailability and latency. Because of the number of partitions, you need to open more file handles, increase the point-to-time latency, and increase the client's memory consumption.
(3). The number of partitions also limits the degree of parallelism of consumer, which means that the number of threads that limit parallel consumer messages cannot be greater than the number of partitions
(4). The number of partitions also limits the producer send message is the specified partition. If the partition is set to 1,producer when the topic is created, a custom partitioning method is used to specify that the partition is 2 or higher, and the number of partitions can be increased by alter–partitions.

replication-factor Copy

1. The replication factor control message is stored on several brokers (servers), typically equal to the number of broker.
2. The number of Default.replication.factor configurations in broker (server.properties) will be used if there is no production message specified at creation time or through the API to a nonexistent topic.

CreateTopic

1. Create the theme, named "Test20180430", replicationfactor=3 (because there are three Kafka servers running). The number of Replicationfactor can be modified according to the number of Kafka servers in the cluster in order to improve system fault tolerance and so on.
2. Open a new command line in the D:\bigData\kafka_2.11-1.1.0\bin\windows directory, enter the command: Kafka-topics.bat--create--zookeeper 192.168.1.130:2181--replication-factor 3--partitions 1--topic test20180430 Enter.

ViewTopic

Kafka-topics.bat--zookeeper 192.168.1.130:2181--describe--topic test20180430

Result Description:
The first line lists the name of the topic, the number of partitions (partitioncount), the number of replicas (replicationfactor), and other configurations (CONFIG.S)
Leader:1 represents the number of the broker that is read-write
Replicas: Indicates that each partition of the topic is saved in those Borker
ISR: Represents the currently valid broker, ISR is a subset of replicas

Create a producer message provider

1. Open a new command line in the D:\bigData\kafka_2.11-1.1.0\bin\windows directory, enter the command: Kafka-console-producer.bat--broker-list 192.168.1.130:9092,192.168.1.101:9093,192.168.1.101:9094--topic test20180430 Enter. ( the window does not close )

CreateCustomer Message Consumer

1. Open a new command line in the D:\bigData\kafka_2.11-1.1.0\bin\windows directory, enter the command: Kafka-console-consumer.bat--zookeeper 192.168.1.130:2181--topic test20180430 Enter.

In the same way, the Server2 and SERVER3 messages on the virtual machine are created, and then the message is sent through the producer on the host.

At this point, the cluster environment has been verified OK.

Cluster fault tolerance

1. First Look at topic information
Kafka-topics.bat--list--zookeeper 192.168.1.130:2181
To view the details of the specified topic:
Kafka-topics.bat--zookeeper 192.168.1.130:2181--describe--topic test20180430

You can see that at this time the election leader is 0, that is, the virtual machine in the Kafka server, now the virtual machine Kafka server to Kill. At this time leader to become 1, consumers can continue to spend.

Note: The number of ZK deployment is the best base, ZK cluster mechanism is as long as more than half of the node OK, the cluster will be able to provide services. Only the ZK node hangs too much, only half or less than half of the nodes can work, and the cluster fails.

Window environment to build Zookeeper,kafka cluster