Windows Deployment Kafka Journal transfer

First, download
Go to Apache's official website (http://kafka.apache.org/downloads.html) to download the latest two-in-plate pressure
Shrink the package. The current version is kafka_2.11-0.8.2.1.tgz.
Second, decompression
Unzip directly to the D-packing directory.
Third, modify the configuration file
Note the versions are different and may have different configuration files. Please refer to the actual changes.
1. Modify "Kafka.logs.dir=logs" in the Log4j.properties file to "Kafka.logs.dir=/tmp/logs".
2. Modify the 4 "File=logs/server.log" in the Test-log4j.properties file to "File=/tmp/logs/server.log".
3. Copy config/server.properties for the first node server-9093.properties, modify the following:
broker.id=9093
port=9093
Host.name=localhost
log.dirs=/tmp/kafka-logs-9093
4. Copy config/server.properties for the first node server-9094.properties, modify the following:
broker.id=9094
port=9094
Host.name=localhost
log.dirs=/tmp/kafka-logs-9094
Iv. Modifying the Startup file
Remove all *.sh files that cannot be run under Windows. Copy the Kafka_2.11-0.8.2.1/bin/windows/*.bat to the Kafka_2.11-0.8.2.1/bin directory.
1. Modify the "%~dp0" in the Zookeeper-server-start.bat file. /.. /"for"%~dp0/. /”。
2. Modify the "%~dp0" in the Kafka-server-start.bat file. /.. /"for"%~dp0/. /”。
3. Modify the "pushd%~dp0" in the Kafka-run-class.bat file. \..” To "pushd%~dp0". Modify the "%base_dir%/config" in the Kafka-run-class.bat file to "%base_dir%/. /config ".
Add the following in the Kafka-run-class.bat file:
REM Classpath addition for Kafka-core dependencies
For%%i in (%base_dir%\core\build\dependant-libs-%scala_version%\*.jar) do (
Call:concat%%i
)
REM Classpath addition for Kafka-perf dependencies
For%%i in (%base_dir%\perf\build\dependant-libs-%scala_version%\*.jar) do (
Call:concat%%i
)
REM Classpath addition for kafka-clients
For%%i in (%base_dir%\clients\build\libs\kafka-clients-*.jar) do (
Call:concat%%i
)
REM Classpath addition for Kafka-examples
For%%i in (%base_dir%\examples\build\libs\kafka-examples-*.jar) do (
Call:concat%%i
)
REM Classpath addition for Contrib/hadoop-consumer
For%%i in (%base_dir%\contrib\hadoop-consumer\build\libs\kafka-hadoop-consumer-*.jar)
Do (
Call:concat%%i
)
REM Classpath addition for Contrib/hadoop-producer
For%%i inch (%base_dir%\contrib\hadoop-producer\build\libs\kafka-hadoop-producer-*.jar) do
(
Call:concat%%i
)
REM Classpath addition for release
For%%i in (%base_dir%\libs\*.jar) do (
Call:concat%%i
)
REM Classpath addition for core
For%%i in (%base_dir%\core\build\libs\kafka_%scala_binary_version%*.jar) do (
Call:concat%%i
)
Modified to:
REM Classpath addition for release
For%%i in (%base_dir%\. \libs\*.jar) Do (
Call:concat%%i
)
Five, start zookeeper
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory: Zookeeper-server-start. /config/zookeeper.properties
Six, start Kafka
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory to start the first node:
Kafka-server-start.bat. \config\server-9093.properties
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory to start the second node:
Kafka-server-start.bat. \config\server-9094.properties
Seven, the creation of topic batch script execution error, does not affect.
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory:
Kafka-topics.bat Kafka.admin.TopicCommand%*
Viii. creation of Topic
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory (Replication-factor is set to Kafka nodes):
Kafka-topics.bat--create--zookeeper localhost:2181--replication-factor 2--partitions 3--topic my-replicated-topic
IX. View topic Description Information
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory: Kafka-topics.bat--describe--zookeeper localhost:2181--topic My-replicated-topic
Ten, Production topic news
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory: Kafka-console-producer.bat--broker-list localhost:9093--topic My-replicated-topic
After the successful start, you can enter the test content in this window, the return can be sent successfully.
XI. Consumer Topic News
Open a command Prompt window and execute the following command in the Kafka_2.11-0.8.2.1/bin directory:
Kafka-console-consumer.bat--zookeeper localhost:2181--from-beginning--topic
My-replicated-topic
Once the launch is successful, you can see the production message in this window in real time.
12. Unresolved exception information
The command prompt also enters some exception information that is not resolved. But for the moment, it doesn't affect use.
1. When starting Kafka:
[2015-05-14 17:36:50,027] INFO initiating client connection, connectstring=localhost:2181
sessiontimeout=6000 [email protected] (Org.apache.zo
Okeeper. ZooKeeper)
[2015-05-14 17:36:50,058] INFO Opening socket connection to server 127.0.0.1/127.0.0.1:2181.
Will isn't attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate login configuration) (ORG.APACHE.ZOOKEEPER.CLIENTCNXN)
[2015-05-14 17:36:50,063] INFO Socket Connection established to 127.0.0.1/127.0.0.1:2181, initiating session (ORG.APACHE.ZOOKEEPER.CLIENTCNXN)
[2015-05-14 17:36:50,110] INFO Session Establishment complete on server
127.0.0.1/127.0.0.1:2181, SessionID = 0x14d51c7fc560000, negotiated timeout = 6000
(ORG.APACHE.ZOOKEEPER.CLIENTCNXN)
[2015-05-14 17:36:50,112] INFO Zookeeper State Changed (syncconnected) (org. I0Itec.zkclient.ZkClient)
[2015-05-14 17:36:50,273] INFO Loading logs. (Kafka.log.LogManager)
[2015-05-14 17:36:50,323] INFO Recovering unflushed segment 0 in log my-replicated-topic-0. (KAFKA.LOG.LOG)
[2015-05-14 17:36:50,335] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
Java.lang.NullPointerException
At kafka.log.offsetindex.kafka$log$offsetindex$ $forceUnmap (offsetindex.scala:301)
At kafka.log.offsetindex$ $anonfun $resize$1.apply (offsetindex.scala:283)
At kafka.log.offsetindex$ $anonfun $resize$1.apply (offsetindex.scala:276)
At Kafka.utils.utils$.inlock (utils.scala:535)
At Kafka.log.OffsetIndex.resize (offsetindex.scala:276)
At Kafka.log.Log.loadSegments (log.scala:179)
At Kafka.log.log.<init> (log.scala:67)
At
kafka.log.logmanager$ $anonfun $loadlogs$2$ $anonfun $3$ $anonfun $apply$7$ $anonfun $apply$1.apply$mcv$sp ( logmanager.scala:142)
At kafka.utils.utils$ $anon $1.run (utils.scala:54)
At Java.util.concurrent.executors$runnableadapter.call (executors.java:439)
At Java.util.concurrent.futuretask$sync.innerrun (futuretask.java:303)
At Java.util.concurrent.FutureTask.run (futuretask.java:138)
At
Java.util.concurrent.threadpoolexecutor$worker.runtask (threadpoolexecutor.java:895)
At Java.util.concurrent.threadpoolexecutor$worker.run (threadpoolexecutor.java:918) at Java.lang.Thread.run ( thread.java:662)
[2015-05-14 17:36:50,344] INFO completed load of log my-replicated-topic-0 with log end offset 1 (kafka.log.Log)
[2015-05-14 17:36:50,351] INFO Recovering unflushed segment 0 in log my-replicated-topic-1. (KAFKA.LOG.LOG)
[2015-05-14 17:36:50,355] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
Java.lang.NullPointerException
At kafka.log.offsetindex.kafka$log$offsetindex$ $forceUnmap (offsetindex.scala:301)
At kafka.log.offsetindex$ $anonfun $resize$1.apply (offsetindex.scala:283)
At kafka.log.offsetindex$ $anonfun $resize$1.apply (offsetindex.scala:276)
At Kafka.utils.utils$.inlock (utils.scala:535)
At Kafka.log.OffsetIndex.resize (offsetindex.scala:276)
At Kafka.log.Log.loadSegments (log.scala:179)
At Kafka.log.log.<init> (log.scala:67)
At
kafka.log.logmanager$ $anonfun $loadlogs$2$ $anonfun $3$ $anonfun $apply$7$ $anonfun $apply$1.apply$mcv$sp ( logmanager.scala:142)
At kafka.utils.utils$ $anon $1.run (utils.scala:54)
At Java.util.concurrent.executors$runnableadapter.call (executors.java:439)
At Java.util.concurrent.futuretask$sync.innerrun (futuretask.java:303)
At Java.util.concurrent.FutureTask.run (futuretask.java:138)
At
Java.util.concurrent.threadpoolexecutor$worker.runtask (threadpoolexecutor.java:895)
At Java.util.concurrent.threadpoolexecutor$worker.run (threadpoolexecutor.java:918) at Java.lang.Thread.run ( thread.java:662)
[2015-05-14 17:36:50,358] INFO completed load of log my-replicated-topic-1 with log end offset (kafka.log.Log)
[2015-05-14 17:36:50,363] INFO Recovering unflushed segment 0 in log my-replicated-topic-2. (KAFKA.LOG.LOG)
[2015-05-14 17:36:50,367] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
Java.lang.NullPointerException
At kafka.log.offsetindex.kafka$log$offsetindex$ $forceUnmap (offsetindex.scala:301)
At kafka.log.offsetindex$ $anonfun $resize$1.apply (offsetindex.scala:283)
At kafka.log.offsetindex$ $anonfun $resize$1.apply (offsetindex.scala:276)
At Kafka.utils.utils$.inlock (utils.scala:535)
At Kafka.log.OffsetIndex.resize (offsetindex.scala:276)
At Kafka.log.Log.loadSegments (log.scala:179)
At Kafka.log.log.<init> (log.scala:67)
At
kafka.log.logmanager$ $anonfun $loadlogs$2$ $anonfun $3$ $anonfun $apply$7$ $anonfun $apply$1.apply$mcv$sp ( logmanager.scala:142)
At kafka.utils.utils$ $anon $1.run (utils.scala:54)
At Java.util.concurrent.executors$runnableadapter.call (executors.java:439)
At Java.util.concurrent.futuretask$sync.innerrun (futuretask.java:303)
At Java.util.concurrent.FutureTask.run (futuretask.java:138)
At
Java.util.concurrent.threadpoolexecutor$worker.runtask (threadpoolexecutor.java:895)
At Java.util.concurrent.threadpoolexecutor$worker.run (threadpoolexecutor.java:918)
At Java.lang.Thread.run (thread.java:662)
[2015-05-14 17:36:50,371] INFO completed load of log my-replicated-topic-2 with log end offset (kafka.log.Log)
2. When viewing topic description information:
D:\kafka_2.11-0.8.2.1\bin>kafka-topics.bat--describe--zookeeper localhost:2181--topic my-replicated-topic
[2015-05-14 17:40:25,209] ERROR Unable to open socket to
0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181 (Org.apache.zookeeper.ClientCnxnSocketNIO)
[2015-05-14 17:40:25,210] WARN Session 0x0 for server null, unexpected error, closing socket
Connection and attempting reconnect (ORG.APACHE.ZOOKEEPER.CLIENTCNXN)
Java.net.SocketException:Address family not supported by protocol Family:connect
At Sun.nio.ch.Net.connect (Native Method)
At Sun.nio.ch.SocketChannelImpl.connect (socketchannelimpl.java:532)
At
Org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect (clientcnxnsocketnio.java:277)
At
Org.apache.zookeeper.ClientCnxnSocketNIO.connect (clientcnxnsocketnio.java:287)
At Org.apache.zookeeper.clientcnxn$sendthread.startconnect (clientcnxn.java:967)
At Org.apache.zookeeper.clientcnxn$sendthread.run (clientcnxn.java:1003)
Topic:my-replicated-topic partitioncount:3 replicationfactor:2 configs:
Topic:my-replicated-topic partition:0 leader:9093 replicas:9094,9093 isr:9093,9094
Topic:my-replicated-topic partition:1 leader:9093 replicas:9093,9094 isr:9093,9094
Topic:my-replicated-topic Partition:2 leader:9093 replicas:9094,9093 isr:9093,9094
3. When producing topic messages
D:\kafka_2.11-0.8.2.1\bin>kafka-console-producer.bat--broker-list localhost:9093--topic my-replicated-topic
[2015-05-14 17:42:27,943] WARN property topic was not valid (kafka.utils.VerifiableProperties)
4. When consuming topic messages
D:\kafka_2.11-0.8.2.1\bin>kafka-console-consumer.bat--zookeeper localhost:2181--from-beginning--topic My-replicated-topic
[2015-05-14 17:47:11,588] ERROR Unable to open socket to 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181 (Org.apache.zookeeper.ClientCnxnSocketNIO)
[2015-05-14 17:47:11,589] WARN Session 0x0 for server null, unexpected error, closing socket
Connection and attempting reconnect (ORG.APACHE.ZOOKEEPER.CLIENTCNXN)
Java.net.SocketException:Address family not supported by protocol Family:connect
At Sun.nio.ch.Net.connect (Native Method)
At Sun.nio.ch.SocketChannelImpl.connect (socketchannelimpl.java:532)
At
Org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect (clientcnxnsocketnio.java:277)
At
Org.apache.zookeeper.ClientCnxnSocketNIO.connect (clientcnxnsocketnio.java:287)
At Org.apache.zookeeper.clientcnxn$sendthread.startconnect (clientcnxn.java:967)
At Org.apache.zookeeper.clientcnxn$sendthread.run (clientcnxn.java:1003)

Problems encountered during the installation process:

1. Kafka prompts unrecognized VM option ' +usecompressedoops ' Could not to create the Java virtual machine after startup.　　　　Began to think is the memory size problem, later found not, is the JDK problem, I use the 32-bit centos,jdk1.6_24, replaced by JDK1.7 still error. View bin/kafka-run-class.sh Find if [-z] $KAFKA _jvm_performance_opts "]; Then
Kafka_jvm_performance_opts= "-server-xx:+usecompressedoops-xx:+useparnewgc-xx:+useconcmarksweepgc-xx:+ Cmsclassunloadingenabled-xx:+cmsscavengebeforeremark-xx:+disableexplicitgc-djava.awt.headless=true "fi Remove-xx:+u Secompressedoops jmx_port=9999 bin/kafka-server-start.sh Config/server.properties &
Startup success

Windows Deployment Kafka Journal transfer