The simplest introduction to Erlang writing Kafka clients

The simplest introduction to Erlang writing Kafka clients

Struggled, finally measured the Erlang to send messages to Kafka, using the Ekaf Library, reference:

Kafka producer written in Erlang

Https://github.com/helpshift/ekaf

1 Preparing the Kafka client

Prepare 2 machines, one is Ekaf running Kafka client (192.168.191.2), one is Kafka server (Zookeeper+kafka) (192.168.122.199). I installed the EKAF on the development Machine (192.168.191.2). Although it is not necessary to install Kafka on a ekaf machine, I have installed Kafka in order to test whether the Kafka command line client is available:

$ echo $KAFKA _home

/usr/local/apache/kafka_2.9.2-0.8.1.1

Open a Kafka client producer:

$ kafka-console-producer.sh--broker-list 192.168.122.199:9092--sync--topic Ekaf

2 Preparing the Kafka service side

The installation of the Kafka server (broker) references my article and has established EKAF this topic:

Zookeeper+kafka one of the cluster installations

Zookeeper+kafka Cluster Installation II

Kafka Broker Installation location in: 192.168.122.199:9092, configuration file server.properties part of the following:

broker.id=0port=9092host.name=192.168.122.199advertised.host.name=192.168.122.199advertised.port= 9092num.partitions=2zookeeper.connect=192.168.122.199:2181,192.168.122.199:2182,192.168.122.199:2183

The/etc/profile configuration contains:

Export Apache_home=/usr/local/apacheexport kafka_home= $APACHE _home/kafka_2.9.2-0.8.1.1export ZK1_HOME= $APACHE _ Home/zk-cluster/zk1/zookeeper-3.4.6export zk2_home= $APACHE _home/zk-cluster/zk2/zookeeper-3.4.6export ZK3_HOME=$ apache_home/zk-cluster/zk3/zookeeper-3.4.6

Start Zookeeper:

$ZK 1_home/bin/zkserver.sh start$zk2_home/bin/zkserver.sh start$zk3_home/bin/zkserver.sh start

After starting zookeeper, wait a little longer (30 seconds) to start the Kafka, or else an error occurs:

$KAFKA _home/bin/kafka-server-start.sh $KAFKA _home/config/server.properties &

After starting Kafka broker, start a consumer and listen for Message Queuing:

$KAFKA _home/bin/kafka-console-consumer.sh--zookeeper 192.168.122.199:2181--topic Ekaf--from-beginning

A message is entered in the Kafka Client Producer window, which appears in the ER window to prove that the message queue is ready.

3 Kafka Client Development

The following actions are performed on the Kafka Client (192.168.191.2).

1) Get Ekaf

$ git clone https://github.com/helpshift/ekaf.git$ CD Ekaf

2) Compiling Ekaf

$ Rebar Get-deps Clean Compile

How to install and use rebar please refer to my other articles:

One of the usage guides for Erlang Rebar: introductory article

3) Running Ekaf

$ Erl-pa './deps/gproc/ebin '-pa './deps/kafkamocker/ebin '-pa './ebin ' 1> application:load (gproc) .ok2> Application:load (Kafkamocker) .ok3> application:load (EKAF) .ok4> application:set_env (Ekaf, EKAF_BOOTSTRAP_ Broker, {"192.168.122.199", 9092}) .ok5> Application:start (Gproc) .ok6> Application:start (kafkamocker) .ok7> Application:start (EKAF) .ok8> Ekaf:produce_sync (<< "Ekaf" >>, << "Hello Kafka" >>). {{sent,0,<0.67.0>}, {produce_response,1,0,                   [{topic,<< "Ekaf" >>,0,[{partition,0,0,0,[],[],0 ,[]}]}]}}9> Ekaf:produce_sync (<< "Ekaf" >>, << "Hello Kafka" >>). {{sent,1,<0.196.0>}, {produce_response,1,0,                   [{topic,<< "Ekaf" >>,0,[{partition,0,0,0,[],[], 0,[]}]}]}}10> Q (). ok

Note that application:set_env (Ekaf, Ekaf_bootstrap_broker, {"192.168.122.199", 9092}). Configuration to ensure consistency with the kafka-console-producer.sh use. I am a use of IP, a use of hostname, resulting in the failure has been unsuccessful, wasted a few days.

The simplest introduction to Erlang writing Kafka clients