Python via SSH tunnel link Kafka

Python via SSH tunnel link Kafka

There's a need to connect recently. kafka ssh connection kafka As for how the connection can be through xshell proxifier python sshtunnel

as the author here The Kafka environment uses Zookeeper for distributed deployments, with multiple broker IP addresses, but this is hard to Big deal map more than a few ports, everything seems to be very smooth,SSH Tunnel set up, ports are configured well.

From Kafka import kafkaproducerimport logging Logging.basicconfig (level=logging.info) # already configured ssh tunnel kafka_host = [' 127. 0.0.1:19091 ', ' 127.0.0.1:19092 ', ' 127.0.0.1:19093 ',]producer = Kafkaproducer (bootstrap_servers=kafka_host) producer . Send (' Test ', B ' some_message_bytes ')

Start Script discovery problem came up, error:

Traceback (most recent): File ' <stdin> ', line ', in <module> file ' c:\python27\lib\site-packages\ kafka\producer\kafka.py ", line 347, in __init__ **self.config) File" C:\python27\lib\site-packages\kafka\client_async . py ", line up, in __init__ if self.config[' api_version ') is None:file" c:\python27\lib\site-packages\kafka\client_as ync.py ", line 861, in check_version except Errors.NodeNotReadyError:NoBrokersAvailable: Nobrokersavailablenobrokersavailable

could not find a connection Brokers, the same code on the server can be directly connected up ...

so I started to find out Kafka the host name is used to resolve the IP address even though the IP address configuration is used when connecting , the hosts The server host name point to the local address (127.0.0.1) not on the line, according to the truth is OK, but found that the connection is no problem, but the message has not been sent successfully.

Check the log carefully and discover:

Info:kafka.conn:<brokerconnection node_id=258 host=kafka-04/127.0.0.1 port=9092>: Connecting to 127.0.0.1:9092

It is true that the corresponding host name is resolved to the local address, but the port has not changed the corresponding ...

I'm not a person who gives up easily. , and began to read Pykafka The source code, the actual is not very difficult, immediately fixed to the problem. The basic is pykafka will connect through the configured address kafka server, and get an available address (returned here kafka Span style= "FONT-FAMILY:CALIBRI;" >port hosts " ip

Solution:

Finally to the solution, not verbose, directly on.

Find The Pykafka directory, the author here is:

C:\Python27\Lib\site-packages\kafka

Add a custom configuration file self_config.py(self-modifying, here are examples)

Self_design = {"kafka-04": 19094, "kafka-03": 19093, "kafka-02": 19092, "kafka-01": 19091,}

Import configuration Files for client_async.py and conn.py , respectively

From. Self_config Import Self_design

and locate the Get_ip_port_afi method in the source code (This method returns the host processing to the IP,Port , AFI )

# ----------------------------------------

client_async.py

Host, port, AFI = Get_ip_port_afi (broker.host) # ==================================# Self addonconn_port = port if broker. Host in self_design else broker.port# ==================================

and Change the following broker.port to conn_port

conn = Brokerconnection (host, Conn_port, AFI, # Broker.port STATE_CHANGE_CALLBACK=CB, node_id=node_id, **self.config)

# ----------------------------------------

conn.py

If ': ' Not in Host_and_port_str: # ================================== # self Addon if host_and_port_str in Self_d ESIGN:AF = _address_family (HOST_AND_PORT_STR) return u ' 127.0.0.1 ', Self_design[host_and_port_str], AF # ================================== af = _address_family (HOST_AND_PORT_STR) return host_and_port_str, DEFAULT_KAFKA_P ORT, AF

# self Addon code that you added for yourself

Connect again, solve the problem successfully

This article is from the "Cloud Inn-leyex Study Notes" blog, please be sure to keep this source http://leyex.blog.51cto.com/4230949/1958453

Python via SSH tunnel link Kafka