Kafka RESTful API feature introduction and use

• mentioned above

Using confluent kafka-rest Proxy to implement the Kafka restful service (refer to the previous note), data transmission via HTTP protocol, it is necessary to pay attention to the use of Base64 encoding (or called encryption), If the message is not used before the Post base64 processing will appear: Server message garbled, program error, etc., so the normal process is:
1. Deal with the message of post first UTF-8 unified processing
2. Processing messages with Base64 encoding package

s='kafka,hi'ad="hi,kafka,i ' m xnchall"AA =ad.encode ()#UTF-8 Unified processing print(aa) b64=base64.b64encode (Ad.encode ()) # Unified Processing of base64 encoding package

• Producing messages with Kafka-rest

post/topics/(String:topic_name)

data={"Records":[{"Key":"a2v5","value":"y29uzmx1zw50"},{"value":"a2fma2e=","Partition": 1},{"value":"bg9ncw=="}]}data1={"Records":[{"value":"5bck5pws55qe5a6i5oi35oko5aw977ymagkga2fma2esigknbsb4bmnoywxs"}]}header={"Content-type":"Application/vnd.kafka.v1+json"}r=requests.post (url=url,json=data,headers=header) R=requests.post (Url=url,json=data1,headers=header)

View Code

• producing messages to a specified partition : Produce messages to one partition of the topic

post/topics/(String:topic_name)/partitions/(int:partition_id)

Ad="hi kafka,i ' m xnchall"Url11="HTTP://192.168.160.101:8082/TOPICS/TEST_KFK_LK/PARTITIONS/1"data2={"Records":[{"value":(Base64.b64encode (Ad.encode ())). Decode ()}]}Print(data2) R2=requests.post (url=url11,json=data2,headers=header)Print(R2)Print(r2.content)

View Code

• Create or register a consumption instance: creating A new consumer instance in the consumer group

post/consumers/(String:group_name)

url3="Http://192.168.160.101:8082/consumers/my_group"data3={"ID":"My_consumer1","format":"binary","Auto.offset.reset":"Smallest","auto.commit.enable":"false"}R3=requests.post (Url=url3,json=data3,headers=header)

View Code

• Commit offset Commit offsets for the consumer

post/consumers/(String:group_name)/instances/(string:instance)/offsets

url4="http://192.168.160.101:8082/consumers/my_group/instances/my_consumer1/offsets"  R4=requests.post (Url=url4,headers=header)

View Code

• Consumer News

get/consumers/(String:group_name)/instances/(string:instance)/topics/(string:topic_name)

url_get2="http://192.168.160.101:8082/consumers/my_group/instances/my_consumer1/topics/test_kfk_ LK"rr2=requests.get (url=url_get2,headers=header)#, params={" Timeout " : 3000000}print(RR2)print(rr2.content)print(rr2.text)

View Code

• Delete Consumer instances Destroy the consumer instance

delete/consumers/(String:group_name)/instances/(string:instance)

# url_del= "Http://192.168.160.101:8082/consumers/test_kfk_lk/instances/my_consumer" # D1=requests.delete (Url_del) #删除消费者实例 # print (D1)

View Code

• gets the specified partition, offset message : Consume messages from one partition of the topic. (API V2)

get/topics/(String:topic_name)/partitions/(int:partition_id)/messages?offset= (int) [&count= (int)]

Fetch Response v1 only contains message format V0. Fetch Response v2 might either contain message format v0 or message Format v1. Possible Error codes* Offset_out_of_range (1) * Unknown_topic_or_partition (3) * Not_leader_for_partition (6) * REPLICA_ Not_available (9) * UNKNOWN (-1)

url_p="http://192.168.160.101:8082/topics/test_kfk/partitions/0/messages"rst=requests.get (url_p,headers=header,params={"Offset": 3,"Count": 2})#, "Count": 2})Print(RST)Print(Len (Rst.json ()))if(rst.status_code!=500): For ITRinchRst.json ():Print(Base64.b64decode (itr['value']). Decode ())Print(Rst.url)#http://192.168.160.101:8082/topics/test_kfk/partitions/0/messages?offset=3&count=2

View Code

• Gets the list of topic for the current subscription . (API V2)

post/consumers/(String:group_name)/instances/(string:instance)/subscription

• gets the partition of the manually specified consumer (API V2)

get/consumers/(String:group_name)/instances/(string:instance)/assignments

Get/consumers/testgroup/instances/my_consumer/assignments Http/1.1host: proxy-instance.kafkaproxy.example.comaccept:application/vnd.kafka.v2+jsonhttp/1.1 okcontent-type:application /vnd.kafka.v2+json{  "Partitions": [    {      "topic": "Test",      "Partition": 0    },    {      "topic": " Test ",      " Partition ": 1    }]}

• overwrite the offset of the message consumers are about to consume (API V2)

post/consumers/(String:group_name)/instances/(string:instance)/positions

Post/consumers/testgroup/instances/my_consumer/positions Http/1.1host: proxy-instance.kafkaproxy.example.comcontent-type:application/vnd.kafka.v2+json{  "offsets": [    {      "topic": "Test", "      Partition": 0,      "offset": $    {"      topic": "Test",      "Partition": 1,      "offset": +    }  ]}

• Gets the last offset of the partition for the given topic

post/consumers/(String:group_name)/instances/(string:instance)/positions/end

Post/consumers/testgroup/instances/my_consumer/positions/end Http/1.1host: proxy-instance.kafkaproxy.example.comcontent-type:application/vnd.kafka.v2+json{  "Partitions": [    {      "topic": "Test",      "Partition": 0    },    {      "topic": "Test",      "Partition": 1    }]}

• Consume topic or partition data using the allocation and subscription APIs

get/consumers/(String:group_name)/instances/(string:instance)/records

get/consumers/testgroup/instances/my_consumer/records?timeout=3000&max_bytes=300000 HTTP/1.1Host: Proxy-instance.kafkaproxy.example.comaccept:application/vnd.kafka.binary.v2+jsonexample binary response:HTTP/1.1 okcontent-type:application/vnd.kafka.binary.v2+json[  {    "topic": "Test",    "key": "A2v5",    "value ":" Y29uzmx1zw50 ","    Partition ": 1,    " offset ": +,  },  {    " topic ":" Test ",    " key ":" A2v5 ",    "value": "A2fma2e=",    "Partition": 2,    "offset": 101,  }]