Pythongevent + zmq + redis with small practices

When talking with darkforday a few days ago, I heard that the libev-based gevent library in Python is also worth noting in coding and performance. A brief introduction to gevent (from tutorial): "response. ItprovidesacleanAPIforavarietyofconcu

When talking with darkforday a few days ago, I heard that the libev-based gevent library in Python is worth noting in coding and performance. a Brief Introduction to gevent (from tutorial): "gevent is a concurrency library based around libev. it provides a clean API for a variety of concu

I heard about Python when talking with darkforday a few days ago.
The code and performance of the libev-based gevent Library are also worth noting.

A Brief Introduction to gevent (from tutorial ):
"
Gevent is a concurrency library based und libev.
It provides a clean API for a variety of concurrency
And network related tasks.

Translation:
Gevent is a libev-based concurrency library that provides a large number of clear and concise concurrency and network
Related task APIs
"

Now I am working on Python development. You can just try gevent first.
Set a small requirement and write a small demo.
The scenario is as follows:
Develop a small service program to pop messages from redis, and then package the messages
The DEALER socket forwarded to [zeromq] for forwarding. If C/C ++ is used,
Think of the following implementation:
1. Use the boost asio interface to implement redis pop protocol to receive messages. boost buffer cache is required.
Zeromq dealer is registered in zmq poller to receive messages regularly.

2. Use libev libevent and Other interfaces. The method is similar to 1.

3. encapsulate epoll, and then similar to 1. Create a wheel to implement an adjustable buffer.

Of course, Python has ready-made redis-py and pyzmq implementation interfaces. In addition, pyzmq supports
Gevent is called, and its socket is handed over to gevent management. redis-py can also
Switch the socket class to a gevent socket [see the blog Concurrent connections to Redis with gevent and redis-py].
The socket classes of both libraries are replaced by the socket class of gevent (This idea may be improved in C/C ++ ).

Code reference:
Gevent with redis-py
Gevent with zmq and zmq poller

Complete demo implementation code is put on github gist

Code details:
Local test definition address Redis-server address and Router Service
Address (the Router code is not available)

# test ip and port REDIS_IP = "127.0.0.1"REDIS_PORT = 6379ROUTER_IP = "127.0.0.1"ROUTER_PORT = 59144 # also we could use 44944, just interesting number in chinese :) 

Import the gevent library after version 1.0.1.Init. Py
AllNo socket is introduced, so use it without modifying the source code
From... Import... Use gevent. socket

# gevent, we use gevent to manipulate io tasks (here all socket classes)import gevent# after gevent 1.0.1 in   __init__.py, the  __all__ has no 'socket' attribute   # another way:   from ... import ... from gevent import socket as gevent_socket 

Import redis, introduce related types and define Redis keys

# redisimport redis.connection# redis's socket resign to gevent.socket redis.connection.socket = gevent_socket#  Instantiate two redis client# redis_pop_client        pop list # redis_setvalue_client   set value from redis import RedisErrorfrom redis import ConnectionPoolfrom redis import Redis  REDIS_CONNECTION_POOL = ConnectionPool(max_connections = 8, host = REDIS_IP, port = REDIS_PORT)redis_pop_client = Redis(connection_pool=REDIS_CONNECTION_POOL)redis_setvalue_client = Redis(connection_pool=REDIS_CONNECTION_POOL)# Redis key TEST_REDIS_LIST_KEY = "kaka:test:list"TEST_REDIS_SETVALUE_KEY = "kaka:test:key"

Zmq. green for importing zmq and pyzmq is implemented based on gevent

# zmq, based on gevent  from zmq import green as zmqcontext = zmq.Context()dealer = context.socket(zmq.DEALER)dealer.set_hwm(1000) # zmq  default  high water mark is 1000, explicitly  here router_addr = "tcp://" + ROUTER_IP + ":" + str(ROUTER_PORT)dealer.connect(router_addr)# regiser dealer in zmq poller, # in each poll task, poller will check dealer ZMQ_POLLIN event poller = zmq.Poller()poller.register(dealer, zmq.POLLIN)

After defining the business logic code, pass the function reference of the business logic to gevent. spawn,
Integrate it into a task list, and finally gevent. joinall (tasks) enters gevent loop
Handles network communication tasks between Redis and zmq.

References:
Gevent source code analysis-(deep analysis of gevent running process)
Gevent official documentation
Gevent official documentation (Chinese)
Gevent socket
Gevent pool
Gevent with redis-py
Gevent with zmq

Digress:
1. asynchronous development of the gevent program for the C/C ++ Network Communication Development you wrote earlier
Provides a new idea.
2. In Python, the socket classes of zmq and redis are handed over to gevent management,
Thanks to the support of Python and third-party interfaces
Is not very well implemented ).
3. After writing a small example, I ran to stackoverflow to answer the questions I have read before :)

Original article, reprinted Please note:Reposted from kaka_ace's blog

Link:Python gevent + zmq + redis with small practices