Python Flask framework application calls Redis queue data

Here we will bring you the Python Flask framework application to call the Redis queue data method, so as to achieve asynchronous and non-blocking, so as to improve the performance of some programs in real-time processing. For more information, see Asynchronous tasks

Open the browser, enter the address, and press enter to open the page. Therefore, an HTTP request is sent from the client to the server. the server processes the request and returns the response content.

Every day, we browse the web page and send large and small requests to the server. Sometimes, when the server receives a request, it will find that it also needs to send a request to another server, or the server also needs to do other things, so the requests sent initially are blocked, that is, wait for the server to complete other tasks.

In more cases, the server does not need to wait for the client. in this case, these additional tasks can be done asynchronously. There are many tools for asynchronous tasks. The main principle is to process notification messages. the queue structure is usually used for notification messages. Produce and consume messages for communication and business implementation.

Production consumption and queue
The implementation of the preceding asynchronous task can be abstracted into a producer consumption model. Like a restaurant, a cook is cooking and food is eating. If the chefs do a lot and cannot be sold out for the moment, the chefs will rest. if there are many customers and the chefs are busy, the customers will have to wait slowly. There are many ways to implement producers and consumers. the following example uses the Python standard library Queue to write a small example:

Import randomimport timefrom Queue import Queuefrom threading import Threadqueue = Queue (10) class Producer (Thread): def run (self): while True: elem = random. randrange (9) queue. put (elem) print "cook {} made {} meals --- {} meals left not sold out ". format (self. name, elem, queue. qsize () time. sleep (random. random () class Consumer (Thread): def run (self): while True: elem = queue. get () print "food {} has eaten {} meals --- and {} meals can be eaten ". format (self. name, elem, queue. qsize () time. sleep (random. random () def main (): for I in range (3): p = Producer () p. start () for I in range (2): c = Consumer () c. start () if _ name _ = '_ main _': main ()

The output is as follows:

Cook Thread-1 Cook 1 meal --- there is still 1 meal not sold out Cook Thread-2 Cook 8 meal --- there are 2 meals left not sold out Cook Thread-3 cook 3 meal --- still there are 3 meals left, not sold out, and Thread-4 eats 1 meal --- there are 2 meals to eat Thread-5 eats 8 meals --- there is also 1 meal to eat goods Thread-4 eats 3 meal --- there are still 0 meals that can be eaten by the Cook Thread-1 and made 0 meals --- there is only one meal left not sold out by the Cook Thread-2 and there are 0 meals left --- there are 2 meals left not sold out by the Cook Thread- 1. I made 1 meal. 3 meals left. 3 meals are not sold. Thread-1 is used to cook 1 meal. 4 meals are left. Thread-4 is used to eat 0 meals. 3 meals are available. I have 3 meals for Thread-3, 4 meals left, and 3 meals have not been sold out. Thread-5 has eaten 0 meals, and 3 meals can be eaten. Thread-5 has eaten 1 meal. --- and 2 meals can be eaten by the Cook Thread-2 for 8 meals --- 3 meals left before the cook Thread-2 for 8 meals --- 4 meals left not sold out

Redis queue
Python has a built-in queue structure. We can also use redis to perform similar operations. And make a simple asynchronous task.

Redis provides two methods for message queue. One is the producer consumption mode, and the other is the publish subscriber mode. The former allows one or more clients to listen to message queues. Once a message arrives, the consumer immediately consumes the message. the consumer first grabs the message. if there is no message in the queue, the consumer continues to listen. The latter is also one or more clients that subscribe to message channels. as long as the publisher publishes a message, all subscribers can receive the message, and all subscribers are pinged.

Production consumption mode
It mainly uses the blpop provided by redis to obtain the queue data. if the queue does not have data, it will block the waiting, that is, listening.

import redisclass Task(object):  def __init__(self):    self.rcon = redis.StrictRedis(host='localhost', db=5)    self.queue = 'task:prodcons:queue'  def listen_task(self):    while True:      task = self.rcon.blpop(self.queue, 0)[1]      print "Task get", taskif __name__ == '__main__':  print 'listen task queue'  Task().listen_task()

Publish and subscribe mode
The pubsub function of apsaradb for redis is used to subscribe to a channel. when a publisher publishes a message to a channel, the channel is a message queue.

import redisclass Task(object):  def __init__(self):    self.rcon = redis.StrictRedis(host='localhost', db=5)    self.ps = self.rcon.pubsub()    self.ps.subscribe('task:pubsub:channel')  def listen_task(self):    for i in self.ps.listen():      if i['type'] == 'message':        print "Task get", i['data']if __name__ == '__main__':  print 'listen task channel'  Task().listen_task()

Flask Portal
We have implemented two types of backend services for asynchronous tasks. by starting them directly, we can listen to messages in redis queues or channels. A simple test is as follows:

Import redisimport randomimport loggingfrom flask import Flask, redirectapp = Flask (_ name _) rcon = redis. strictRedis (host = 'localhost', db = 5) prodcons_queue = 'task: prodcons: queue 'pubsub _ channel = 'task: pubsub: channel' @ app. route ('/') def index (): html = """

 
  
Redis Message Queue
  
Production consumer model
  

  
Publish subscriber mode
 "" Return html@app.route ('/prodcons') def prodcons (): elem = random. randrange (10) rcon. lpush (prodcons_queue, elem) logging.info ("lpush {}--{}". format (prodcons_queue, elem) return redirect ('/') @ app. route ('/pubsub') def pubsub (): ps = rcon. pubsub () ps. subscribe (pubsub_channel) elem = random. randrange (10) rcon. publish (pubsub_channel, elem) return redirect ('/') if _ name _ = '_ main _': app. run (debug = True)

Start the script and use

siege -c10 -r 5 http://127.0.0.1:5000/prodconssiege -c10 -r 5 http://127.0.0.1:5000/pubsub

Asynchronous messages can be seen in the script input of the listener respectively. Some time-consuming operations can be performed in asynchronous tasks. of course, these operations do not know the asynchronous execution results. if you need to know the asynchronous execution results, you can consider designing coroutine tasks or using tools such as RQ or celery.

For more articles about calling Redis queue data for Python Flask framework applications, refer to PHP Chinese network!