Simple Python Queue with Redis

Simple Python Queue with Redis

Posted on June 3, 2012
# Python # redis # nosql

The following article shows how to use
Redis to build a simplemulti-producer, multi-consumer Queue with an interface similar to thepythonstandardlib queue. With this queue you can easily share databetween multiple processes or offload time
Consumig calculations to multiple workerprocesses.

To store the data we use
Redis list data type. Redis Lists storessimple strings sorted by insertion order.

The following redis commands are used:

• RpushInsert an element at the tail of the list
• BlpopGet an element from the head of the list, block if list is empty
• LpopGet an element from the head of the list, return nothing list is empty
• LlenReturn the length of the list

The implementation uses
Redis-py library to talk to the server.

import redisclass RedisQueue(object):    """Simple Queue with Redis Backend"""    def __init__(self, name, namespace='queue', **redis_kwargs):        """The default connection parameters are: host='localhost', port=6379, db=0"""       self.__db= redis.Redis(**redis_kwargs)       self.key = '%s:%s' %(namespace, name)    def qsize(self):        """Return the approximate size of the queue."""        return self.__db.llen(self.key)    def empty(self):        """Return True if the queue is empty, False otherwise."""        return self.qsize() == 0    def put(self, item):        """Put item into the queue."""        self.__db.rpush(self.key, item)    def get(self, block=True, timeout=None):        """Remove and return an item from the queue.         If optional args block is true and timeout is None (the default), block        if necessary until an item is available."""        if block:            item = self.__db.blpop(self.key, timeout=timeout)        else:            item = self.__db.lpop(self.key)        if item:            item = item[1]        return item    def get_nowait(self):        """Equivalent to get(False)."""        return self.get(False)

Usage:

>>> from RedisQueue import RedisQueue>>> q = RedisQueue('test')>>> q.put('hello world')

Now if we have a look at the redis database withredis-cliClient it showsthe expected results:

redis 127.0.0.1:6379> keys *1) "queue:test"redis 127.0.0.1:6379> type queue:testlistredis 127.0.0.1:6379> llen queue:test(integer) 1redis 127.0.0.1:6379> lrange queue:test 0 11) "hello world"

We can get the item from a different script:

>>> from RedisQueue import RedisQueue>>> q = RedisQueue('test')>>> q.get()'hello world'

A subsequent callq.get()Will block until anotherone puts a new item beyond the Queue.

The next step wocould be to an endoder/decoder (e. gpython-json) to theQueue so that you are not limited to send strings.

There alredy exists the nice and simple
Hotqueue library which has thesame interface as the above example and provides encoding/decoding.

Other mentionable queue implementations with a redis backend are:

• Flask-redisA basic Message Queue with Redis for flask.
• CeleryAn asynchronous task queue/job queue based on distributedmessage passing. Much more advanced. Can be used with different storagebackends.
• RqSimple python library for queueing jobs and processing them in the background with workers.
• ResqueIs a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. Used at github. Could des a nice
  Monitoring web interface.
• PyresA resque clone in python.