Celery with RABBITMQ task queue to realize asynchronous dispatch execution of tasks

Objective:

This thing in the task scheduling aspect, has the very set, the study he has a period of time, oneself also tried to use in the project, but suffers from does not have this opportunity. These two days to the department to write a rest of the SMS interface, but always feel that things will be blocked from time to time, so that other logic run, so the message to send this task backstage asynchronous execution, and so I think up, and then see the results.

For the above scene, I have used Tornado and gevent solution, but in my understanding, the feeling is not mature. Tornado the task asynchronously, it always affects his performance. And Gevent, he is a simulation of events, Super good, but asynchronous, sometimes inexplicable bugs, heard is Gevent 1.1 solution, have time I try again. "Personal understanding is limited, forget to guide"

Why do you use celery?

In my application, it can be done asynchronously to the background, remember, put him to the front.

What's the relationship with RABBITMQ?

The relationship with RABBITMQ is that celery does not have a message storage function, he needs media, such as RABBITMQ Redis MySQL MongoDB can be. With this controllable stuff, you can do it in the library. Recommended use of RABBITMQ, his speed and availability are very high, redis this thing is afraid of accidents, of course, you are lucky, not afraid of him accidentally hung, is available.

Celery and RABBITMQ are two levels of things.
Celery is a distributed task queue. Its basic job is to manage the assignment of tasks to different servers and get results. As for how the servers communicate with each other. This celery itself cannot be solved.
Therefore, RABBITMQ as a Message Queuing management tool is introduced into and celery integration, responsible for processing the communication between the server task.

Of course, later celery has added some support for Redis,mongodb. The reason is that although RABBITMQ is strong enough, it may be too much (complex) for some relatively simple business environments. This allows the user to have a few more choices.

Introduction to Celery

Celery (celery) is an asynchronous task queue/job queue based on distributed message delivery. It focuses on real-time operations, but it is also good for scheduling support.

Celery is used in production systems to handle millions of tasks per day.

Celery is written in Python, but the protocol can be implemented in any language. It can also be implemented with other languages through Webhooks.

The proposed message agent is RABBITMQ, but provides limited support for Redis, Beanstalk, MongoDB, CouchDB, and databases (using SQLAlchemy or Django ORM).

Celery is easy to integrate Django, pylons and flask, with Django-celery, celery-pylons and Flask-celery add-on packages.

Officials say they're using it.

Installation:

1	Pip Install celery

Compiling the installation

1 2 3 4

tar Xvfz celery-0.0.0. tar.gz cd celery-0.0.0 python setup. PY build PYT