Celery Introduction to use

Transferred from: http://liuzxc.github.io/blog/celery/

Celery is a simple, flexible and reliable distributed system that handles large volumes of messages, a task queue that focuses on real-time processing, and also supports task scheduling. There are two more key concepts in celery:

1. Worker:worker is a standalone process that continuously monitors the queue for tasks that need to be handled;
2. Broker:broker is also known as an intermediary or coordinator, and the broker is responsible for coordinating client and worker communication. The client adds a message to the queue and the broker is responsible for sending the message to the worker.

Installing celery

To install directly using the Python tool pip or Easy_install:

$ pip install celery

Install Broker

Celery supports a variety of brokers, but mainly RabbitMQ and Redis-based, others are experimental, although can be used, but there is no dedicated maintainer. How do we choose between RabbitMQ and Redis?

RabbitMQ is feature-complete, stable, durable and easy to install. It ' s an excellent choice for a production environment.

Redis is also feature-complete, but was more susceptible to data loss in the event of abrupt termination or power failures.

Celery official expressly recommends the use of Rabbitmq,redis in production environments with data loss problems. So if your business can tolerate job loss due to worker crash or power failure, using Redis is a good choice, and this article takes Redis as an example.

Celery support for Redis requires installation-related dependencies, and the following commands can install both celery and redis-related dependencies, but Redis server must be installed separately.

$ pip install -U celery[redis]#-U means to upgrade all specified packages to the latest version

Configuration and use of celery

Celery itself is a lot of configuration items, but if you want it to run, you only need to add one line of configuration:

BROKER_URL = ‘redis://localhost:6379//‘

This line is to tell celery broker the address and select Redis db, which by default is 0. The following is a very simple example of how celery is used:

# task.pyfrom celery import Celeryapp = Celery(‘tasks‘, broker=‘redis://localhost//‘)@app.task()def add(x, y): return x + y

We created a celery instance app that allows you to create tasks and manage workers. In the example above, we created a simple task, which returns the result of the addition of two numbers.

Next, run the celery worker and use it to listen for any tasks to be handled.

$ celery -A task worker

See more parameter options with celery WORKER–HELP

Then we open a shell window and go to the Python console to invoke the Add task:

>>> from task import add>>> add.delay(1, 2)<AsyncResult: 42ade14e-c7ed-4b8d-894c-1ca1ec7ca192>

Delay is a shorthand for apply_async, which is used for a task message. We found that the Add task does not return 3, but an object AsyncResult, which is used to check the status of the task, wait for the task to complete or get the result of the task, and if the task fails, it returns the exception information or the call stack.

Let's try to get the execution result of the task first:

 >>> result = add.delay< Span class= "O" > (1, 2) >>> result.get () Traceback in <module> File  "/usr/local/lib/python2.7/dist-packages/celery/result.py" , line 169, in get no_ack=no_ack, File /usr/local/ lib/python2.7/dist-packages/celery/backends/base.py ", line 604, in _is_disabled  No result backend configured. ' Notimplementederror:no result backend configured. Please see the documentation for more information.        

Error: No result backend configured. The error message tells us that result backend is not configured. Because celery will save the status or result of the task in the result Backend,result backend choice is also many, in this case we still use Redis as result backend.

Celery Task Result Backend

We modify the task.py code, add the settings on the result backend, and restart the celery worker after saving.

# task.py...app = Celery(‘tasks‘, backend=‘redis://localhost‘, broker=‘redis://localhost//‘)...

Then call the Add task again to see if we get the execution result?

>>> from task import add>>> result = add.delay(1,2)>>> result.get()3

We got the results right! How about the use of celery is not very simple?

Celery Flower

Flower is a celery monitoring tool that provides a graphical user interface that allows us to monitor the execution of tasks, perform detailed and historical records, and provide statistical functionality.

Flower Installation

$ pip install flower

Or

$ easy_install flower

Flower Introduction to use

Start the flower process starting with the command line first:

$ flower -A proj --port=5555

Then open the browserhttp://localhost:5555/

Celery Task Type Apply_async

Calling an asynchronous task, which is one of the most common task types, has the same effect as delay, except that delay does not support additional parameters in Apply_async. This method has several important parameters, which are often used in practical applications:

1. Countdown: The number of seconds the task is deferred, and the default is immediately executed;
2. ETA: The absolute time the task was executed

Crontab

Celery also supports timed tasks:

FromCelery.schedulesImportCrontabCelerybeat_schedule= {# executes every Monday morning at a.m. : { ' task ' :  ' Tasks.add '  ' schedule ' : crontab  (hour=7,  Minute=30day_of_week= 1),  ' args ' :  ( 1616},}     

Crontab-schedules

To start a timed task, you need to start a heartbeat process:

$ celery -A proj beat

Celery Introduction to use