Celery -- distributed task queue

I. Introduction

Celery is a distributed asynchronous message task queue developed based on python. It is used to process a large number of messages and provides tools required to maintain such systems for operations.
It is a task queue that focuses on real-time processing and supports task scheduling. If you need asynchronous tasks in your business scenario, you can consider using celery.

Ii. instance scenarios

1. It may take a long time for you to execute a batch command on 100 machines, but you don't want your program to wait for the result to be returned, but return a task id for you, after a while, you only need to hold this task ID to get the task execution result. When the task is running ing, you can continue to do other things.
2. You want to make a scheduled task. For example, you can check the information of all your customers every day. If you find that today is the customer's birthday, send a text message to him.

Iii. Advantages

• 1. Simple: 1. After familiarizing yourself with the celery workflow, it is relatively simple to configure and use.
• 2. High Availability: when the task fails to be executed or the connection is interrupted during execution, celery automatically tries to re-execute the task.
• 3. Fast: a single process's celery can process millions of tasks per minute
• 4. Flexibility: almost all celery components can be expanded and customized.

Iv. Getting Started

Celery requires a solution to send and receive messages, which is usually in the form of a separate service called message proxy.
There are several solutions, including:
1. rabbitmq (message queue, a communication mode between programs)
Rabbitmq features complete, stable, durable, and easy to install. It is an excellent choice for the production environment.
If you are using Ubuntu or Debian, run the following command to install rabbitmq:

$ sudo apt-get install rabbitmq-server

After the command is complete, the proxy has been running in the background, ready to move the message for you :. Starting rabbitmq-server: Success
Ii. redis

Apsaradb for redis is fully functional, but data is more likely to be lost when a sudden stop or power failure occurs.

V. Installation

$ pip install celery 

Vi. Application

Create a tasks. py file

from celery import Celeryapp = Celery(‘tasks‘, broker=‘pyamqp://[email protected]//‘)@app.taskdef add(x, y):    return x + y

The first parameter celery is the name of the current module. The name is automatically generated only when the task is defined in the _ main _ module.
The second parameter is the broker keyword parameter, which specifies the URL of the message proxy to be used. Rabbitmq is used here (also the default option ).
You can use rabbitmq amqp: // localhost, or you can use redis: // localhost.
You have defined a task named add and return the sum of two numbers.

 1 from __future__ import absolute_import 2             import os 3             from celery import Celery 4             from django.conf import settings 5             # set the default Django settings module for the ‘celery‘ program. 6             os.environ.setdefault(‘DJANGO_SETTINGS_MODULE‘, ‘saruman_server.settings‘) 7             app = Celery(‘saruman_server‘) 8  9             # Using a string here means the worker will not have to10             # pickle the object when using Windows.11             app.config_from_object(‘django.conf:settings‘)12             app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)13 14             @app.task(bind=True)15             def debug_task(self):16                 print(‘Request: {0!r}‘.format(self.request))

Instance with Django

7. Run the celery work server

You can run worker by using the worker parameter to execute our program:

celery -A tasks worker --loglevel=info

For a complete list of available command line options, perform the following operations:

$ celery worker --help

There are several other available commands that can also help:

$ celery help

8. Call a task

To call our task, you can use the delay () method.
Apply_async () can better control task execution

>>> from tasks import add>>> add.delay(4, 4)

The call task returns an asyncresult instance. This can be used to check the status of a task, wait for the task to complete, or obtain its return value (or if the task fails, an exception or backtracking is obtained ).

9. Maintain results

If you want to track the task status, celery needs to store or send the status somewhere. There are several built-in result backend options: sqlalchemy/Django Orm, memcached, redis, RPC (rabbitmq/amqp), and-or you can define your own.
In this example, we use the RPC result backend, which sends the status back as a transient message. The backend uses the backend parameter to specify celery.

app = Celery(‘tasks‘, backend=‘rpc://‘, broker=‘pyamqp://‘)

Alternatively, if you want to use redis as the result backend, but still use rabbitmq as the message proxy (a popular combination ):

app = Celery(‘tasks‘, backend=‘redis://localhost‘, broker=‘pyamqp://‘)

Now that the result backend is configured, let's call the task again. This time, you will keep the instance returned when asyncresult calls the task:

>>> result = add.delay(4, 4)

This ready () method returns whether the task has been processed:

>>> result.ready()False 

10. Configuration

Like consumer appliances, celery runs without too many configurations. It has an input and an output. The input must be connected to the proxy, and the output can be
Select to the result backend.
You can set configurations directly on the application or using the dedicated configuration module. For example, you can change the task_serializer settings to configure the default serialization program for serialization task loads:

app.conf.task_serializer = ‘json‘

If you have configured many settings at a time, you can use update:

app.conf.update(task_serializer=‘json‘,accept_content=[‘json‘], # Ignore other contentresult_serializer=‘json‘,timezone=‘Europe/Oslo‘,enable_utc=True,)

Dedicated configuration modules are recommended for large projects. Hard-coded periodic task interval and task routing are not encouraged. It is much better to save them in a centralized location. This is especially true for libraries because it enables users to control the behavior of their tasks. Centralized configuration also allows your SysAdmin to make simple changes when a system failure occurs.
You can call app. config_from_object () to tell the celery instance to use the configuration module:

app.config_from_object(‘celeryconfig‘)

This module is usually called "celeryconfig", but you can use any module name.
In the preceding example, a module named celeryconfig. py must be loaded from the current directory or Python path. It may look like this:
Celeryconfig. py:

broker_url = ‘pyamqp://‘result_backend = ‘rpc://‘task_serializer = ‘json‘result_serializer = ‘json‘accept_content = [‘json‘]timezone = ‘Europe/Oslo‘enable_utc = True

To verify that the configuration file is working properly and does not contain any syntax errors, try to import it:
######################################## ############

python -m celeryconfig

To demonstrate the powerful functions of the configuration file, you can route tasks with improper behaviors to a dedicated queue:

Celeryconfig. py:
Task_routes = {
'Tasks. add': 'low-priority ',
}
Or, instead of routing it, You can limit the speed of the task. In this way, you can only process 10 such tasks in one minute (10/m:

Celeryconfig. py:
Task_annotations = {
'Tasks. add': {'rate _ limit': '10/m '}
}
If you use rabbitmq or redis as the proxy, you can also instruct the staff to set a new rate limit for the task at runtime:

$ Celery-A tasks control rate_limit tasks. Add 10/m
[Email protected]: OK
New rate limit set successfully

11. How to use celery in the project

1. You can configure celery as an application.
2. The directory structure is as follows:

proj/__init__.py    /celery.py    /tasks.py

3. proj/celery. py content

from __future__ import absolute_import, unicode_literalsfrom celery import Celeryapp = Celery(‘proj‘,    broker=‘amqp://‘,    backend=‘amqp://‘,    include=[‘proj.tasks‘])# Optional configuration, see the application user guide.app.conf.update(    result_expires=3600,)if __name__ == ‘__main__‘:    app.start()

4. Content in proj/tasks. py

from __future__ import absolute_import, unicode_literalsfrom .celery import app@app.taskdef add(x, y):    return x + y@app.taskdef mul(x, y):    return x * y@app.taskdef xsum(numbers):    return sum(numbers)

5. Start worker

$ celery -A proj worker -l info

Output

-------------- [email protected] v4.0.2 (latentcall)---- **** -------- * *** * -- Darwin-15.6.0-x86_64-i386-64bit 2017-01-26 21:50:24-- * - **** ---- ** ---------- [config]- ** ---------- .> app: proj:0x103a020f0- ** ---------- .> transport: redis://localhost:6379//- ** ---------- .> results: redis://localhost/- *** --- * --- .> concurrency: 8 (prefork)-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)--- ***** ------------------- [queues].> celery exchange=celery(direct) key=celery

 

Celery -- distributed task queue