The use of asynchronous task queue celery in Django

Some time ago in the development of the Django Web platform, some requests performed for a long time (a few minutes), in order to speed up the user's response time, so decided to take the asynchronous task in the background to perform these tasks. Under the guidance of colleagues to contact celery This asynchronous task queue framework, in view of the online about celery and django combination of documents less, most of it is only coarse rough introduction of the approximate process, in the course of practice encountered a lot of pits, hoping to record down to help needy friends.

One, the asynchronous request in Django

The process for the Django web to originate from an HTTP request to get a response back to the HTML page is as follows: HTTP request initiation-HTTP handling (Request parsing)-URL mapping (URL regular match to find the corresponding view)-- Logical processing in view, data calculation (including calling the model class for database additions and deletions)--pushes the data to the template and returns the corresponding template/response.

Figure 1. Django Architecture Overview

Sync Request: All logical processing, data calculation tasks return response after processing in view. The user waits while the view is processing the task until the page returns the result.

Asynchronous request: View returns response before processing the task in the background. Users can continue to browse the site without waiting. When the task processing is complete, we can inform the user again.

Second, about celery

Celery is a distributed task queue framework based on Python that enables task scheduling on a step machine/process/thread using a task queue.

Figure 2. Celery architecture

Figure 2 shows the architecture of celery, which uses a typical production-consumer model, consisting mainly of three parts: broker (Message Queuing), workers (consumer: Processing Task), backend (store result). In practical application, the user initiates a request from the Web front end, we only need to throw the task that the request handles to the task queue broker, the idle worker to handle the task, the result of processing will be temporarily in the background database backend. We can simultaneously work on multiple worker processes on a single machine or multiple machines to achieve distributed parallel processing tasks.

Third, the realization of celery in Django

In actual use, it is found that the implementation of celery in Django is quite different from the implementation in the general. py file, and Django has a specific way of using celery. This article focuses on the implementation of celery in Django, and briefly describes the differences in how they are implemented in the general. py file.

1. Establish Message Queuing

First, we must have a broker message queue for sending and receiving messages. The celery website provides options for multiple brokers: RabbitMQ, Redis, Database (not recommended), and other message middleware. We use RABBITMQ as our message broker with strong recommendations from our official website. The installation on Linux is as follows:

sudo apt-get install Rabbitmq-server

Once the command executes successfully, the Rabbitmq-server is already installed and running in the background.

Alternatively, you can stop the server by using command Rabbitmq-server to start the RABBITMQ server and the command Rabbitmqctl stop.

For more commands, refer to RABBITMQ's user manual: Https://www.rabbitmq.com/manpages.html

2. Installing Django-celery

pip install celerypip install Django-celery

3. Configure settings.py

First, add the following configuration code to the Django project's settings.py file:

Import Djcelerydjcelery.setup_loader () Broker_url ' Amqp://[email protected]// '  'amqp://[email protected]//'

When Djcelery.setup_loader () runs, celery will look at tasks.py files in all the app directories included in Installd_apps, find a method labeled task, and register them as celery task. Broker_url and Celery_result_backend respectively refer to the proxy address of your broker and the backend (RESULT store) data store address. In Django, if backend is not set, its default background database is used to store the data. Note that the backend settings here are configured with the keyword celery_result_backend , which differs from the backend settings implemented celery in a generic. py file. In general. PY is configured directly by setting the backend keyword, as follows:

App = celery (' tasks ', backend= ' Amqp://[email protected]//', broker= ' Amqp://[email protected]//')

Then, add Djcelery to the Installed_apps:

Installed_apps = (    ...)   
    ' QV ',     ' Djcelery '     ...   )   

4. In the app root directory where you want to use the task queue (such as QV), create a tasks.py, such as:

In tasks.py we can encode the task logic that we need to execute, import the task at the beginning, and then use the adorner @task at the beginning of the task method to be executed. It is important to note that, unlike the generic. PY implementation celery, tasks.py must be built in the root directory of each app and cannot be named at will.

5. Production Tasks

In the view that needs to perform the task, the task is created in a build_job.delay way and fed into the message queue. Like what:

6. Start the worker command

# start the server first python manage.py runserver #  python manage.py celery worker-c 4--logievel=info

Iv. Supplementary

Django to see other celery commands, including parameter configuration, how to start a multi-worker process, can be viewed through Python manage.py celery--help:

In addition, celery provides a tool flower to monitor and visualize the execution of individual tasks, the health status of individual workers, as shown in the following:

Django is implemented in the following ways:

1. Install Flower:

Pip Install Flower

2. Start Flower (the default is to start a webserver with Port 5555):

Python manage.py celery Flower

3. Enter http://localhost:5555 to view.

　　

　　　　

　　

　　

The use of asynchronous task queue celery in Django