Python Learning note-day14-celery asynchronous task

Celery Overview

For the definition of celery, first look at the official website:

Celery (celery) is a simple, flexible and reliable distributed system that handles a large number of messages and provides the necessary tools to maintain such a system.

In short, it is a python-developed distributed asynchronous message task queue that performs task scheduling on distributed machines, processes, and threads in a way that uses task queues. It makes it easy to implement asynchronous processing of tasks, and if you need to use asynchronous tasks in your business scenario, consider using celery, which can be used in several instances of the scenario:

1. You want to execute a batch command on 100 machines, it may take a long time, but you do not want your program to wait for the results to return, but to return a task ID, you can only take this task ID for a period of time to get the task execution results, when the task execution ing, you can continue to do other things.
2. You want to do a timed task, such as every day to test all of your customer's information, if found today is the customer's birthday, send him a message blessing.

Celery The task requires a middleman (message middleware) to receive and send task messages, as well as the storage of task results, complete list of middlemen please check the official website

PS: A task queue is a mechanism for distributing tasks between threads or machines.

PS: The input to a message queue is a unit of work called a task, and the stand-alone worker process continuously monitors the queue for new tasks that need to be processed.

Celery Introduction

The celery system can include multiple and intermediate people for high availability and scale-out, with a basic architecture consisting of three parts, message middleware, task execution Unit (worker), and task execution result store (task result Store).

1. message Middleware, Celery itself does not provide a messaging service, but it can be easily integrated with third-party messaging middleware, typically using RABBITMQ or Redis, and, of course, MySQL as well as MongoDB.
2. Task Execution Unit, A worker is a unit of task execution provided by celery, and the worker runs concurrently in a distributed System node.
3. task result store, The task result store is used to store the results of tasks performed by the worker, and celery supports different ways to store the results of the task, including Redis,mongodb,django ORM,AMQP.

Main features of celery:

1. Simple: Once you are familiar with the celery workflow, configuration and use are relatively simple
2. High availability: Celery automatically attempts to re-execute a task when the task execution fails or a connection break occurs during execution
3. Fast: A single-process celery can handle millions of tasks per minute
4. Flexible: Almost celery components can be expanded and customized

Celery Basic Work Flow chart

According to the previous introduction, we can draw the following flowchart:

1. The user application says the task is put into the broker by celery.

2. More than one worker gets the task and executes it through the broker.

3, after the completion of the worker, the results of the task, status and other information will be returned to the broker store, for the user program to read.

Basic use of the celery module

To use celery you need to first install the Celery module, the following example uses Python3 for example.

# Install celery module with pip3 command PIP3 install celery# test for successful installation [[email protected] ~]# Python3python 3.6.4 (default, Dec 21 2017, 17:26:4 3) [GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on Linuxtype ' help ', ' copyright ', ' credits ' or ' license ' for more INFORMATION.&G t;>> import Celery      # no error indicates module installation normal >>>

PS: If you are compiling the installation of Python3, do not necessarily represent the correct installation after performing the above steps, but also need to execute the celery command at the command line, if the error, please refer to this article: Python3 after installing the celery module, execute the celery command error.

Preliminary study on celery

PS: The following example uses Redis as the role of the message middleman.

First create a celery application to define the task list, where the name of the task file is called task.py (note that the file name is used later).

From celery import celery app = celery (' Task ', # Task name, typically the current file name    broker= "redis://10.0.0.3:6379/0",     # The address    of the middleman Backend= "REDIS://10.0.0.3:6379/1" # result data storage address) @app. Task # uses celery to identify a task, and multiple tasks need to use the adorner def add (x, y):    return x +y

Run a worker

Celery-a task worker--loglevel=debug#-A indicates the worker's name # worker marked as worker process #--loglevel marked log output format

　　

Python Learning note-day14-celery asynchronous task