Basic tutorial on installing and using the asynchronous task queue package Celery in Python

Source: Internet
Author: User
This article describes how to install and use the asynchronous task queue package Celery in a Python environment. The Distributed Task Management of Celery is suitable for managing and maintaining server clusters. For more information, see 1. Introduction

Celery (celery) is an asynchronous task queue/Job Queue Based on Distributed message transmission. It focuses on real-time operations, but it also supports scheduling well.
Celery is used by the production system to process millions of tasks every day.
Celery is written in Python, but the Protocol can be implemented in any language. It can also be implemented through webhooks with other languages.
Suggested message proxy RabbitMQ, but limited support for Redis, Beanstalk, MongoDB, CouchDB, and databases (using SQLAlchemy or Django's ORM ).
Celery is easy to integrate with Django, Pylons and Flask. You can use django-celery, celery-pylons and Flask-Celery to add packages.

2. Installation
With the above concepts, you need to install such things: RabbitMQ, SQLAlchemy, Celery
The installation method is also simple: RabbitMQ:
Mac:

brew install rabbitmq

Linux:

sudo apt-get install rabbitmq-server

The remaining two are Python. You can directly install pip. For those who have never installed the MySQL driver, you may need to install MySQL-python.
After the installation is complete, start the service:

$ Rabbitmq-server [Press enter]

Do not close the window after it is started. Next, create a window (Tab)

3. Simple cases
Make sure that your previous RabbitMQ has been started.
In the example on the official website, create a file named tasks. py in any directory. The content is as follows:

from celery import Celeryapp = Celery('tasks', broker='amqp://guest@localhost//')@app.taskdef add(x, y):  return x + y

Run the following command in the same directory:

$ celery -A tasks worker --loglevel=info

This command starts a worker and puts the task (add (x, y) in tasks into the queue.
Keep the window open. open a new window to enter the interaction mode, python or ipython:

>>> from tasks import add>>> add.delay(4, 4)

So far, you can use celery to execute the task. In the python interaction mode above, you can simply call the add task and pass the 4 and 4 parameters.
However, there is a problem. you suddenly want to know the execution result and status of the task. Therefore, you need to set backend.
Modify the code in task. py as follows:

# coding:utf-8import subprocessfrom time import sleepfrom celery import Celerybackend = 'db+mysql://root:@192.168.0.102/celery'broker = 'amqp://guest@192.168.0.102:5672'app = Celery('tasks', backend=backend, broker=broker)@app.taskdef add(x, y):  sleep(10)  return x + y@app.taskdef hostname():  return subprocess.check_output(['hostname'])

In addition to adding backend, a who method is added to test multi-server operations. After the modification is complete, start it as before.
Enter the python interaction model:

>>> From tasks import add, hostname >>> r = add. delay (4, 4) >>> r. execute ready () # within 10 s, and False will be output, because sleep for 10 s >>>>> r = hostname in add. delay () >>> r. result # output your hostname

4. Test multiple servers
After the above test, a doubt emerged. Celery is called Distributed Task Management. Which of the following is its distribution? How are all its tasks executed? On which machine is the execution performed?
If the celery service on the current server is not disabled, install Celery on the other server in the same way and start:

$ celery -A tasks worker --loglevel=info

The previous server's Celery service output the hostname of the server you just started, provided that the server is connected to your rabbitmq.
Then enter the python interaction mode:

>>> From tasks import hostname >>>>> for I in range (10 ):... r = hostname. delay ()... print r. result # output your hostname >>>

Check that the entered content has observed the output of the celery service on the two servers.

5. remote connection to RabbitMQ
At the beginning of the test, the remote server could not connect to the local RabbitMQ service, and later found that the permission needs to be set, in the/usr/local/etc/rabbitmq/rabbitmq-env.conf file, modify the ip address in NODE_IP_ADDRESS = 127.0.0.1 to 0.0.0.0.

6. Summary
This article briefly introduces the use of Celery, which focuses on distributed usage. What is not so nice is that the Code (tasks. py) is deployed once, instead of directly sharing tasks. Maybe Celery matches different worker through tasks? I don't know much about it yet. I will talk about it later.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.