Asynchronous execution using Django + celery + rabbitmq

Rabbitmq is a famous message queue. It's a pity that I have heard of its name recently. After learning it, I can't help but scream that there is such a thing in the world! I immediately felt that I had a hammer in my hand. I thought everything was a nail. I threw all the operations that the main website was unwilling to do to rabbitmq.

Let's get down to the truth. Let's first introduce this article.ArticleApplication scenarios. We know that the performance of large websites is very important, but sometimes we have to do some time-consuming operations. For example, in the "new things" system of the SNS website, after I post a message, I will push a notification to all people who follow me. At first glance, there is no difficulty. After posting, find the person who follows me and generate the corresponding message record. But the problem is that 100 people are paying attention to me and will execute 100 insert queries. Even worse, the web server is synchronizing these 100 queries, users cannot see the results.

What should we do? It's the message queue's turn to play. After posting, you only need to send a message to the queue, tell the queue "I posted", and then return the posting result to the user. In this case, another process called worker will retrieve the message and execute the 100 insert queries. In this way, the push notification operation is executed asynchronously in the background, and the user can immediately view the posting result. What's more, you can run multiple workers to achieve distributed processing, and all the heavy tasks will be overwhelmed.

Now, let's take a look at the main character of today:

• DJANGO: The Web framework can only be regarded as a secondary role;
• Rabbitmq: A Message Queue System that stores messages;
• Celery: worker process. It also provides the function of creating tasks in webapp.

Install

The installation environment is MACOs. If you are using another operating system, adjust the installation command on your own.

The installation and configuration of Django will not be mentioned.

First install rabbitmq:

 
$ Sudo port install-N rabbitmq
 

Start rabbitmq:

 
$ Sudo rabbitmq-server-detached
 

Then install celery. Download celery from http://pypi.python.org/pypi/celery#downloads and install:

 
$ Tar xzvf celery-2.2.7.tar.gz $ CD celery-2.2.7 $ Python setup. py build $ sudo Python setup. py install
 

In this process, several dependent packages will be installed, including pyparsing, kombu, amqplib, and anyjson. If Automatic Installation is difficult, you can download and compile it on your own.

To use it in Django, We have to install the Django-celery module. Download Django-celery from the http://pypi.python.org/pypi/django-celery#downloads and install:

 
$ Tar xzvf django-celery-2.2.4.tar.gz $ CD django-celery-2.2.4 $ Python setup. py build $ sudo Python setup. py install
 

This process will install the dependent package Django-picklefield. If necessary, download and compile the package.

Application Program Example

Create a test application:

 
$ Django-admin.py startproject celerytest $ CD celerytest $ django-admin.py Startapp Hello $ CD hello
 

Modify settings. py and add the following content to installed_apps:

 
Installed_apps = (... 'djcelery ', # Add celery 'hello', # test the application}
 

Add the configuration of rabbitmq at the end of settings. py:

 
Import djcelerydjcelery. setup_loader () broker_host = "localhost" broker_port = 5672broker_user = "guest" broker_password = "guest" broker_vhost = "/"
 

Of course, don't forget to configure the database option, because djcelery needs to use the database. After configuration, execute:

 
$ Python manage. py syncdb
 

Run Python manage. py to check whether the djcelery application adds many commands starting with celery * to manage. py. These are the commands used to control worker.

Next we will write a task. Create Hello/tasks. py with the following content:

 
From celery. decorators Import Task @ taskdef add (x, y): Return x + y
 

Modifier @ task converts the Add function into an asynchronous task. Calling add in the webapp does not immediately execute this function. Instead, it packs function names and parameters into messages and sends them to the Message Queue. Then, the worker executes the actualCode(Return X + Y ).

Of course, do not forget the indispensable worker:

$ Python manage. py celeryd-l info
 

Test it in another console:

 
$ Python manage. PY shell> From hello. tasks import add >>> r = add. delay (3, 5) # Run this line to view the running status in the worker log> r. wait () 8
 

As you can see, the Add function runs on the worker and achieves the asynchronous effect. Of course, the queue feature determines that tasks are not executed in real time and may be delayed or sometimes lost. Therefore, the queue is not suitable for executing key tasks. But those tasks that have no impact on the execution results and do not have high real-time requirements can be boldly handed over to rabbitmq to free up the webapp.

 

Reposted from: Use Django + celery + rabbitmq for asynchronous execution,Charlee