[Celery] celery Best Practices

Orangleliu Translation Original Text click to view

If your work is related to Django, and sometimes you need to execute some long background tasks. Maybe you have used a task queue. Celery is the most popular project in the python (and Django) World to solve similar problems.

When celery was used as a task queue in some projects, I summarized some best practices and decided to put them down. However, there are some reflections on what you should do, and some functions provided but not fully utilized by celery.

No.1 do not use relational databases as amqp proxies

Let me explain why I think this is wrong.

Unlike rabbitmq, relational databases are designed specifically as amqp proxies. It will be suspended at a certain point in time, and may not be based on transmission/user in production.

I guess the biggest reason people use relational databases is that they already have a database working for Web applications. Why not reuse it. The configuration is very simple and you don't need to worry about other components (such as rabbitmq)

Assume that you have four backend processes. You can put these tasks in the database. This means that there are four processes that frequently go to database polling and check whether there are new tasks. This does not include these four processes. At some point in time, you will find that your task process is very slow. If some tasks are not processed, more tasks will come in, and you will naturally add worker to process the tasks. A large number of workers poll the database to obtain new tasks, resulting in a sudden slowdown in the database, and the disk Io reaches the bottleneck. Your Web applications will also be affected and become slower and slower, because these workers are conducting basic DDoS attacks on the database.

When you have an amqp proxy like rabbitmq, this will not happen because these queues exist in the memory and will not hurt your hard disk. These workers do not need frequent polling because the queue will push new tasks to the worker. If amqp cannot work for some reason, at least it will not affect all usage of Web applications.

I have to say that you should not use relational databases as proxies in the development environment. For example, docker and pre-built images can provide you with a rabbitmq environment in the sandbox.

No. 2 uses multiple queues instead of the default one)

Celery is quite simple to start. It will start a default queue. Unless you define other queues, it will put all the tasks in this queue. The most common is as follows.

@app.task()def my_taskA(a, b, c):    print("doing something here...")@app.task()def my_taskB(x, y):    print("doing something here...")

The two tasks are put in the same Queue (if they are not in celeryconfig. in Py ). I can clearly see what happened, because you only have such a decorator in your background tasks. Here, I am concerned that TASKA and taskb do two completely different things. Maybe one of them is more important than the other, why should they be thrown into a basket? Although a worker can handle these two tasks, imagine a large number of taskb at a certain time, but the more important task is not paid enough attention by the worker? In this case, after adding a worker, all workers will still treat the two tasks equally. In the case of a large number of taskb tasks, TASKA still cannot get the attention it deserves. This brings us to the next point.

No. 3 Use priority wokers

The solution to the above problem is to put TASKA in one queue, taskb in another queue, and assign X Workers to process Q1 queues, in Q2 queues, more tasks need to be processed, and other workers are allocated to Q2 queues. In this way, you can ensure that task kb has enough workers and maintain several high-priority queues for task ka. When the task comes, it can be processed without waiting for a long time.

Therefore, manually define the queue

CELERY_QUEUES = (    Queue(‘default‘, Exchange(‘default‘), routing_key=‘default‘),    Queue(‘for_task_A‘, Exchange(‘for_task_A‘), routing_key=‘for_task_A‘),    Queue(‘for_task_B‘, Exchange(‘for_task_B‘), routing_key=‘for_task_B‘),)

Your routes determines that different tasks are assigned to different queues.

CELERY_ROUTES = {    ‘my_taskA‘: {‘queue‘: ‘for_task_A‘, ‘routing_key‘: ‘for_task_A‘},    ‘my_taskB‘: {‘queue‘: ‘for_task_B‘, ‘routing_key‘: ‘for_task_B‘},}

Then you can start different workers for each task.

celery worker -E -l INFO -n workerA -Q for_task_Acelery worker -E -l INFO -n workerB -Q for_task_B

No. 4 Use the celery's error handling mechanism

Most of the tasks I have seen are that there is no error handling concept at all. If a task fails, the task fails. In some cases, this is a good solution. However, the most common problems I have seen are third-party API errors, network problems, or resource unavailability. The simplest way to handle this error is to retry the task. Some third-party APIs are caused by service or network problems, but they can be recovered quickly. Why don't we give it a try?

@app.task(bind=True, default_retry_delay=300, max_retries=5)def my_task_A():    try:        print("doing stuff here...")    except SomeNetworkException as e:        print("maybe do some clenup here....")        self.retry(e)

I prefer to define a Retry Interval and the number of Retries for each task (the default_retry_delay and max_retries parameters respectively ). This is the most basic error handling method that I have seen most. Of course, celery also provides many processing methods, but I will leave you the celery document address.

No. 5 Use flower

Flower is a great tool that can be used to monitor celery tasks and workers. It is web-based, so you can see the task process, details, worker status, and start new workers. You can view all its functions through the previous link.

No. 6 the task result can be tracked only when necessary

The task status indicates whether the task execution result is successful or failed. It is useful for some subsequent analysis. One problem that needs to be noted is that the exit result is not the result of task execution. The information is more similar to the impact on data (for example, updating the user's friend list)

What I have seen most in projects is that they do not care about the status of these tasks during execution. Some of them are only saved using the default SQLite database, it is better to save time in a regular database (such as ipvs or other databases)

Why does it add the burden on Web application databases for no reason? Use the celery_ignore_result = true configuration to discard the execution status in your celeryconfig. py configuration file.

No. 7 do not execute tasks through databases or ORM objects

A few people suggested that I add this article to the list of best practices after posting this article at a local Python conference. What is this suggestion about? Do not use database objects (such as your user model) to execute background tasks, because object sequences contain outdated data. What you want to do is to put the userid in the task, and then get the latest user object from the data when the task is executed.

[Celery] celery Best Practices