Python retry retry When an unexpected crash occurs using asynchronous task celery

Objective:

Python under the celery is something that everyone should have to understand, is an asynchronous task framework. In other words, I have written an alarm platform project, also need to expand the task of distributed, then always feel that with celery is not so reliable, so I wrote a distributed task distribution system.

Today and friends talk about the distributed crawler, this buddy said the task sometimes often collapse, but celery retry mechanism some meaning, finally looked at the following document, and studied the parameters of the next retry, and then put their own some of the actual combat to share to everyone.

#xiaorui. Cc@celery.task (Bind=true,max_retries=3,default_retry_delay=1 * 6) def sum (self, num): try:f = open (' PLO        G ', ' a ') f.write (' retry\n ') f.close () num = num + 1 return num except Exception as exc: Raise Self.retry (Exc=exc, countdown=60)

In fact, the most important of those parameters, the official website is also very able to write, to give an example. hehe ~

Bind=true is open

Max_retries is the time to try again

Default_retry_delay is the default interval time, the time to try

The following code, you should understand. is to catch the exception.

Countdown is also time, this time priority is greater than the above default_retry_delay.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/48/28/wKiom1QFykOCq-JIAADh1XYHEAs045.jpg "title=" screen shot 2014-09-02 pm 9.46.30.png "alt=" Wkiom1qfykocq-jiaadh1xyheas045.jpg "/>

This time I can see, I have just set, after encountering an exception, re-executed three times.

Note that this exception is my own deliberately thrown out, do not understand the above the py.  Another thing is that celery himself will sleep time. I have defined 60s.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/48/2B/wKioL1QFzLfDAJRpAAyb981Zi7w970.jpg "title=" Qq20140902-3_meitu_1.jpg "alt=" Wkiol1qfzlfdajrpaayb981zi7w970.jpg "/>

Then we test, restart the celery, the task must be normal operation, after all, is placed in the queue. When you start celery, he just takes the task from the queue. When I write to celery, just make sure the back-end queue is not dead.

REDIS 127.0.0.1:6379> LRANGE CELERY 0 -11)   "{\" body\ ":  \" Gaj9cqeovqdlehbpcmvzcqjovqn1dgnxa4hvbgfyz3nxbfueaxagyxefhxegvqvjag9yzhehtlujy2fsbgjhy2tzcqhovqhlcnjiywnrc3ejtluhdgfza3nld Hektlucawrxc1ukytzkmtjkztmtyjuzoc00zjmxlwfinzmtnjexntqwyjy5nmzkcqxvb3jldhjpzxnxdusdvqr0yxnrcq5vcxrhc2tzlnn1bxepvql0aw1lbg Ltaxrxee5ohnervqnldgfxelugmjaxnc0wos0wmlqxmjoxmjozoc4wnde4otyrmda6mdbxe1uga3dhcmdzcrr9crv1lg==\ ",  \" Headers\ ":  {\" redelivered\ ":  true}, \" content-type\ ":  \" application/x-python-serialize\ ",  \ "properties\":  {\ "body_encoding\":  \ "base64\",  \ "delivery_info\":  {\ "priority\":  0, \ "routing_key\":  \ "celery\",  \ "exchange\":  \ "celery\"}, \ "delivery_mode\":  2, \ "correlation_id\":  \ "a6d12de3-b538-4f31-ab73-611540b696fd\",  \ "reply_to\":  \ " 4459d9e6-2cff-35c9-be5b-45a2d976911e\ ",  \" delivery_tag\ ":  \" bd4480dd-d04a-4401-876b-831b30b55f4e\ "},  \ "Content-encoding\ ":  \" Binary\ "}" 2)   "{\" body\ ":  \" Gaj9cqeovqdlehbpcmvzcqjovqn1dgnxa4hvbgfyz3nxbfueaxagyxefhxegvqvjag9yzhehtlujy2fsbgjhy2tzcqhovqhlcnjiywnrc3ejtluhdgfza3nld Hektlucawrxc1ukndczmzfhytgtnzzhoc00n2e1ltg1mgitnzzkyty0yjy2yzm1cqxvb3jldhjpzxnxdusbvqr0yxnrcq5vcxrhc2tzlnn1bxepvql0aw1lbg Ltaxrxee5ohnervqnldgfxelugmjaxnc0wos0wmlqxmjoxmjo1ns40nja0mzarmda6mdbxe1uga3dhcmdzcrr9crv1lg==\ ",  \" Headers\ ":  {\" redelivered\ ":  true}, \" content-type\ ":  \" application/x-python-serialize\ ",  \ "properties\":  {\ "body_encoding\":  \ "base64\",  \ "delivery_info\":  {\ "priority\":  0, \ "routing_key\":  \ "celery\",  \ "exchange\":  \ "celery\"}, \ "delivery_mode\":  2, \ "correlation_id\":  \ "47331aa8-76a8-47a5-850b-76da64b66c35\",  \ "reply_to\":  \ " 4459d9e6-2cff-35c9-be5b-45a2d976911e\ ",  \" delivery_tag\ ":  \" 9fa3c120-0bfd-4453-9539-1465e6e820ff\ "},  \ "content-encoding\":  \ "Binary\"} "redis 127.0.0.1:6379> 

In fact, I am more concerned about the collapse of the processing, such as our celery has done a distributed extension. When a node has gone to the task, but suddenly oom, SX.  I thought celery was using RABBOTMQ's ACK mechanism to handle this situation, but my test results told me that celery's retry mechanism was confined to local play. In fact, even if we don't use his retry decorator, we can also write a for loop, and then filter the exception.

What I do now is, every time I get to a task, before I do something, I need to call back an interface, then push the things I want to do, then make a marker, say I'm doing it, and if you don't erase it after 10 minutes, you're stuck in the queue.

Of course there are some setbacks, but have been online for some time, there is no big problem, but in the case of too many tasks, monitoring the task of the thread seems to be more than a crash. Later, you can use gevent pool as a way to perform rotation monitoring events.

If you are the kind of platform nature of the task release, on the page for a long time is loading .... Status, you can easily make an analysis.

But everyone is not so unlucky ~ as long as the exception handling good.

Individual or the development of their own points. Celery may be too powerful to overlook a few places.

This article comes from "Feng Yun, on her." "blog, declined reprint!"

Python retry retry When an unexpected crash occurs using asynchronous task celery