Python Learning note 3-celery Distributed task processor

celery is an asynchronous task framework written in Python, very powerful, specific instructions can be viewed on the official website, where the main point of the demo lets you quickly use the framework 1. Installation of the environmentRedis is installed by defaultpip install celerya carrier used by Redis as a task message 2.tasks.py 

Importsysreload (SYS) sys.setdefaultencoding ('utf-8 ') # Without this sentence, printing Chinese log will make an error  fromCeleryImportcelery Celery= Celery ('Tasks', broker='redis://127.0.0.1:6379/0'# Select Local Redis db=0 as the message carrier, the first parameter is the task name fromCelery.utils.logImportGet_task_logger#pour the log module into the celeryLogger = Get_task_logger (__name__) @celery. Task (Bind=true, max_retries=10, Default_retry_delay=1 * 6)#bind means open, max_retries is the number of retries, Default_retry_delay is the default interval, the time to trydefExec_task_order_overtime (self, order_id):#after the order expires, the task of performing the order invalidation    Try: Logger.info ('===================> exec_task_order_overtime order_id=%s'%order_id) Success=BaseHandler.context_services.order_overtime_task_service.process_over_time (order_id)ifSuccess isFalse:logger.error ('<================order_overtime_task_service.process_over_time Failed, order_id=%s'%order_id)RaiseReturn (False)Else: Logger.info ('<=================order_overtime_task_service.process_over_time Success, order_id=%s'%order_id)exceptException as Exc:logger.info ('exec_task_order_overtime Retry, order_id=%s'%order_id)RaiseSelf.retry (Exc=exc, countdown=3)#try again after 3 seconds, where the countdown priority is higher than the default_retry_delay in the adorner 

the file path executes the command, after which celery begins to perform the task as a consumption$celery worker-a Tasks--loglevel=info What about the producers? After executing the following statement, the celery key of db=0 in Redis is a list type, which holds the execution task, if the celery is not opened can be seen clearly; open celery may have been executed

 fromCeleryImportcelery Celery= Celery ('Tasks', broker='redis://127.0.0.1:6379/0')#Message Carrierpush_task_id = Celery.send_task ('Tasks.exec_task_order_overtime', [order_id]#parameters, must be list, the concrete visible source code, the third can be dict, we do not use, countdown=10)#how long to delay execution of push messages 

Question 1:

Some people will wonder why Exec_task_order_overtime has self, sometimes found no, difference in the adorner task, if @celery.task, is not an adorner function call, then there is no self,the bind argument to the task decorator would give access to self (the task type instance) Question 2:Celery constructor parameters, the first is the module name, that is, the name of this file, the second is the Redis carrier address, crossing network introductionthe first argument Celery to was the name of the current module, this only needed so names can be automatically generate D when the tasks is defined in the __main__module.The second argument is the Broker keyword argument and specifying the URL of the message broker you want to use. Here using RabbitMQ (also the default option).  Question 3:celery when memory is low, there is no feedback mechanism:we know that in the socket network transmission, when the receiving end is too late to deal with, the sending side will block; celery has no relevant mechanism at all;we do not need to send the end of the block, of course, but our celery too late to deal with, should be cached in Redis, although it will cause the message processing is not timely, but also not enough memory problems appear, here may need to do some of their own processing;For example: Retry count control as little as possible, the task failed to record after execution, according to business needs such as 10 minutes after the re-thrown into the Redis message queue    

Python Learning note 3-celery Distributed task processor