A practical guide to celery

Celery's Practice guide celery principle: Celery actually implements a typical producer-consumer model of message processing/task scheduling, and consumers (workers) and producers (clients) can have any of them communicating through the message system (broker). The typical scenario is:

1. The client initiates a process (producer), and when certain operations of the user take a long time or are more frequent, consider accessing the message system and sending a task to the broker.
2. A worker process (consumer) is started in the background, and when it is found that a task has been saved in the broker to the execution time, he will take it and execute it according to the task type and parameters.

Typical scenarios in practice:

1. Simple Scheduled Tasks:
   2. Replace crontab's celery notation:
      1. 
         From celery import celery
         From Celery.schedules import crontab
         
         App = Celery ("Tasks", backend= "Redis://localhost", broker= "Redis://localhost")
         
         App.conf.update (Celerybeat_schedule = {
         "Add": {
         "Task": "Celery_demo.add",
         "Schedule": Crontab (minute= "*"),
         "Args": (16, 16)
         },
         })
         
         @app. Task
         def add (x, y):
         return x + y
         
         
   4. Run the celery worker, let him run as consumer, and automatically get the task from the broker and execute it.
      1. ' Celery-a Celery_demo worker '
   6. Run the celery client to automatically produce the task msg according to schedule, and publish it to the broker.
      1. ' Celery-a Celery_demo beat '
   8. Install and run flower for easy monitoring of task running status
      1. ' Celery flower-a celery_demo '
      2. or set the login password ' celery flower-a celery_demo--basic_auth=user1:password1,user2:password2
3. Multi-sync tasks-chained tasks-
5. Task that failed to automatically retry
   1. Failed Retry method: Add the task code function parameter to self, bind bind at the same time.
   2. Demo Code:
      1. @app. Task (Bind=true, default_retry_delay=300, max_retries=5)
         def my_task_a (self):
         Try
         Print ("Doing stuff here ...")
         Except Somenetworkexception as E:
         Print ("Maybe do some clenup ....")
         Self.retry (e)
   3. After automatic retries, do you want to queue up the task after it is queued, or wait for the specified time? Can be specified by the Self.retry () parameter.
7. Dispatched to a different queue of queue task
   1. A task is automatically mapped to multiple queue methods by configuring the Routing_key naming pattern for task and queue.
      1. For example: Configure the queue's exchange and routing_key as a common pattern:
      2. Define the name of the Routing_key for the task:
   2. The different Exchange policies that are available:
      1. Direct: Directly by definition Routing_key
      2. Topic:exchange pushes a message to more than one queue based on a wildcard character.
      3. Fanout: Splitting messages into different queues, typically for extra-large tasks and time-consuming tasks.
   3. Reference: Http://celery.readthedocs.org/en/latest/userguide/routing.html#routers
9. Advanced Configuration
   1. Whether result is saved
   2. Failed Email Notification:
   3. Close Rate Limit:
11. Auto_reload method (*nix System):
    1. Celery automatically reload by monitoring changes to the source code directory
    2. How to use: 1. Dependent on inotify (Linux) 2. Kqueue (OS X/bsd)
    3. Install Dependent: $ pip Install pyinotify
    4. Optionally, specify the dependency of the Fsnotify: $ env celeryd_fsnotify=stat celery worker-l Info--autoreload
    5. Start: celery-a appname worker--autoreload
13. Auto-scale Method:
    1. Enable Auto-scale
    2. Temporarily increase the number of worker processes (increase consumer): $ celery-a proj control Add_consumer foo-d worker1.local
    3. Temporarily reduce the number of worker processes (reduce consumer):
15. To change the configuration of the scheduled task from app.conf to DB:
    1. You need to specify the custom schedule class name at startup, such as the default: Celery.beat.PersistentScheduler.
       1. Celery-a proj Beat-s Djcelery.schedulers.DatabaseScheduler
16. To start the Stop worker method:
    1. Start as Daemon:http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#daemonizing
       2. Root user can use Celeryd
       3. Non-privileged users: Celery multi start worker1-a appname-autoreload--pidfile= "Hom e /r Un/cele< Span id= "mathjax-span-20" class= "Mi" >ry / Home/run/celery/home/log/celery/%n.log "
       5. or celery Worker-detach.
    3. Stop it
    5. PS AUXWW | grep ' Celery worker ' | awk ' {print $} ' | Xargs kill-9
18. Methods for integration with flask
    1. After integration, Flask will act as producer to create and send a task to the broker, and the standalone worker process initiated at celery will get the task from the broker and execute, returning the result.
    2. Method of obtaining a task result asynchronously in flask: Add.delay (x, y), sometimes needing to be named after the parameter is passed or Add.apply_async (args= (x, y), countdown=30)
    3. Flask get
20. Boot issues after integration with flask
    1. Because the default routing_key of celery is set according to the import level of the producer in the code, the worker side should be aware at startup that its startup directory should be on the top-level directory of the project and that no one will appear keyerror.
22. Performance improvements: Eventlet and Greenlet

Official reference: http://docs.celeryproject.org/en/latest/userguide/index.html

A practical guide to celery