How to implement asynchronization of the distributed task huey of python

A lightweight task queue does not have powerful functions and related brokers. It focuses on lightweight tasks, and the code reading is also relatively simple.


About huey: (lighter than celery, better than mrq and rq !)

A lightweight alternative.

Written in python

No deps outside stdlib, doesn't redis (or roll your own backend)

Support for django

Supports:

Multi-threaded task execution

Scheduled execution at a given time

Periodic execution, like a crontab

Retrying tasks that fail

Task result storage


Installation:

The code is as follows:

Installing Huey can be installed very easily using pip.   Pip install huey Huey has no dependencies outside the standard library, but currently the only fully-implemented queue backend it ships with requires redis. To use the redis backend, you will need to install the python client.   Pip install redis Using git If you want to run the very latest, feel free to pull down the repo from github and install by hand.   Git clone https://github.com/coleifer/huey.git Cd huey Python setup. py install You can run the tests using the test-runner:   Python setup. py test

For more information about huey APIs, see the following sections.

The code is as follows:
From huey import RedisHuey, crontab   Huey = RedisHuey ('My-app', host = 'redis .myapp.com ')   @ Huey. task () Def add_numbers (a, B ): Return a + B   @ Huey. periodic_task (crontab (minute = '0', hour = '3 ')) Def nightly_backup (): Sync_all_data ()

Some cli parameters when juey is used as a woker.


Commonly used:

-L execution of log files.

-The number of wworkers, the value of-w is too large, and it must increase the processing capability of the task.

-P -- periodic: When the huey worker is started, he will find the task for crontab from tasks. py, and will send several threads to handle these tasks.

-N does not start the pre-cycle execution in crontab. Only when you trigger the execution will the tasks of the cycle week be executed.

-- Threads means you understand.
1

The code is as follows:

# Original: The following table lists the options available for the consumer as well as their default values.   -L, -- logfile Path to file used for logging. when a file is specified, by default Huey will use a rotating file handler (1 MB/chunk) with a maximum of 3 backups. you can attach your own handler (huey. logger) as well. the default loglevel is INFO. -V, -- verbose Verbose logging (equates to DEBUG level). If no logfile is specified and verbose is set, then the consumer will log to the console. This is very useful for testing/debugging. -Q, -- quiet Only log errors. The default loglevel for the consumer is INFO. -W, -- workers Number of worker threads, the default is 1 thread but for applications that have threads I/O bound tasks, increasing this number may lead to greater throughput. -P, -- periodic Indicate that this consumer process shocould start a thread dedicated to enqueueing "periodic" tasks (crontab-like functionality). This defaults to True, so shocould not need to be specified in practice. -N, -- no-periodic Indicate that this consumer process shocould not enqueue periodic tasks. -D, -- delay When using a "polling"-type queue backend, the amount of time to wait between polling the backend. Default is 0.1 seconds. -M, -- max-delay The maximum amount of time to wait between polling, if using weighted backoff. Default is 10 seconds. -B, -- backoff The amount to back-off when polling for results. Must be greater than one. Default is 1.15. -U, -- utc Indicates that the consumer shocould use UTC time for all tasks, crontabs and scheduling. Default is True, so in practice you shoshould not need to specify this option. -- Localtime Indicates that the consumer shocould use localtime for all tasks, crontabs and scheduling. Default is False. Examples   Running the consumer with 8 threads, a logfile for errors only, and a very short polling interval:   Huey_consumer.py my. app. huey-l/var/log/app. huey. log-w 8-B 1.1-m 1.0

Task queue huey uses redis to store queue tasks. Therefore, we need to install redis-server and redis-py in advance. The installation method will not be mentioned. Search for it by yourself.


First, create the linked instance of the huey:

The code is as follows:
# Config. py From huey import Huey From huey. backends. redis_backend import RedisBlockingQueue   Queue = RedisBlockingQueue ('test-queue ', host = 'localhost', port = 6379) Huey = Huey (queue)

Then there is the task, that is, who you want to bring to the task queue circle, which is represented by tasks. py like celey, rq, and mrq.

The code is as follows:
From config import huey # import the huey we instantiated in config. py     @ Huey. task () Def count_beans (num ): Print '-- counted % s beans --' % num

Another one is actually executed. Main. py is equivalent to the producer, and tasks. py is equivalent to the consumer relationship. Main. py is responsible for feeding data.

The code is as follows:

Main. py From config import huey # import our "huey" object From tasks import count_beans # import our task     If _ name _ = '_ main __': Beans = raw_input ('How many beans? ') Count_beans (int (beans )) Print 'enqueued job to count % s beans '% beans Ensure you have Redis running locally Ensure you have installed huey Start the consumer: huey_consumer.py main. huey (notice this is "main. huey" and not "config. huey "). Run the main program: python main. py

Like celery and rq, the result is obtained in your config. in the py or main code, specify the storage method. Currently, huey only supports redis, but it is enough for its features and size!


Just a few words. Import the RedisDataStore database and declare the storage address.

The code is as follows:
From huey import Huey From huey. backends. redis_backend import RedisBlockingQueue From huey. backends. redis_backend import RedisDataStore # ADD THIS LINE     Queue = RedisBlockingQueue ('test-queue ', host = 'localhost', port = 6379) Result_store = RedisDataStore ('result', host = 'localhost', port = 6379) # ADDED   Huey = Huey (queue, result_store = result_store) # ADDED result store

At this time, when we try ipython again, we will find that we can get tasks. the return value in py is actually in main. when it is obtained in py, it is still obtained from redis through uuid.

The code is as follows:
>>> From main import count_beans >>> Res = count_beans (100) >>> Res # what is "res "? <Huey. api. AsyncData object at 0xb7471a4c> >>> Res. get () # get the result of this task 'Counted 100 beans'

Huey also supports delayed execution of celey and crontab. These functions are very important. You can customize the priority or no longer need to use linux crontab.


The usage is very simple. Just add another delay time. After reading the source code of huey, it will be executed immediately by default. Of course, it depends on whether your thread is in the state to be executed.

The code is as follows:
>>> Import datetime >>> Res = count_beans.schedule (args = (100,), delay = 60) >>> Res <Huey. api. AsyncData object at 0xb72915ec> >>> Res. get () # this returns None, no data is ready >>> Res. get () # still no data... >>> Res. get (blocking = True) # OK, let's just block until its ready 'Counted 100 beans'

Another retry introduction, huey also has retry, which is very practical. If you see the introduction of the celery retry mechanism in my previous article, you can also understand what huey is. Yes, he actually made a decorator in front of the specific function in tasks. There is a func try exception retry logic in the decorator. Everyone knows.

The code is as follows:
# Tasks. py From datetime import datetime   From config import huey   @ Huey. task (retries = 3, retry_delay = 10) Def try_thrice (): Print 'trying... % s' % datetime. now () Raise Exception ('nope ')

Huey is a chance for you to repent ~ That is to say, if you want to cancel another scheduled task of deley, you can simply revoke it.

The code is as follows:
# Count some beans Res = count_beans (10000000)   Res. revoke () The same applies to tasks that are scheduled in the future:   Res = count_beans.schedule (args = (100000,), eta = in_the_future) Res. revoke ()   @ Huey. task (crontab (minute = '*')) Def print_time (): Print datetime. now ()

Task ()-transparent decorators make your functions more elegant.

Periodic_task ()-this is a periodic task.

Crontab ()-periodic task with crontab when the worker is started.

BaseQueue-task queue

BaseDataStore-after the task is executed, you can insert the result. BAseDataStore can be rewritten by yourself.

 

The official git library of huey provides the relevant test code:


Main. py

The code is as follows:
From config import huey From tasks import count_beans     If _ name _ = '_ main __': Beans = raw_input ('How many beans? ') Count_beans (int (beans )) Print ('enqueued job to count % s beans '% beans)

Tasks. py

The code is as follows:

Import random Import time From huey import crontab   From config import huey     @ Huey. task () Def count_beans (num ): Print "start ..." Print ('-- counted % s beans --' % num) Time. sleep (3) Print "end ..." Return 'counted % s beans '% num   @ Huey. periodic_task (crontab (minute = '*/5 ')) Def every_five_mins (): Print ('consumer prints this every 5 mins ')   @ Huey. task (retries = 3, retry_delay = 10) Def try_thrice (): If random. randint (1, 3) = 1: Print ('OK ') Else: Print ('about to fail, will retry in 10 seconds ') Raise Exception ('crap something went wrong ')   @ Huey. task () Def slow (n ): Time. sleep (n) Print ('slept % s' % n)

Run. sh

The code is as follows:
#! /Bin/bash Echo "huey consumer" Echo "-------------" Echo "In another terminal, run 'Python main. Py '" Echo "Stop the consumer using Ctrl + C" PYTHONPATH =.: $ PYTHONPATH Python.../huey/bin/huey_consumer.py main. huey -- threads = 2 =>

We can clone the huey code library first. There is a directory of examples. We can see that it supports django, but this is not the focus!

The code is as follows:

[Xiaorui @ devops/tmp] $ git clone https://github.com/coleifer/huey.git Cloning into 'huine '... Remote: Counting objects: 1423, done. Remote: Compressing objects: 100% (9/9), done. Grouping objects: 34% (497/1423), 388.00 KiB | 29.00 KiB/s   Grouping objects: 34% (498/1423), 628.00 KiB | 22.00 KiB/s     Remote: Total 1423 (delta 0), reused 0 (delta 0) Grouping objects: 100% (1423/1423), 2.24 MiB | 29.00 KiB/s, done. Resolving deltas: 100% (729/729), done. Checking connectivity... done. [Xiaorui @ devops/tmp] $ cd huey/examples/simple [Xiaorui @ devops simple (master)] $ ll Total 40 -Rw-r -- 1 xiaorui wheel 79B 9 8 README -Rw-r -- 1 xiaorui wheel 0B 9 8 08:49 _ init _. py -Rw-r -- 1 xiaorui wheel 56B 9 8 08:49 config. py -Rwxr-xr-x 1 xiaorui wheel 227B 9 8 08:49 cons. sh -Rw-r -- 1 xiaorui wheel 205B 9 8 08:49 main. py -Rw-r -- 1 xiaorui wheel 607B 9 8 tasks. py [Xiaorui @ devops simple (master)] $