python的分布式任務huey如何?非同步化任務講解

一個輕型的任務隊列，功能和相關的broker沒有celery強大，重在輕型，而且代碼讀起來也比較的簡單。


關於huey的介紹:  （比celery輕型，比mrq、rq要好用 ！）

a lightweight alternative.

    written in python

    no deps outside stdlib, except redis (or roll your own backend)

    support for django

supports:

    multi-threaded task execution

    scheduled execution at a given time

    periodic execution, like a crontab

    retrying tasks that fail

    task result storage


安裝:

代碼如下

複製代碼

Installing huey can be installed very easily using pip.   pip install huey huey has no dependencies outside the standard library, but currently the only fully-implemented queue backend it ships with requires redis. To use the redis backend, you will need to install the python client.   pip install redis Using git If you want to run the very latest, feel free to pull down the repo from github and install by hand.   git clone https://github.com/coleifer/huey.git cd huey python setup.py install You can run the tests using the test-runner:   python setup.py test

關於huey的api，下面有詳細的介紹及參數介紹的。

代碼如下	複製代碼
from huey import RedisHuey, crontab   huey = RedisHuey('my-app', host='redis.myapp.com')   @huey.task() def add_numbers(a, b):     return a + b   @huey.periodic_task(crontab(minute='0', hour='3')) def nightly_backup():     sync_all_data()

juey作為woker的時候，一些cli參數。


常用的是: 

-l                  關於記錄檔的執行 。

-w                 workers的數目，-w的數值大了，肯定是增加任務的處理能力

-p --periodic     啟動huey worker的時候，他會從tasks.py裡面找到 需要crontab的任務，會派出幾個線程專門處理這些事情。

-n                  不啟動關於crontab裡面的預周期執行，只有你觸發的時候，才會執行循環星期的任務。

--threads   意思你懂的。
1

代碼如下

複製代碼

# 原文:     The following table lists the options available for the consumer as well as their default values.   -l, --logfile Path to file used for logging. When a file is specified, by default Huey will use a rotating file handler (1MB / chunk) with a maximum of 3 backups. You can attach your own handler (huey.logger) as well. The default loglevel is INFO. -v, --verbose Verbose logging (equates to DEBUG level). If no logfile is specified and verbose is set, then the consumer will log to the console. This is very useful for testing/debugging. -q, --quiet Only log errors. The default loglevel for the consumer is INFO. -w, --workers Number of worker threads, the default is 1 thread but for applications that have many I/O bound tasks, increasing this number may lead to greater throughput. -p, --periodic Indicate that this consumer process should start a thread dedicated to enqueueing “periodic” tasks (crontab-like functionality). This defaults to True, so should not need to be specified in practice. -n, --no-periodic Indicate that this consumer process should not enqueue periodic tasks. -d, --delay When using a “polling”-type queue backend, the amount of time to wait between polling the backend. Default is 0.1 seconds. -m, --max-delay The maximum amount of time to wait between polling, if using weighted backoff. Default is 10 seconds. -b, --backoff The amount to back-off when polling for results. Must be greater than one. Default is 1.15. -u, --utc Indicates that the consumer should use UTC time for all tasks, crontabs and scheduling. Default is True, so in practice you should not need to specify this option. --localtime Indicates that the consumer should use localtime for all tasks, crontabs and scheduling. Default is False. Examples   Running the consumer with 8 threads, a logfile for errors only, and a very short polling interval:   huey_consumer.py my.app.huey -l /var/log/app.huey.log -w 8 -b 1.1 -m 1.0

任務隊列huey 是靠著redis來實現queue的任務儲存，所以需要咱們提前先把redis-server和redis-py都裝好。 安裝的方法就不說了，自己搜搜吧。


我們首先建立下huey的連結執行個體 :

代碼如下	複製代碼
# config.py from huey import Huey from huey.backends.redis_backend import RedisBlockingQueue   queue = RedisBlockingQueue('test-queue', host='localhost', port=6379) huey = Huey(queue)

然後就是關於任務的，也就是你想讓誰到任務隊列這個圈子裡面，和celey、rq，mrq一樣，都是用tasks.py表示的。

代碼如下	複製代碼
from config import huey # import the huey we instantiated in config.py     @huey.task() def count_beans(num):     print '-- counted %s beans --' % num

再來一個真正去執行的 。  main.py 相當於生產者，tasks.py相當於消費者的關係。  main.py負責喂資料。

代碼如下

複製代碼

main.py from config import huey  # import our "huey" object from tasks import count_beans  # import our task     if __name__ == '__main__':     beans = raw_input('How many beans? ')     count_beans(int(beans))     print 'Enqueued job to count %s beans' % beans Ensure you have Redis running locally Ensure you have installed huey Start the consumer: huey_consumer.py main.huey (notice this is “main.huey” and not “config.huey”). Run the main program: python main.py

和celery、rq一樣，他的結果擷取是需要在你的config.py或者主代碼裡面指明他的儲存的方式，現在huey還僅僅是支援redis，但相對他的特點和體積，這已經很足夠了 ！


只是那幾句話而已，匯入RedisDataStore庫，申明下儲存的地址。

代碼如下	複製代碼
from huey import Huey from huey.backends.redis_backend import RedisBlockingQueue from huey.backends.redis_backend import RedisDataStore  # ADD THIS LINE     queue = RedisBlockingQueue('test-queue', host='localhost', port=6379) result_store = RedisDataStore('results', host='localhost', port=6379)  # ADDED   huey = Huey(queue, result_store=result_store) # ADDED result store

這個時候，我們在ipython再次去嘗試的時候，會發現可以擷取到tasks.py裡面的return值了 其實你在main.py裡面擷取的時候，他還是通過uuid從redis裡面取出來的。

代碼如下	複製代碼
>>> from main import count_beans >>> res = count_beans(100) >>> res  # what is "res" ? <huey.api.AsyncData object at 0xb7471a4c> >>> res.get()  # get the result of this task 'Counted 100 beans'

huey也是支援celey的順延強制和crontab的功能 。  這些功能很是重要，可以自訂的優先順序或者不用再藉助linux本身的crontab。


用法很簡單，多加一個delay的時間就行了，看了下huey的源碼，他預設是立馬執行的。當然還是要看你的線程是否都是待執行的狀態了。

代碼如下	複製代碼
>>> import datetime >>> res = count_beans.schedule(args=(100,), delay=60) >>> res <huey.api.AsyncData object at 0xb72915ec> >>> res.get()  # this returns None, no data is ready >>> res.get()  # still no data... >>> res.get(blocking=True)  # ok, let's just block until its ready 'Counted 100 beans'

再來一個重試retry的介紹，huey也是有retry,這個很是實用的東西。 如果大家有看到我的上面文章關於celery重試機制的介紹，應該也能明白huey是個怎麼個回事了。  是的，他其實也是在tasks裡具體函數的前面做了裝飾器，裝飾器裡面有個func try 異常重試的邏輯 。 大家懂的。

代碼如下	複製代碼
# tasks.py from datetime import datetime   from config import huey   @huey.task(retries=3, retry_delay=10) def try_thrice():     print 'trying....%s' % datetime.now()     raise Exception('nope')

huey是給你反悔的機會餓 ~  也就是說，你做了deley的計劃任務後，如果你又想取消，那好看，直接revoke就可以了。

代碼如下	複製代碼
# count some beans res = count_beans(10000000)   res.revoke() The same applies to tasks that are scheduled in the future:   res = count_beans.schedule(args=(100000,), eta=in_the_future) res.revoke()   @huey.task(crontab(minute='*')) def print_time():     print datetime.now()

task() - 透明的裝飾器，讓你的函數變得優美點。

periodic_task() - 這個是周期性的任務

crontab() - 啟動worker的時候，附帶的crontab的周期任務。

BaseQueue - 任務隊列

BaseDataStore - 任務執行後，可以把 結果塞入進去。  BAseDataStore可以自己重寫。

 

官方的huey的git庫裡面是提供了相關的測試代碼的：


main.py

代碼如下	複製代碼
from config import huey from tasks import count_beans     if __name__ == '__main__':     beans = raw_input('How many beans? ')     count_beans(int(beans))     print('Enqueued job to count %s beans' % beans)

tasks.py

代碼如下

複製代碼

import random import time from huey import crontab   from config import huey     @huey.task() def count_beans(num):     print "start..."     print('-- counted %s beans --' % num)     time.sleep(3)     print "end..."     return 'Counted %s beans' % num   @huey.periodic_task(crontab(minute='*/5')) def every_five_mins():     print('Consumer prints this every 5 mins')   @huey.task(retries=3, retry_delay=10) def try_thrice():     if random.randint(1, 3) == 1:         print('OK')     else:         print('About to fail, will retry in 10 seconds')         raise Exception('Crap something went wrong')   @huey.task() def slow(n):     time.sleep(n)     print('slept %s' % n)

run.sh

代碼如下	複製代碼
#!/bin/bash echo "HUEY CONSUMER" echo "-------------" echo "In another terminal, run 'python main.py'" echo "Stop the consumer using Ctrl+C" PYTHONPATH=.:$PYTHONPATH python ../../huey/bin/huey_consumer.py main.huey --threads=2 =>

咱們可以先clone下huey的程式碼程式庫。 裡面有個examples例子目錄，可以看到他是支援django的，但是這不是重點 ！

代碼如下

複製代碼

[xiaorui@devops /tmp ]$ git clone https://github.com/coleifer/huey.git Cloning into 'huey'... remote: Counting objects: 1423, done. remote: Compressing objects: 100% (9/9), done. Receiving objects:  34% (497/1423), 388.00 KiB | 29.00 KiB/s   KiB/s   Receiving objects:  34% (498/1423), 628.00 KiB | 22.00 KiB/s     remote: Total 1423 (delta 0), reused 0 (delta 0) Receiving objects: 100% (1423/1423), 2.24 MiB | 29.00 KiB/s, done. Resolving deltas: 100% (729/729), done. Checking connectivity... done. [xiaorui@devops /tmp ]$cd huey/examples/simple [xiaorui@devops simple (master)]$ ll total 40 -rw-r--r--  1 xiaorui  wheel    79B  9  8 08:49 README -rw-r--r--  1 xiaorui  wheel     0B  9  8 08:49 __init__.py -rw-r--r--  1 xiaorui  wheel    56B  9  8 08:49 config.py -rwxr-xr-x  1 xiaorui  wheel   227B  9  8 08:49 cons.sh -rw-r--r--  1 xiaorui  wheel   205B  9  8 08:49 main.py -rw-r--r--  1 xiaorui  wheel   607B  9  8 08:49 tasks.py [xiaorui@devops simple (master)]$