How to implement asynchronization of the distributed task huey of python _ PHP Tutorial

Source: Internet
Author: User
How to implement asynchronous tasks in the distributed task huey of python. How to implement asynchronous tasks for the python distributed task huey this article will share a python lightweight task queue program, he can enable the distributed task huey of python to implement asynchronization. how can the distributed task huey of python implement asynchronization?

In this article, we will share a python lightweight task queue program, which enables the python distributed task huey to implement asynchronous tasks. if you are interested, you can see it.

A lightweight task queue does not have powerful functions and related brokers. it focuses on lightweight tasks, and the code reading is also relatively simple.


About huey: (lighter than celery, better than mrq and rq !)

A lightweight alternative.

Written in python

No deps outside stdlib, doesn't redis (or roll your own backend)

Support for django

Supports:

Multi-threaded task execution

Scheduled execution at a given time

Periodic execution, like a crontab

Retrying tasks that fail

Task result storage


Installation:

The code is as follows:
Installing
Huey can be installed very easily using pip.

Pip install huey
Huey has no dependencies outside the standard library, but currently the only fully-implemented queue backend it ships with requires redis. To use the redis backend, you will need to install the python client.

Pip install redis
Using git
If you want to run the very latest, feel free to pull down the repo from github and install by hand.

Git clone https://github.com/coleifer/huey.git
Cd huey
Python setup. py install
You can run the tests using the test-runner:

Python setup. py test




For more information about huey APIs, see the following sections.

The code is as follows:
From huey import RedisHuey, crontab

Huey = RedisHuey ('My-app', host = 'redis .myapp.com ')

@ Huey. task ()
Def add_numbers (a, B ):
Return a + B

@ Huey. periodic_task (crontab (minute = '0', hour = '3 '))
Def nightly_backup ():
Sync_all_data ()




Some cli parameters when juey is used as a woker.


Commonly used:

-L execution of log files.

-The number of wworkers, the value of-w is too large, and it must increase the processing capability of the task.

-P -- periodic: When the huey worker is started, he will find the task for crontab from tasks. py, and will send several threads to handle these tasks.

-N does not start the pre-cycle execution in crontab. only when you trigger the execution will the tasks of the cycle week be executed.

-- Threads means you understand.
1

The code is as follows:
# Original:
The following table lists the options available for the consumer as well as their default values.

-L, -- logfile
Path to file used for logging. when a file is specified, by default Huey will use a rotating file handler (1 MB/chunk) with a maximum of 3 backups. you can attach your own handler (huey. logger) as well. the default loglevel is INFO.
-V, -- verbose
Verbose logging (equates to DEBUG level). If no logfile is specified and verbose is set, then the consumer will log to the console. This is very useful for testing/debugging.
-Q, -- quiet
Only log errors. The default loglevel for the consumer is INFO.
-W, -- workers
Number of worker threads, the default is 1 thread but for applications that have threads I/O bound tasks, increasing this number may lead to greater throughput.
-P, -- periodic
Indicate that this consumer process shocould start a thread dedicated to enqueueing "periodic" tasks (crontab-like functionality). This defaults to True, so shocould not need to be specified in practice.
-N, -- no-periodic
Indicate that this consumer process shocould not enqueue periodic tasks.
-D, -- delay
When using a "polling"-type queue backend, the amount of time to wait between polling the backend. Default is 0.1 seconds.
-M, -- max-delay
The maximum amount of time to wait between polling, if using weighted backoff. Default is 10 seconds.
-B, -- backoff
The amount to back-off when polling for results. Must be greater than one. Default is 1.15.
-U, -- utc
Indicates that the consumer shocould use UTC time for all tasks, crontabs and scheduling. Default is True, so in practice you shoshould not need to specify this option.
-- Localtime
Indicates that the consumer shocould use localtime for all tasks, crontabs and scheduling. Default is False.
Examples

Running the consumer with 8 threads, a logfile for errors only, and a very short polling interval:

Huey_consumer.py my. app. huey-l/var/log/app. huey. log-w 8-B 1.1-m 1.0





Task queue huey uses redis to store queue tasks. Therefore, we need to install redis-server and redis-py in advance. The installation method will not be mentioned. search for it by yourself.


First, create the linked instance of the huey:

The code is as follows:
# Config. py
From huey import Huey
From huey. backends. redis_backend import RedisBlockingQueue

Queue = RedisBlockingQueue ('test-queue ', host = 'localhost', port = 6379)
Huey = Huey (queue)


Then there is the task, that is, who you want to bring to the task queue circle, which is represented by tasks. py like celey, rq, and mrq.

The code is as follows:
From config import huey # import the huey we instantiated in config. py


@ Huey. task ()
Def count_beans (num ):
Print '-- counted % s beans --' % num




Another one is actually executed. Main. py is equivalent to the producer, and tasks. py is equivalent to the consumer relationship. Main. py is responsible for feeding data.

The code is as follows:
Main. py
From config import huey # import our "huey" object
From tasks import count_beans # import our task


If _ name _ = '_ main __':
Beans = raw_input ('How many beans? ')
Count_beans (int (beans ))
Print 'enqueued job to count % s beans '% beans


Ensure you have Redis running locally

Ensure you have installed huey

Start the consumer: huey_consumer.py main. huey (notice this is "main. huey" and not "config. huey ").

Run the main program: python main. py




Like celery and rq, the result is obtained in your config. in the py or main code, specify the storage method. Currently, huey only supports redis, but it is enough for its features and size!


Just a few words. import the RedisDataStore database and declare the storage address.

The code is as follows:
From huey import Huey
From huey. backends. redis_backend import RedisBlockingQueue
From huey. backends. redis_backend import RedisDataStore # ADD THIS LINE


Queue = RedisBlockingQueue ('test-queue ', host = 'localhost', port = 6379)
Result_store = RedisDataStore ('result', host = 'localhost', port = 6379) # ADDED

Huey = Huey (queue, result_store = result_store) # ADDED result store




At this time, when we try ipython again, we will find that we can get tasks. the return value in py is actually in main. when it is obtained in py, it is still obtained from redis through uuid.

The code is as follows:
>>> From main import count_beans
>>> Res = count_beans (100)
>>> Res # what is "res "?

>>> Res. get () # get the result of this task
'Counted 100 beans'




Huey also supports delayed execution of celey and crontab. These functions are very important. you can customize the priority or no longer need to use linux crontab.


The usage is very simple. just add another delay time. after reading the source code of huey, it will be executed immediately by default. Of course, it depends on whether your thread is in the state to be executed.

The code is as follows:
>>> Import datetime
>>> Res = count_beans.schedule (args = (100,), delay = 60)
>>> Res

>>> Res. get () # this returns None, no data is ready
>>> Res. get () # still no data...
>>> Res. get (blocking = True) # OK, let's just block until its ready
'Counted 100 beans'





Another retry introduction, huey also has retry, which is very practical. If you see the introduction of the celery retry mechanism in my previous article, you can also understand what huey is. Yes, he actually made a decorator in front of the specific function in tasks. there is a func try exception retry logic in the decorator. Everyone knows.

The code is as follows:
# Tasks. py
From datetime import datetime

From config import huey

@ Huey. task (retries = 3, retry_delay = 10)
Def try_thrice ():
Print 'trying... % s' % datetime. now ()
Raise Exception ('nope ')





Huey is a chance for you to repent ~ That is to say, if you want to cancel another scheduled task of deley, you can simply revoke it.

The code is as follows:
# Count some beans
Res = count_beans (10000000)

Res. revoke ()
The same applies to tasks that are scheduled in the future:

Res = count_beans.schedule (args = (100000,), eta = in_the_future)
Res. revoke ()

@ Huey. task (crontab (minute = '*'))
Def print_time ():
Print datetime. now ()


Task ()-transparent decorators make your functions more elegant.

Periodic_task ()-this is a periodic task.

Crontab ()-periodic task with crontab when the worker is started.

BaseQueue-task queue

BaseDataStore-after the task is executed, you can insert the result. BAseDataStore can be rewritten by yourself.


The official git library of huey provides the relevant test code:


Main. py

The code is as follows:
From config import huey
From tasks import count_beans


If _ name _ = '_ main __':
Beans = raw_input ('How many beans? ')
Count_beans (int (beans ))
Print ('enqueued job to count % s beans '% beans)




Tasks. py

The code is as follows:
Import random
Import time
From huey import crontab

From config import huey


@ Huey. task ()
Def count_beans (num ):
Print "start ..."
Print ('-- counted % s beans --' % num)
Time. sleep (3)
Print "end ..."
Return 'counted % s beans '% num

@ Huey. periodic_task (crontab (minute = '*/5 '))
Def every_five_mins ():
Print ('consumer prints this every 5 mins ')

@ Huey. task (retries = 3, retry_delay = 10)
Def try_thrice ():
If random. randint (1, 3) = 1:
Print ('OK ')
Else:
Print ('about to fail, will retry in 10 seconds ')
Raise Exception ('crap something went wrong ')

@ Huey. task ()
Def slow (n ):
Time. sleep (n)
Print ('slept % s' % n)




Run. sh

The code is as follows:
#! /Bin/bash
Echo "huey consumer"
Echo "-------------"
Echo "In another terminal, run 'Python main. py '"
Echo "Stop the consumer using Ctrl + C"
PYTHONPATH =.: $ PYTHONPATH
Python.../huey/bin/huey_consumer.py main. huey -- threads = 2

=>



We can clone the huey code library first. There is a directory of examples. we can see that it supports django, but this is not the focus!

The code is as follows:
[Xiaorui @ devops/tmp] $ git clone https://github.com/coleifer/huey.git
Cloning into 'huine '...
Remote: Counting objects: 1423, done.
Remote: Compressing objects: 100% (9/9), done.
Grouping objects: 34% (497/1423), 388.00 KiB | 29.00 KiB/s

Grouping objects: 34% (498/1423), 628.00 KiB | 22.00 KiB/s


Remote: Total 1423 (delta 0), reused 0 (delta 0)
Grouping objects: 100% (1423/1423), 2.24 MiB | 29.00 KiB/s, done.
Resolving deltas: 100% (729/729), done.
Checking connectivity... done.
[Xiaorui @ devops/tmp] $ cd huey/examples/simple
[Xiaorui @ devops simple (master)] $ ll
Total 40
-Rw-r -- 1 xiaorui wheel 79B 9 8 README
-Rw-r -- 1 xiaorui wheel 0B 9 8 08:49 _ init _. py
-Rw-r -- 1 xiaorui wheel 56B 9 8 08:49 config. py
-Rwxr-xr-x 1 xiaorui wheel 227B 9 8 08:49 cons. sh
-Rw-r -- 1 xiaorui wheel 205B 9 8 08:49 main. py
-Rw-r -- 1 xiaorui wheel 607B 9 8 tasks. py
[Xiaorui @ devops simple (master)] $



In this article, we will share a python lightweight task queue program, which enables the distributed task huey of python to be asynchronous...

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.