Further explore the use of Python's adorners

Adorners are quite extensive in python, and if you have used some Python web frames, then you must be familiar with the "route () decorator" in it, so let's look at a specific case today.

Let's simulate a scene, need you to grab a page, and then this page has a lot of URLs to be crawled separately, and after entering these sub-URLs, there is data to crawl. Simply put, we will look at the three level, then our code is as follows:

def func_top (URL):  data_dict= {}   #在页面上获取到子url  sub_urls = xxxx   data_list = [] for  it in Sub_urls:    Data_list.append (Func_sub (IT))   data_dict[' data ' = Data_list   return data_dict def func_sub (URL):  data_dict= {}   #在页面上获取到子url  bottom_urls = xxxx   data_list = [] for  it in Bottom_urls:    data_ List.append (Func_bottom (IT))   data_dict[' data ' = Data_list   return data_dict def func_bottom (URL):  # Retrieve Data  = xxxx  return

Func_top is the processing function of the upper page, func_sub is the processing function of the sub-page, Func_bottom is the processing function of the deepest page, Func_top will traverse the call func_sub,func_sub after fetching the URL of the child page.
If this is the case, the site you are trying to crawl may be extremely unstable, often not linked, resulting in data not being taken.
So this time you have two choices:
1. Stop after encountering an error and start again from the point of disconnection
2. Encountered an error continue, but to run again after, this time already have data do not want to go to the site to pull again, and only to pull the data not taken
The first scenario is basically impossible, because if the URL of someone else's site is adjusted, the location of your record is not valid. Then there is only the second solution, plainly, is to take the data that has been obtained to the cache, and so on when necessary, directly from the cache.
OK, the target is already there, how to achieve it?
If it is in C + +, this is a very troublesome thing, and the code written must be ugly, but fortunately, we use Python, and Python has a function decorator.
So the implementation of the scheme is also:
Defines an adorner that takes data directly from the cache if it was previously taken, and then pulls it from the site and caches it if it was not previously taken.
The code is as follows:

Import Osimport hashlib def deco_args_recent_cache (category= ' dumps '): ' Adorner, return the latest cache data ' ' Def deco_recent_cache (f UNC): Def func_wrapper (*args, **kargs): sig = _mk_cache_sig (*args, **kargs) data = _get_recent_cache (category , func.__name__, SIG) if data is not none:return data = func (*args, **kargs) if data are not N One: _set_recent_cache (category, func.__name__, SIG, data) return data return Func_wrapper return Deco_r Ecent_cache def _mk_cache_sig (*args, **kargs): "By passing in parameters, generate a unique identity ' ' Src_data = repr (args) + repr (Kargs) m = hashlib. MD5 (src_data) sig = M.hexdigest () return sig def _get_recent_cache (category, Func_name, sig): Full_file_path = '%s/%s/%     S '% (category, Func_name, SIG) if Os.path.isfile (Full_file_path): Return eval (file (Full_file_path, ' R '). Read ()) Else: Return None def _set_recent_cache (category, Func_name, SIG, data): Full_dir_path = '%s/%s '% (category, func_name) I F Not Os.path.isdir (full_dir_path): Os.makedirs (full_dir_path) Full_file_path = '%s/%s/%s '% (category, Func_name, sig) F = File (full_file_p Ath, ' w+ ') f.write (repr (data)) F.close ()

Then we just need to add deco_args_recent_cache to each func_top,func_sub,func_bottom.
Get! The biggest advantage of this is that, because top,sub,bottom, each layer will dump data, so for example, a sub-layer of data dump, is not going to his corresponding bottom layer, reducing a lot of overhead!
OK, that's it ~ life is short, I use python!

Note:

Python3 has natively supported this feature! The links are as follows:

Http://docs.python.org/py3k/whatsnew/3.2.html#functools

Recommended reading:

Https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize

