Further explore the use of Python decorators, explore python decorations

Further explore the use of Python decorators, explore python decorations

Decorator is widely used in python. If you have used some python web frameworks, you must be familiar with the route () decorator, let's look at a specific case today.

To simulate a scenario, you need to capture a page. Then there are many URLs on the page to capture separately. After entering these suburls, there is still data to capture. Simply put, let's look at the three layers. Then our code is as follows:
 

Def func_top (url): data_dict ={}# obtain the suburl sub_urls = xxxx data_list = [] for it in sub_urls: data_list.append (func_sub (it) on the page )) data_dict ['data'] = data_list return data_dict def func_sub (url): data_dict ={}# obtain the suburl bottom_urls = xxxx data_list = [] for it in bottom_urls on the page: data_list.append (func_bottom (it) data_dict ['data'] = data_list return data_dict def func_bottom (url): # retrieve data = xxxx return data

Func_top is the processing function of the upper page, func_sub is the processing function of the sub page, func_bottom is the processing function of the deepest page, func_top will traverse and call func_sub after obtaining the url of the sub page, the same is true for func_sub.
Under normal circumstances, this is indeed enough to meet the needs, but the website you want to crawl may be extremely unstable and often cannot be linked, resulting in Data not available.
At this time, you have two options:
1. Stop when an error occurs, and re-run from the disconnected location
2. If an error occurs, continue, but run it again later. At this time, the existing data does not want to be pulled again on the website, but only the data that has not been retrieved.
The first scheme is basically not feasible, because if the URLs of other websites are adjusted in order, the recorded location will be invalid. There is only the second solution. To put it bluntly, we need to cache the obtained data and retrieve it directly from the cache when necessary.
OK. The target already exists. How can this problem be achieved?
If it is in C ++, this is a very troublesome thing, and the code written must be ugly, but fortunately, we use python, python has a modifier for functions.
So the implementation scheme is as follows:
Define a decorator. If the data is obtained before, the data in the cache is directly obtained. If the data is not obtained before, the data is pulled from the website and saved to the cache.
The Code is as follows:
 

Import osimport hashlib def deco_args_recent_cache (category = 'dumps'): ''', returns the latest cache data ''' def deco_recent_cache (func): def func_wrapper (* args, ** kargs): sig = _ mk_cache_sig (* args, ** kargs) data = _ get_recent_cache (category, func. _ name __, sig) if data is not None: return data = func (* args, ** kargs) if data is not None: _ set_recent_cache (category, func. _ name __, sig, data) return data return func_wrapper return deco_recent_cache def _ mk_cache_sig (* args, ** kargs): ''' parameters are passed in, generate the unique identifier ''' src_data = repr (args) + repr (kargs) m = hashlib. md5 (src_data) sig = m. hexdigest () return sig def _ get_recent_cache (category, func_name, sig): full_file_path = '% s/% s' % (category, func_name, sig) if OS. path. isfile (full_file_path): return eval (file (full_file_path, 'R '). read () else: return None def _ set_recent_cache (category, func_name, sig, data): full_dir_path = '% s/% s' % (category, func_name) if not OS. path. isdir (full_dir_path): OS. makedirs (full_dir_path) full_file_path = '% s/% s' % (category, func_name, sig) f = file (full_file_path, 'W +') f. write (repr (data) f. close ()

Then, we only need to add deco_args_recent_cache to each func_top, func_sub, and func_bottom ~~
Done! The biggest advantage of this is that, because top, sub, bottom, and each layer will dump data, for example, after a sub layer data dump, it will not go to the corresponding bottom layer at all, reducing a lot of Overhead!
OK ~ My life is short. I use python!

Note:

Python3 already supports this native function! The link is as follows:

Http://docs.python.org/py3k/whatsnew/3.2.html#functools

Recommended reading:

Https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize