[Python] Understanding adorners in a scene

Originally I also through check data, to learn python decorator, but the effect is not good. Because there is no contact with the need to use the adorner scene, so

All the information is only on paper, but today happened to see vimer this article: http://www.vimer.cn/2011/04/python%E8%A3%85%E9%A5%B0%E5%99%A8%E7%9A%84%E4%B8 %80%e4%b8%aa%e5%a6%99%e7%94%a8.html

We'll follow the idea of this article to understand the python decorator in the scene

One of the scenarios is: when crawling data, the target site is unstable. Crawlers may be unusually crawling when fetching data.

Solution:

idea: To crawl each method crawling data, all exist on the hard disk, the storage rules are as follows:

file_path:/method_name/

file_name:MD5_number.txt

Explanation:Md5_number is a unique identifier computed by the method name and parameters passed in (Searching for MD5) to do so well , without having to crawl the data down to determine whether or not to crawl.

finally: If the method crawls the data of the destination URL, the method is not executed. Run the last few more times this reptile script will do!

The crawler calls three methods to design a common method: the Unique identifier (MD5_NUMBE) is computed by MD5 based on the method name and the parameters passed in each method. Then locate the file named Md5_numbe in the File_path directory and store the data according to the storage rules if there is no such file in File_path. If it is found, it is not stored on the hard disk and is returned directly to data for the next method.

————————————

End of Scene

++++++++++++

In the scenario described above, where is the decorator?

That's right! That's the generic method that needs to be implemented with decorators, why? Step by step to

All things in Python are objects. Method can be used as an object to pass!

1. What is a method?

Pass in the parameters and then perform a series of operations on the passed in parameters.

2. What is an advanced method

The method accepts the parameter as a method, that's all.

3. What is closures

When method 1 has a method 2, Method 2 calls the method 1 receives the parameter, the final result returns method 2. The technology that provides this implementation is called closures.

After all, the decorator is a method, and the passed-in parameter is a method, and a series of operations are performed on the incoming method in the decorator (the closure is used here). Thus, the new operation is added on the basis of not modifying the original method. For example, in the previous scenario, the test has no repeated crawl data.

Code snippets in a crawl script

Func_top is the processing function of the upper page, func_sub is the processing function of the sub-page, Func_bottom is the processing function of the deepest page, Func_top will traverse the call func_sub,func_sub after fetching the URL of the child page.

1 deffunc_top (URL):2data_dict= {}3  4     #get a child URL on a page5Sub_urls =xxxx6  7Data_list = []8      forItinchSub_urls:9 Data_list.append (Func_sub (IT))Ten   Onedata_dict['Data'] =data_list A   -     returndata_dict -   the deffunc_sub (URL): -data_dict= {} -   -     #get a child URL on a page +Bottom_urls =xxxx -   +Data_list = [] A      forItinchBottom_urls: at Data_list.append (Func_bottom (IT)) -   -data_dict['Data'] =data_list -   -     returndata_dict -   in deffunc_bottom (URL): -     #Get Data todata =xxxx +     returnData

So the implementation of the scheme is also:
Defines an adorner that, if previously taken, takes the data from the cache (File_path) and, if not previously, pulls it from the site and caches it.

1 ImportOS2 ImportHashlib3  4 defDeco_args_recent_cache (category='dumps'):5     " "6 adorner, returning data from the latest cache7     " "8     defDeco_recent_cache (func):9         defFunc_wrapper (*args, * *Kargs):Tensig = _mk_cache_sig (*args, * *Kargs) Onedata = _get_recent_cache (category, Func.)__name__, SIG) A             ifData is  notNone: -                 returnData -   theData = func (*args, * *Kargs) -             ifData is  notNone: -_set_recent_cache (category, Func.)__name__, SIG, data) -             returnData +   -         returnFunc_wrapper +   A     returnDeco_recent_cache at   - def_mk_cache_sig (*args, * *Kargs): -     " " - generate unique identities by passing in parameters -     " " -Src_data = repr (args) +repr (Kargs) inm =hashlib.md5 (src_data) -sig =m.hexdigest () to     returnSig +   - def_get_recent_cache (category, Func_name, SIG): theFull_file_path ='%s/%s/%s'%(category, Func_name, SIG) *     ifOs.path.isfile (full_file_path): $         returnEval (File (Full_file_path,'R'). Read ())Panax Notoginseng     Else: -         returnNone the   + def_set_recent_cache (category, Func_name, SIG, data): AFull_dir_path ='%s/%s'%(category, Func_name) the     if  notOs.path.isdir (full_dir_path): + os.makedirs (Full_dir_path) -   $Full_file_path ='%s/%s/%s'%(category, Func_name, SIG) $f = File (Full_file_path,'w+') - F.write (repr (data)) -F.close ()

Finally, we just need to add deco_args_recent_cache to each func_top,func_sub,func_bottom.

Resources:

Plain understanding closure:https://serholiu.com/python-closures

Decorator:http://jnotes.googlecode.com/svn/trunk/Notes/NotesOnPythonLearning/Python_decorator.html

[Python] Understanding adorners in a scene