Python print scrapy spider crawl Tree structure method

In this paper, we describe the method of Python to print scrapy spider tree structure. Share to everyone for your reference. Specific as follows:

Through the following code can be at a glance to know scrapy crawl page structure, call is very simple

#!/usr/bin/env pythonimport fileinput, refrom collections import Defaultdictdef print_urls (Allurls, Referer, indent=0):  URLs = allurls[referer] for  URL in URLs:    print "*indent + referer    if URL in allurls:      print_urls (All URLs, URLs, indent+2) def main ():  log_re = Re.compile (R '
 
 
  
   \ (referer: (. *?) \) ')  allurls = defaultdict (list)  for L in Fileinput.input ():    m = Log_re.search (l)    if m:      URL, ref = M.groups ()      allurls[ref] + = [url]  print_urls (allurls, ' None ') Main (
 )
 

Hopefully this article will help you with Python programming.

