Python print scrapy spider crawl tree structure method _python

This article is an example of how Python prints the Scrapy spider crawl tree structure. Share to everyone for your reference. Specifically as follows:

The following code can be understood at a Glance scrapy crawl page structure, the call is very simple

#!/usr/bin/env python
import fileinput, re from
collections import defaultdict
def print_urls (Allurls, Referer, indent=0):
  urls = allurls[referer] for
  URL in URLs:
    print ' *indent + referer
    if URL in allurls:
      print_urls (allurls, URL, indent+2)
def main ():
  log_re = Re.compile (R ' <get (. *?) > (Referer: (. *?) \) ')
  allurls = defaultdict (list)
  for L in Fileinput.input ():
    m = Log_re.search (l)
    if m:
      URL, ref = M.groups ()
      allurls[ref] + = [url]
  print_urls (allurls, ' None ')
main ()

I hope this article will help you with your Python programming.