Write a web crawler in Python-start from scratch 2 Web site map crawler

General web site will have robots.txt files, in this file to allow web crawler access to the directory, also provides a directory to prohibit crawler access.
The reason to pay attention to this file is that access to the Forbidden directory will be banned from your IP address access

The following defines a Web site map crawler,

def crawl_sitemap (URL):
    # Download the sitemap file
    sitemap = Download (URL)
    # Extract the Sitemap links
    Links = Re.findall (' <loc> (. *?) </loc> ', sitemap)
    # Download each link
    For link in Links:
        html = Download (link)
        # Scrape HTML here
        # ...

Write a web crawler in Python-start from scratch 2 Web site map crawler