Python and web crawler

1, the definition of reptiles

Crawler: A program that automatically crawls Internet data.

2, Crawler's main frameThe main framework of the crawler, as shown, the crawler terminal through the URL manager to get the URL to crawl URLs, if there is a URL manager to crawl URL link, crawler scheduler called the Web page downloader download the corresponding page, It then invokes the Web page parser to parse the page and adds a new URL to the URL manager that will have the value of the data output. 3. The sequence diagram of the crawler
4. URL ManagerThe URL Manager manages the collection of URLs to be crawled and the collection of crawled URLs, preventing duplicate crawls and loop fetching. The main functions of the URL Manager are as follows:
in the implementation of the URL manager, Python mainly uses memory (set), and relational database (MySQL). For small programs, typically implemented in memory, Python's built-in set () type automatically determines whether elements are duplicated. For larger programs, the database is generally used to implement. 5. Web Downloaderthe Web page downloader in Python mainly uses the Urllib library, which is a Python-brought module. For the URLLIB2 library in the 2.x release, it is integrated into the urllib in the python3.x, in its request and other sub-modules. The Urlopen function in Urllib is used to open the URL and get the URL data. The parameters of the Urlopen function can be URL links, can also make the request object, for a simple Web page, directly using URL string parameters is sufficient, but for complex web pages, with anti-crawler web pages, and then use the Urlopen function, you need to add the HTTP header. For Web pages with a login mechanism, you need to set a cookie. 6. Web Parserthe page parser extracts valuable data and new URLs from the URL data downloaded to the Web page downloader. For data extraction, you can use methods such as regular expressions and beautifulsoup. Regular Expressions use string-based fuzzy matching, which has a good effect on the target data with distinct characteristics, but the generality is not high. BeautifulSoup is a third-party module that is used for structured parsing of URL content. The content of the downloaded Web page is parsed into a DOM tree, which is part of the output of a Web page in the Baidu Encyclopedia that is captured using BeautifulSoup printing.

For the specific use of BeautifulSoup, in a later blog post again. The following code uses Python to crawl other league-related entries in the League of Legends in the Baidu Encyclopedia, and to save these entries in the new Excel. On the code:

From BS4 import beautifulsoupimport reimport xlrd<span style= "FONT-SIZE:18PX;" >import xlwtfrom urllib.request Import URLOPENEXCELFILE=XLWT. Workbook () Sheet=excelfile.add_sheet (' League of Legend ') # #  Baidu Encyclopedia: League of Legends # #html =urlopen ("http://baike.baidu.com/ Subview/3049782/11262116.htm ") Bsobj=beautifulsoup (Html.read ()," Html.parser ") #print (Bsobj.prettify ()) row=0for Node in Bsobj.find ("div", {"Class": "Main-content"}). FindAll ("div", {"Class": "Para"}):    Links=node.findall ("a", Href=re.compile ("^ (/view/) [0-9]+\.htm$")) for    link in Links:        if ' href ' in link.attrs:            print (link.attrs[' href '],link.get_text ())            sheet.write (row,0,link.attrs[' href ')            sheet.write (Row,1,link.get_text ())            Row=row+1excelfile.save (' E:\Project\Python\lol.xls ') </span>    

the part of the output is as follows:

the Excel section is as follows:

Python and web crawler