Python cralwer (crawler) experience, pythoncralwer
Recently I made a small crawler using python, which can automatically sort out some website content and push it to the local file to make a small summary.
The main lib is urllib and beautifulsoup.
Urllib and urllib2 are convenient web page extraction libraries. The core is to send various custom url requests and then return the webpage content. The simplest function is to determine whether a webpage exists:
def isUrlExists(url): req = urllib2.Request(url, headers=headers) try: urllib2.urlopen(req) except: return 0; return 1;
Headers can be customized or left empty. The main purpose of customization is to simulate the header of a general browser and bypass some websites to block crawlers.
If you want to get the website content and get the content that returns an exception, you can do this:
def fetchLink(url): req = urllib2.Request(url, headers=headers) try: response = urllib2.urlopen(req) except urllib2.URLError, e: print 'Got Url Error while retrieving: ', url, ' and the exception is: ', e.reason except urllib2.HTTPError, e: print 'Got Http Error while retrieving: ', url, ' with reponse code: ', e.getcode(), ' and exception: ', e.reason else: htmlContent = response.read() return htmlContent
The above Code directly returns html.
BeautifulSoup (http://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a concise library of html analysis tools. After obtaining html, you can directly use the regular expression that comes with python to obtain the desired information, but it is a little tedious. Beautifulshop analyzes html directly using json-like methods to form a standard tree structure and can directly obtain elements. In addition, element search is supported.
content = bs4.BeautifulSoup(content,from_encoding='GB18030') posts = content.find_all(class_='post-content') for post in posts: postText = post.find(class_='entry-title').get_text()
In this example, content is first converted to a bs4 object, then all the blocks of class = post-content are found, and the text of class = entry-title is obtained. Note: You can select encoding when parse is in the first line. Here we use simplified Chinese.
The above are the methods for obtaining html text content. If there is an image, the above Code will directly generate an image and connect it to the original image location. You can use the urlretrieve method to download any files. I will not talk about it here.