Python cralwer (crawler) experience, pythoncralwer

Source: Internet
Author: User

Python cralwer (crawler) experience, pythoncralwer

Recently I made a small crawler using python, which can automatically sort out some website content and push it to the local file to make a small summary.


The main lib is urllib and beautifulsoup.


Urllib and urllib2 are convenient web page extraction libraries. The core is to send various custom url requests and then return the webpage content. The simplest function is to determine whether a webpage exists:

def isUrlExists(url):  req = urllib2.Request(url, headers=headers)  try:    urllib2.urlopen(req)  except:    return 0;  return 1;

Headers can be customized or left empty. The main purpose of customization is to simulate the header of a general browser and bypass some websites to block crawlers.

If you want to get the website content and get the content that returns an exception, you can do this:

def fetchLink(url):  req = urllib2.Request(url, headers=headers)  try:    response = urllib2.urlopen(req)  except urllib2.URLError, e:    print 'Got Url Error while retrieving: ', url, ' and the exception is: ', e.reason  except urllib2.HTTPError, e:    print 'Got Http Error while retrieving: ', url,  ' with reponse code: ', e.getcode(), ' and exception: ', e.reason  else:    htmlContent = response.read()    return htmlContent


The above Code directly returns html.

BeautifulSoup (http://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a concise library of html analysis tools. After obtaining html, you can directly use the regular expression that comes with python to obtain the desired information, but it is a little tedious. Beautifulshop analyzes html directly using json-like methods to form a standard tree structure and can directly obtain elements. In addition, element search is supported.


  content = bs4.BeautifulSoup(content,from_encoding='GB18030')  posts = content.find_all(class_='post-content')  for post in posts:    postText = post.find(class_='entry-title').get_text()

In this example, content is first converted to a bs4 object, then all the blocks of class = post-content are found, and the text of class = entry-title is obtained. Note: You can select encoding when parse is in the first line. Here we use simplified Chinese.


The above are the methods for obtaining html text content. If there is an image, the above Code will directly generate an image and connect it to the original image location. You can use the urlretrieve method to download any files. I will not talk about it here.


Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.