Application code analysis for crawling web content using Python

Today, we will use this article to introduce two different ways for Python to capture web content. You can use this as a reference object to select an application method that suits your needs in actual application development.

Method 1: Use the urllib2/sgmllib package to list all URLs of the target webpage.

 
 

  
  
import urllib2  
  
  
from sgmllib import SGMLParser   
  
  
class URLLister(SGMLParser):  
  
  
def reset(self):   
  
  
SGMLParser.reset(self)  
  
  
self.urls = []  
  
  
def start_a(self, attrs):   
  
  
href = [v for k, v in attrs if k=='href']   
  
  
if href:  
  
  
self.urls.extend(href)  
  
  
f = urllib2.urlopen("http://www.donews.com/")  
  
  
if f.code == 200:  
  
  
parser = URLLister()  
  
  
parser.feed(f.read())  
  
  
f.close()  
  
  
for url in parser.urls: print url 
 
 

Method 2: Use Python to call IE to capture the url and size of all images on the target Web page Require win32com and pythoncom.

 
 

  
  
import win32com.client, pythoncom  
  
  
import time  
  
  
ie = win32com.client.DispatchEx('InternetExplorer.Application.1')  
  
  
ie.Visible = 1 
  
  
ie.Navigate("http://news.sina.com.cn")  
  
  
while ie.Busy:  
  
  
time.sleep(0.05)  
  
  
doc = ie.Document  
  
  
for i in doc.images:  
  
  
print i.src, i.width, i.height 
 
 

This method can use the Javascript. DHTML support of IE to automatically submit the Form and process Javascript.

The above are two different methods for capturing webpage content in Python.