Python crawls web content and python Crawlers

Python crawls web content and python Crawlers

Recently, I want to capture data on the Internet for research. Just a bit of python, let's look at a simple implementation method.

For example, I want to capture Obama's weekly speech.

Is there a one-step approach that can be quickly implemented using a powerful language such as python.

First, let's look at the source code of this webpage.

We can find that the information we need is in such a small url.

More specifically, we need to traverse every.

Okay. Start writing code.

First open this directory page and save it in content

import sys,urlliburl="http://www.putclub.com/html/radio/VOA/presidentspeech/index.html"wp = urllib.urlopen(url)print "start download..."content = wp.read()

Next we will extract the content of each speech.

The specific idea is to search for the content between each "href =" and "target" after "center_box. See the source code of the webpage.

The result is the url of each article, and www.putclub.com is the url of each article.

print content.count("center_box")index =  content.find("center_box")content=content[content.find("center_box")+1:]content=content[content.find("href=")+7:content.find("target")-2]filename = contenturl ="http://www.putclub.com/"+contentprint contentprint urlwp = urllib.urlopen(url)print "start download..."content = wp.read()

With the url of the article content, you can filter the content in the same way.

#print contentprint content.count("<div class=\"content\"")#content = content[content.find("<div class=\"content\""):]content = content[content.find("<!--info end------->"):]content = content[:content.find("<div class=\"dede_pages\"")-1]filename = filename[filename.find("presidentspeech")+len("presidentspeech/"):]

Finally, save and print

filename = filename.replace('/',"-",filename.count("/"))fp = open(filename,"w+")fp.write(content)fp.close()print content

OK, all done! Save it as the. pyw file. In the future, you only need to double-click it to directly Save the content of obama's weekly speech ~

How does one obtain content on a webpage in Python? I want to use python to read different words and the number of times each word appears on the webpage.

Hi!
First, the content displayed by the browser to the user is completely based on the html source code. Therefore, all the content displayed by the browser you want to obtain is the content that exists in the html file.
The words on the statistics page must be read from the html source file.
You can use the urllib2 Library and the re library for matching and search. The Code is as follows:
Import urllib2import reword = 'urlfile = urllib2.urlopen ('content of the URL you want to open, remember to bring the Protocol prefix, such as http ') # Open a webpage in the format of html = urlfile. read () # read htmlwordList = re from a webpage file. findall (re. compile (word), html) # regular findall search for all the words print len (wordList) # The number of re modules is hard to remember. If you haven't tried it, try it. If not, try again.


How can I extract the main content of a captured webpage using python?

Use beautiful soup
There are too many specific code. Check the link.
Reference: www.crummy.com/...h.html