How to crawl content between two spans in Python

Python used for data processing is quite good, if you want to make a crawler, Python is a good choice, it has a lot of well-written class package, as long as the call to complete a lot of complex functions, this article all the functions are based on BeautifulSoup this package.

1 Pyhton Get the content of the Web page (i.e. source code)

 

page = Urllib2.urlopen (URL)   contents = Page.read ()   #获得了整个网页的内容也就是源代码  Print (contents)  

URL represents the URL, contents represents the source code corresponding to the URL, Urllib2 is needed to use the package, the above three lines of code will be able to obtain the entire source code of the Web page

2 Get the content you want in the Web page (first to get the source code of the Web page, then analyze the source code, find the corresponding tag, and then extract the contents of the tag)

2.1 In the case of Watercress film rankings

URL is http://movie.douban.com/top250?format=text, enter the URL after the following diagram appears

Now I need to get the names of all movies on the current page, rating, number of ratings, links

By painting The red circle is what I want to get the content, draw the Blue Line for the corresponding label, so the analysis is finished, now is the code implementation, Python provides a lot of ways to get the desired content, I use BeautifulSoup to achieve, very simple

 

#coding: Utf-8 '  @author: Jsjxy '  import urllib2   import re from   BS4 import BeautifulSoup  from Distutils.filelist import findall        page = Urllib2.urlopen (' http://movie.douban.com/top250?format=text ')   Contents = Page.read ()    #print (contents)  soup = beautifulsoup (contents, "Html.parser")  print (" Watercress movie TOP250 "+" \ n "+" film name              rating       rating number     link ")    for tag in Soup.find_all (' div ', class_= ' info '):       # Print Tag      m_name = Tag.find (' span ', class_= ' title '). Get_text ()            M_rating_score = float (tag.find (' span ', class_= ' Rating_num '). Get_text ())              m_people = tag.find (' div ', class_= ' star ')      M_span = M_people.findall (' span ')      m_peoplecount = m_span[3].contents[0]      m_url=tag.find (' a '). Get (' href ')      print (m_name+ "        "  +  str (m_rating_score)   + "           " + M_peoplecount + "    + m_url)   

  

Console output, you can also write to the file

The first three lines of code get the entire page source code, and then began to use BeautifulSoup for label analysis, the Find_all method is to find all the contents of this tag, and then continue to look in this tab, if the label has a special attribute declaration can be found in one step, If there is no special attribute declaration, just like the number of people in this figure before the label has only one ' span ' then all the span tags are found, in order from the selected corresponding, in this figure is the third, so this method can find the content of a particular row or column. Code is relatively simple, it is easy to achieve, if there is something wrong, also please point out that we learn together.

Source code Address: http://download.csdn.net/detail/danielntz/9577390

Turn from: 51861168

How to crawl content between two spans in Python