Using python2.7 to capture the Douban film top250

Using python2.7 to capture the top250 Douban Movie, you can conveniently view many contents including HTML.

Open the top P250 page of Douban film rankings and find 25 movies on each page. There are 10 pages in total. each page url has the following features:

Http://movie.douban.com/top250? Start = 0

Http://movie.douban.com/top250? Start = 25

Http://movie.douban.com/top250? Start = 50

Http://movie.douban.com/top250? Start = 75

......

Therefore, you only need to use the loop to process the following 0, 25,... 225.

On the webpage, click the Chinese name of any movie and right-click "View elements" to view the HTML source code:

You can find that the movie name is in, and the English name is also in.

You can use the regular expression (. *) to match the Chinese name and English name of a movie. However, you only need to obtain the Chinese name. Therefore, you need to filter the English name.

The filter method can be implemented using the find (str, pos_start, pos_end) function to remove the features unique to English names: ''and '/'. for details, see the code.

3. code implementation

The code here is relatively simple, so you don't need to define a function.

#! /Usr/bin/python #-*-coding: UTF-8-*-# import requests, sys, refrom bs4 import BeautifulSoupreload (sys) sys. setdefaultencoding ('utf-8') print' is capturing data from the Douban film Top250 ...... 'for page in range (10): url = 'https: // movie.douban.com/top250? Start = '+ str (page-1) * 25) print' --------------------------- crawling the '+ str (page + 1) +' page ...... -------------------------------- 'HTML = requests. get (url) html. raise_for_status () try: soup = BeautifulSoup (html. text, 'HTML. parser ') soup = str (soup) # use a regular expression to convert the webpage text into a string title = re. compile (r '(. *) ') names = re. findall (title, soup) for name in names: if name. find ('') =-1 and name. find ('/') =-1: # Remove the English name (the English name features include ''and '/') pr Int name # Creation name, scoring failed t Exception as e: print eprint 'crawling is complete! '