Python crawler tool PyQuery module Installation Guide, pythonpyquery

Source: Internet
Author: User

Python crawler tool PyQuery module Installation Guide, pythonpyquery

Windows installation:
: Https://pypi.python.org/pypi/pyquery/#downloads
Download and install:

C:\Python27>easy_install E:\python\pyquery-1.2.4.zip 

You can also install it online:

C:\Python27>easy_install pyquery 

Pyquery is a python library similar to jquery. It can use a syntax like jquery to extract any data on a webpage, this is a good third-party library for html webpage data extraction and mining. Next, let's take a look at the usage of pyquery.

Extract information from html strings

#! /Usr/bin/python #-*-coding: UTF-8-*-from pyquery import PyQuery as pqhtml = ''' 

The code segment above provides common pyquery operations. We first defined an html code, and then used a series of pyquery methods to operate the html code, mainly to obtain specific elements and text. Of course, pyquery can not only obtain elements, but also set element attributes and add elements. Given that the most common method we use in the code above, other methods will not be introduced here.

Extract information from a url or local html file

Of course, pyquery can not only parse html strings like above, but also like this:

d = pq(url='http://www.baidu.com/')

We can directly load a URL, which is no different from the above operation method. By default, the urllib module is used for http requests. However, if requests is installed in your system, requests is used for http requests, this means that you can use any requests parameter, for example:

pq('http://www.baidu.com/', headers={'user-agent': 'pyquery'})

Or, if you already have the corresponding html file in your local machine, you can also do this:

d = pq(filename=path_to_html_file)

The preceding method directly specifies the local html file, and the operation method is still the same as above.
As you can see, pyquery provides us with ample convenience to select any element, just like jquery.

Using pyquery to capture the Douban film top250

After reading the pyquery syntax, let's look at an example to capture the top250 Douban movie.
Because Douban anti-crawler is very powerful, it won't be able to catch it again after several operations. I had to download the page with requests and directly use the pyquery analysis PAGE method to extract information:

from pyquery import PyQuery as pqimport requestshead_req = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36', 'Referer':'https://movie.douban.com/top250?start=0',}r=requests.get("https://movie.douban.com/top250?start=0",headers=head_req)with open("1.html","wb") as html: html.write(r.content)d=pq(filename="1.html")# print d('ol').find('li').html()for data in d('ol').items('li'): print data.find('.hd').find('.title').eq(0).text() print data.find('.star').find('.rating_num').text() print data.find('.quote').find('.inq').text() print

Run the following command to check the result:

Xiao shenke's salvation 9.6 hopes to make people free. This killer is not too cold. The story that xiao shu and Xiao luoli have to tell. Arganzheng uploads 9.4 modern American history. Overlord farewell to Ji 9.4. Beautiful Life: 9.5 of the most beautiful lies. Qianyi qianxun 9.2 best Hayao Miyazaki, the best long Shi let. The Singler list saves one person, that is, the whole world. Every one of the 9.2 piano masters at sea must take a firm path, even if it is broken. Robot story 9.3 xiaowali, big life. Dreamspace 9.2 Nolan gave us a dream that we could not steal. The loss of titanic 9.1 is eternal. Three dumbys, the handsome version of Bollywood 9.1, and the high EQ version, thank you. In spring 9.2, the sound of a child was the closest to God. 9.2 The Story of the dog's eight male will never forget your loved ones. Chinamoocs.com 9.1 has a chinamoocs.com in the hearts of all people, so that childhood will never disappear. The secret of the westward journey is the love of 9.1 of your life. The Godfather should never hate your opponent. This will make you irrational. The beautiful man 9.2 Tomorrow is another day. Heaven cinema 9.1 those kiss scenes, those youth, all in the darkness of the cinema were wiped out with tears. When happiness knocks on the door, there are 8.9 civilian inspirational films. The 9.0 evil and mediocre Fight Club lie in the same parent and confront each other at a specific time. Good morning, good afternoon, good night if you can never see you again in the world of Chu men 9.0. An elegant comedy with 9.1 warmth. Lord of the Rings 3: the final chapter of the invincible 9.1 epic. Rome holiday 8.9 love only one day.

Of course, this is only 25 items on the first page. We already know that the url of the Douban movie top250 is

Https://movie.douban.com/top250? Start = 0
The start parameter ranges from 0 to 25 each time

Https://movie.douban.com/top250? Start = 225
Therefore, you can write a loop to capture them all.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.