Python web crawler (1)--url asked about parameter settings

Source: Internet
Author: User
Tags urlencode sublime text python web crawler

Environment: Python2.7.9/sublime Text 2/chrome

1.url access, call the Urllib library function directly

Import urllib2url='http://www.baidu.com/'= urllib2.urlopen (URL) HTML =response.read ()print html

2. Access with parameters, with the Baidu search function as an example

Use Chrome browser to access the effect, chrome search engine set to Baidu, the address bar input test, the effect is as follows:

You can see the URL of Baidu search is https://www.baidu.com/s?ie=UTF-8&wd=test

Modify code to increase access parameters

#Coding=utf-8ImportUrllibImportUrllib2#URL addressUrl='https://www.baidu.com/s'#Parametersvalues={        'IE':'UTF-8',        'WD':'Test'           }#for parametric encapsulationData=Urllib.urlencode (values)#assemble the full URLreq=Urllib2. Request (Url,data)#access the full URLResponse =Urllib2.urlopen (req) HTML=Response.read ()PrintHtml

Run the code and get the result

The prompt to access the page does not exist, this time need to consider the way to access the problem. Urllib2. Request (Url,data) access mode is post, you need to use Get method to try, change the code to

#Coding=utf-8ImportUrllibImportUrllib2#URL addressUrl='https://www.baidu.com/s'#Parametersvalues={        'IE':'UTF-8',        'WD':'Test'           }#for parametric encapsulationData=Urllib.urlencode (values)#assemble the full URL#Req=urllib2. Request (Url,data)url=url+'?'+Data#access the full URL#response = Urllib2.urlopen (req)Response =urllib2.urlopen (URL) HTML=Response.read ()PrintHtml

Run again to get the result

HTTPS has been redirected and needs to use HTTP

#Coding=utf-8ImportUrllibImportUrllib2#URL address#url= ' https://www.baidu.com/s 'Url='http://www.baidu.com/s'#Parametersvalues={        'IE':'UTF-8',        'WD':'Test'           }#for parametric encapsulationData=Urllib.urlencode (values)#assemble the full URL#Req=urllib2. Request (Url,data)url=url+'?'+Data#access the full URL#response = Urllib2.urlopen (req)Response =urllib2.urlopen (URL) HTML=Response.read ()PrintHtml

Run again to achieve normal access

Python web crawler (1)--url asked about parameter settings

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.