No. 322 section, web crawler, requests request
Requests request, is to use Yhthon requests Module Simulation browser request, return HTML source code
There are two types of simulated browser requests, one is a request that does not require user login or authentication, and a request that requires user login or authentication.
A request that does not require user login or authentication
This is relatively simple, directly using the requests module to send a request to get HTML source code
#!/usr/bin/env python#-*-Coding:utf8-*-ImportRequests#Import the Mock browser request modulehttp=requests.get (url="http://www.iqiyi.com/")#Send HTTP requestHttp.encoding ="Utf-8" #HTTP request EncodingNeir = Http.text#get the HTTP string codePrint(Neir)
Get HTML source code
<!DOCTYPE HTML><HTML><Head><title>New Hot List of drawers-aggregating daily hot, funny and interesting information</title> <MetaCharSet= "Utf-8" /> <Metaname= "keywords"content= "Drawer new hot List, information, satin, picture, public places unsuitable, science and technology, news, moral integrity, funny" /> <Metaname= "description"content= "Drawer new hot list, gather daily funny jokes, popular pictures, interesting news." It will be microblogging, portals, communities, BBS, social networking sites and other massive content aggregation, through user recommendations to generate the hottest list. Look at the new hot List of drawers, a daily hot and interesting information. " /> <Metaname= "Robots"content= "Index,follow" /> <Metaname= "GOOGLEBOT"content= "Index,follow" /> <Metaname= "Author"content= "funny" /> <Metahttp-equiv= "X-ua-compatible"content= "Ie=emulateie8"> <Linktype= "Image/x-icon"href= "/images/chouti.ico"rel= "icon"/> <Linktype= "Image/x-icon"href= "/images/chouti.ico"rel= "Shortcut Icon"/> <Linktype= "Image/x-icon"href= "/images/chouti.ico"rel= "Bookmark"/> <Linktype= "Application/opensearchdescription+xml"href= "Opensearch.xml"title= "Drawer new Hot List"rel= "Search" />
No. 322 section, web crawler, requests request