When using Python to achieve a regular static web crawl, it is often used urllib2 to get the entire HTML page and then search the corresponding keyword verbatim from the HTML file. As shown below:
Copy Code code as follows:
Import Urllib2
Url= "Http://mm.taobao.com/json/request_top_list.htm?type=0&page=1"
Up=urllib2.urlopen (URL) #打开目标页面, storing variable up
Cont=up.read () #从up中读入该HTML文件
key1= ' <a href= ' http ' #设置关键字1
Key2= "Target" #设置关键字2
Pa=cont.find (Key1) #找出关键字1的位置
Pt=cont.find (KEY2,PA) #找出关键字2的位置 (find after Word 1)
URLX=CONT[PA:PT] #得到关键字1与关键字2之间的内容 (that is, the data you want)
Print URLX
However, in the dynamic page, the displayed content is often not through the HTML page rendering, but by calling JS and other ways from the database to get the data, echoing back to the Web page. Take the "Record Information" (http://beian.hndrc.gov.cn/) on the NDRC website as an example, to crawl some of the archival items on this page. For example, "http://beian.hndrc.gov.cn/indexinvestment.jsp?id=162518".
So, open this page in the browser:
All the information is displayed, but if you follow the previous approach:
Copy Code code as follows:
Up=urllib2.urlopen (URL)
Cont=up.read ()
Can not capture the above content.
We look at this page corresponding to the source code:
From the source can be seen, this "record confirmation" belongs to the form of "fill in the blanks", HTML provides a text template, JS based on different IDs to provide different variables, "fill in" to the text template, formed a specific "record confirmation." So simply crawl this HTML, can only get some text templates, and can not get specific content.
So, how do you find those specific content? You can use Chrome's "developer Tools" to find out who is the real content provider.
Open the Chrome browser and press the keyboard F12 to exhale the tool. The following figure:
At this point, select the "Network" tab, enter this page "http://beian.hndrc.gov.cn/indexinvestment.jsp?id=162518" in the address bar, the browser will analyze the whole process of this response, and the Red box files, This is the response, all communication between the browser and the Web backend.
Because to obtain different information of different enterprise, the request that the browser sends to the server must have a parameter related to the current enterprise ID.
So, what are the parameters? The URL has, is "jsp?id=162518", the question mark means to invoke the parameter, followed by the ID number is called parameters. and through the analysis of these files, it is clear that the enterprise information exists in the "indexinvestment.action" file.
However, double hitting opens this file without getting enterprise information, but a bunch of code. Because there is no corresponding parameter for it to indicate the number of information to display. As shown in figure:
So how do you pass the argument to it? Then we still look at the F12 window:
The process of this response is clearly shown in the header column:
For the target URL, a parameter with an ID of 162518 is passed in the post manner.
Let's start by hand. How does js invoke parameters? Yes, as mentioned above: the question mark + variable name + equals + variable corresponds to the number. In other words, when you submit a parameter with ID 162518 to the "http://beian.hndrc.gov.cn/indexinvestment.action" page, you should add the following URL
"? id=162518", ie
"http://beian.hndrc.gov.cn/indexinvestment.action?id=162518".
We paste this URL into the browser to see:
Seems to have some content, but are garbled ah, how to break? A familiar friend may see it at a glance, which is the problem of coding. Because the response back is different from the browser's default encoding. Just click on the chrome upper right-hand corner menu--More tools--code--"Automatic detection". (actually this is UTF-8 's code, and chrome defaults to Chinese Simplified). The following figure:
Well, the real source of information has been dug up, and the rest is to use Python to process the strings on these pages, and then cut, splice, and rearrange the new project record book.
And then use for, while and other loops, batch access to these "record book."
Just as "static Web pages, dynamic Web pages, analog landing, etc., must first analyze, understand the logic, and then write code" said that the programming language is just a tool, the important thing is to solve the problem of ideas. With the idea, and then look for the hands of the tool to solve, OK.