I currently have a job that needs to obtain data from the Sina news website, including the title, body, and participants. I want to get 820, but it is generated dynamically. The url of the news is a red-headed header file from Meixian County, Shaanxi Province: Officials selling cement to buy a house are included in the assessment & lt; ahref # J_Comment_Wraprel. I have a saying that ac has a job and needs to get data from Sina news website, including the title, body, and number of participants.
I want to get 820, but it is generated dynamically. The url of the news is a red-headed file from Meixian County, Shaanxi Province: officials who sell cement to buy a house are included in the assessment
I have something to say
This code generates 820 participants. What tools are used to obtain them? I want to use a simple code or module that can complete the above work? Thank you for your reply.
In this case, it is generally an asynchronous request for json or jsonp. You can directly monitor the network:
Take chrome as an example.
1. Right-click the page-review element-Network, switch to the network panel, and refresh the page.
Then, the communication between the browser and the web backend is recorded.
Exclude images and css.
Number of comments for the current news,
The request sent by the browser to the server must contain a parameter related to the current news id.(Of course, in theory, id transfer is also implemented through referer, but after all, it is amazing and will not be considered ).
So
1. If method is GET, there must be a special string in Name to identify which news comment is to be requested.
2. If the method is POST, there will be a parameter in the POST parameter that can identify the current news:
This is the action for the public platform to obtain chat information, and fromfakeid is the id of the chat peer. This is the action for the public platform to obtain chat information, and fromfakeid is the id of the chat peer.
If you want to use the POST method to obtain the number of comments of a news item, a parameter is also provided here to identify the news. (For example, "175827642839 ").
After troubleshooting, the image css is ignored, so it is easy to find
Http://comment5.news.sina.com.cn/page/info? Format = js & channel = gn & newsid = 1-1-27642839 & group = 0 & compress = 1 & ie = gbk & oe = gbk & page = 1 & page_size = 100 & jsvar = requestId_19606520
Can be reduced to http://comment5.news.sina.com.cn/page/info? Format = js & channel = gn & newsid = 1-1-27642839
Among them, newsid parameters 1-1-27642839 and 175827642839 have obvious relationships.
This is a javascript code, which contains the following sentence: "count": {"qreply": 706, "total": 823, "show": 95}. Here is what you are looking.
You can directly extract the total from the string, or remove the "var data =" in the js header ". Get a json string and parse it into an object to get total.
Find "1-1-27642839" in the source code of the news page and find "newsid: '1-1-27642839 ',"
The analysis process is basically over.
Then, you can:
1. Analyze the News Page code and obtain newsid.
2. Request http://comment5.news.sina.com.cn/page/info? Format = js & newsid =
"Newsid"
3. parse the obtained js
import seleniumfrom selenium import webdriverfrom selenium.common.exceptions import NoSuchElementExceptionfrom selenium.webdriver.common.keys import Keysimport timebrowser = webdriver.Firefox() # Get local session of firefoxbrowser.get("http://news.sina.com.cn/c/2013-07-11/175827642839.shtml ") # Load pagetime.sleep(5) # Let the page loadtry: element = browser.find_element_by_xpath("//span[contains(@class,'f_red')]") # get element on page print element.text # get element textexcept NoSuchElementException: assert 0, "can't find f_red"browser.close()
>>> from selenium import webdriver>>> c = webdriver.Chrome()>>> c.get('http://news.sina.com.cn/c/2013-07-11/175827642839.shtml')>>> comment = c.find_element_by_id('media_comment')>>> count = comment.find_element_by_class_name('f_red')>>> count.textu'823'
I used the python class library webkitgtk to parse the webpage, extract the html structure, and obtain data. However, the installation and use of webkitgtk are a little painful and requires a specific version to be installed to obtain data. Hello, I have read your master's thesis, I am also very interested in this paper to recommend this topic, but I did not understand the TV-IDF algorithm you improved on the basis of LDA, using the 4-2 formula does not calculate the value of your TV-IPF in the 4-3 table, can you explain or show me the specific calculation process? Thanks for using the ghost. py module to simulate a browser to open a webpage. The obtained webpage is dynamically generated.
