Selenium crawl news into Word cloud (and Selenium XPath lookup method)

The first is to use QQ space to say the word cloud, but QQ space needs to use cookies and other encrypted things, back to the next to engage in the news bar.

directly on the code.

#-*-coding:utf-8-*- fromSeleniumImportWebdriverImportWordcloud#Word cloud makerImportJieba#powerful Chinese word thesaurus fromScipy.miscImportImread#Reading PicturesImportTimeurl='https://news.nuist.edu.cn/main.htm'News=Webdriver. Chrome () news.get (URL)defgetnews (): Raw=""News.find_element_by_link_text ('+ More'). Click () forNinchRange (1,60): News.find_element_by_class_name ('Pagenum'). Send_keys (str (n)) News.find_element_by_class_name ('Pagingjump'). Click ()#page Break ReadBoxes=news.find_elements_by_xpath ('//span[@class = "Col_title"]/a[@target = "_blank"]') XPath reads news headlines forBoxinchBoxes:raw+=Box.textreturnRawdefDelit (RAW):#remove Chinese characters, and later find it seems like one more stepR =',. , "" "":; () "'     forIinchr:ifIinchRaw:raw=raw.replace (I,"')    returnRawdefMakecloud (Raw): words=' '. Join (Jieba.lcut (raw)) Mask=imread ('apicture. JPG')#Background ImageCloud=wordcloud. Wordcloud (font_path='MSYH.TTC', width=1080,height=720,mask=mask,max_words=400) cloud.generate (words) cloud.to_file ('Pic.png')defMain (): Raw=getnews () raw1=Delit (Raw) Makecloud (RAW1) News.close () main ()

Sometimes we in the HTML source difficult to get the elements we want to id,class only the special tag time to do? The XPath method is used to find it.

I hate to give you a bunch of definitions, so let's go straight to the example:

I want to extract this paragraph of text, but can not use find_element_by_id, also can not use classname, even because the link is changing so even linktext also not, this time can only bite the bullet to learn XPath

First XPath provides the//method is arbitrary location lookup, relative/root of a long list of absolute paths, here our elements can be entered as:

spam[@class = "Col_title"]/a[@target = "_blank"]

What do you mean, the picture above we can see a bunch of hierarchical relationships we don't care, we just look for news tags A and parent, a has the target property value of _blank, then a[@target = ' _blank ') is the tag name A, the property is called Target and the value is _blank element

But we go down some irrelevant information is also this value, then this tag with his father is not the same, plus the parent/, the label is span, the attribute class value is "Col_title", it is always different.

Of course, no longer called the grandfather head of the headquarters. This method is HTML5, of course selenium also added this function.

For the use of Wordcloud and Jieba, see Https://www.icourse163.org/learn/BIT-268001?tid=1002788003#/learn/content?type=detail &id=1004116227

All right, so much for showing the results.

Selenium crawl news into Word cloud (and Selenium XPath lookup method)