Selenium working with Dynamic HTML

When some websites use dynamic loading, how do we fix them?

There are two ways: one is to capture content directly in JavaScript, but it is cumbersome to operate, and second, use Python's third-party library to run JavaScript and directly capture what you see in the browser.

This is the use of a Python automated test tool selenium, which can be based on our instructions, let the browser automatically load the page, to obtain the required data. We can install selenium directly with the command line: Pip install Selenium. But there is a problem that selenium itself is not browser-capable and needs to be used in conjunction with third-party browsers, such as PHANTOMJS. However, the selenium began to gradually not support the built-in browser, we can use chrome instead, but need to install chromedriver, and add environment variables.

　　　　

Because I installed the chromedriver version is too low, so there will be an error. But you can see that Google Chrome has been successfully opened and that it is being controlled by automated test software.

　　　

Simulates the loading of a dynamic Web page.

1 #here is the test module for importing Python2 ImportUnitTest3  fromSeleniumImportWebdriver4  fromBs4ImportBeautifulSoup5 6 7 #define a class and inherit the test module8 classDouyuselenium (unittest. TestCase):9     defsetUp (self):TenSelf.driver=Webdriver. PHANTOMJS () One  A         #Self.driver=webdriver. Chrome () -     #specific test methods, to start with test -     defTestdouyu (self): theSelf.driver.get ("Https://www.douyu.com/directory/all") -          whileTrue: -Soup=beautifulsoup (Self.driver.page_source,"lxml") -Titles=soup.find_all ('H3',{'class':'ellipsis'}) +Nums=soup.find_all ('span',{'class':'Dy-num FR'}) -  +              forTitle,numinchZip (titles,nums): A                 Print(U'Number of spectators'+ Num.get_text (). Strip (), U'\ t Room title:'+Title.get_text (). Strip ()) at             #determine if the last page is reached -             ifSelf.driver.page_source.find ('Shark-pager-disable-next')!=-1: -                  Break -            #Analog Click -Self.driver.find_element_by_class_name ('Shark-pager-next'). Click () -  in     defteartest (self): -         Print(U'Loading Complete ... ') to self.driver.quit () +  -  the if __name__=="__main__": *Unittest.main ()

　　　　

　　　　

You can see the number of crawled rooms and the title.

Selenium working with Dynamic HTML