Python simulation Browser implements user response

In recent work, a problem has been encountered in which tasks running on a cluster may not end normally or will not start properly. This causes this batch of running tasks not to end properly, in a pending state, causing the subsequent tasks not to start properly.

The problem has plagued our project for about half a year and has never thought of a good solution. The main reason is that the status of the task can only be seen in the browser, not through the background of the log or database query. In a browser, if we see a task that has not changed in time and state for a long time, we can take the task as a " zombie " task, so that the task can be manually ended (kill).

After the Spring Festival on the Internet to see some of the crawler articles, which mentioned that there is a crawler is to simulate the behavior of the browser (including login, click, etc.) to get the data of the Web page, and then the Web crawl, useful information extraction. So I thought, the problem of our project and the interaction of the browser, there are only a few cases, can be solved in this way " zombie " task. After a week or so of research and a week of intermittent coding, finally to solve the problem, and now solve the main ideas and key technical difficulties to write down, hoping one can deepen their impressions, and secondly can help the people in need. Because the implementation of the task is relatively simple, and the implementation of the process is relatively hasty, code is mainly to achieve a number of functions, no optimization, there is no reference to what coding specifications, design patterns and so on. After a bigger problem, think about it.

Technical points:

(1) Python package:Selenium, with this package, you can interact with the browser, such as open a browser (Chrome,firefox, etc.), login to verify the website (enter the username & password), Click on a specific icon and so on, here are two links to selenium :

Https://www.baidu.com/link?url=tTeJRPOMKX8noXyTa2YPgpaD6vVlGQ2-RVAfwRg4Yvm&wd=&eqid= Acd0879a0043c2e9000000045741cd39

Http://www.cnblogs.com/fnng/archive/2013/05/29/3106515.html

(2)selenium Phantomjs, This is a virtual browser, it can be seen as a browser running in the background, the user cannot see the browser page, but the other functions and ordinary browser basically the same, such as can, click on an icon, Crawl web information, and so on, the reason used to imitate the browser, because our server can not install a normal browser, only run in terminal mode program;

http://phantomjs.org/

(3)XPath, this is my programming the most time-consuming module, the main reasons are several, one is the problem of element positioning, the site is a second refresh, the last second to get the elements of the next second can not be found; second, there are too many similar elements, the hierarchical relationship is too complex, with the general relative path to find, It is possible to find some unwanted elements, so it is time-consuming and laborious to find the element process. Here are two of the introduction of XPath, more practical, especially in the web crawler (I also want to introduce the crawler):

Http://www.cnblogs.com/fdszlzl/archive/2009/06/02/1494836.html

Http://www.ruanyifeng.com/blog/2009/07/xpath_path_expressions.html

The following is the core code, because of the privacy of the project, some sensitive content is replaced with *******. If there is any problem, can give me a message.

"' Command:python tools/webscrap/killjobs.py-url=********-screenshotpath=****** ' from selenium import Webdriverfrom selenium.common.exceptions Import nosuchelementexceptionfrom selenium.webdriver.common.keys Import                    Keysimport reimport timeimport argparseimport sysimport osmailreceiver = [*********,  ]zombie_job_list = {"List1": [], "List2": [], "list3": []}def getmailreceiver (): receiver = ' for recv in mailreceiver:receiver = receiver + recv + ' return receiverdef kill_zombie_jobs (screensho Tpath, URL): # browser = Webdriver. Phantomjs () # Get Local session of Phantomjs browser = Webdriver.    Firefox () # Get Local session of Firefox browser.set_window_size (targeturl) = "http://%s/#JOBS"%url Print "URL:", targeturl job_to_be_kill_indicate = 0 Browser.get (targeturl) # Load page userName = bro Wser.find_elements_by_class_name ("Gwt-textbox") Password = BroWser.find_elements_by_class_name ("Gwt-passwordtextbox") Submitbutton = Browser.find_elements_by_class_name (" Gwt-button ") If Len (userName) = = 0 or len (password) = = 0 or len (submitbutton) = = 0:print" error in open URL :%s "%targeturl browser.quit () return Username[0].send_keys (" ****** ") Password[0].send_keys (" ") Time.sleep (1) Submitbutton[0].click () Time.sleep (4) Sceen_shot_name = Screenshotpath +"/bef Ore_kill_jobs_screen_shot.png "Browser.save_screenshot (sceen_shot_name) # Get the Job_name # Job_name_pattern = "/html/body/div[2]/div[2]/div/div[4]/div/div[3]/div/div[4]/div/div[2]/div/div[2]/div/div/div/div[3]/table[2]/ Tbody/tr[1]/td/fieldset/table/tbody/tr/td/table/tbody/tr[1]/td/table/tbody[1]/tr[1]/td[1]/div "# Jobs_name_ Pattern_0 = "//body/div[2]/div[2]/div/div[4]/div/div[3]/div/div[4]/div/div[2]/div/div[2]/div/div/div/div[3]/ Table[2]/tbody/tr[1]/td/fieldset/table/tbody/tr/td/table/tbody/tr/td/table/tbody/tr/td[@class = ' gjofo-mdoc gjofo-mdad gjofo-mdbd ' or @class = ' gjofo-mdoc Gjofo-mdae gjofo-mdbd '] "jobs_name_ Pattern_0 = "//body/div[2]/div[2]/div/div[4]/div/div[3]/div/div[4]/div/div[2]/div/div[2]/div/div/div/div[3]/ TABLE[2]/TBODY/TR[1]/TD/FIELDSET/TABLE/TBODY/TR/TD/TABLE/TBODY/TR/TD/TABLE/TBODY/TR/TD[1] "Jobs_name_pattern ="/ /body/div[2]/div[2]/div/div[4]/div/div[3]/div/div[4]/div/div[2]/div/div[2]/div/div/div/div[3]/table[2]/tbody/ TR[1]/TD/FIELDSET/TABLE/TBODY/TR/TD/TABLE/TBODY/TR/TD/TABLE/TBODY/TR[ORDER]/TD[1] "Jobs_kill_pattern ="//body/ div[2]/div[2]/div/div[4]/div/div[3]/div/div[4]/div/div[2]/div/div[2]/div/div/div/div[3]/table[2]/tbody/tr[1]/ TD/FIELDSET/TABLE/TBODY/TR/TD/TABLE/TBODY/TR/TD/TABLE/TBODY/TR[ORDER]/TD[3] "Jobs_duration_pattern ="//body/div[ 2]/div[2]/div/div[4]/div/div[3]/div/div[4]/div/div[2]/div/div[2]/div/div/div/div[3]/table[2]/tbody/tr[1]/td/   FIELDSET/TABLE/TBODY/TR/TD/TABLE/TBODY/TR/TD/TABLE/TBODY/TR[ORDER]/TD[5] "For I in range (1, 4):     tmp_list = "List" +str (i) job_name_elements_list = Browser.find_elements_by_xpath (jobs_name_pattern_0) J  Ob_length = Len (job_name_elements_list) for index in range (1, job_length+1): Job_name_pattern = Jobs_name_pattern.replace ("Order", str (index)) Job_duration_pattern = Jobs_duration_pattern.replace ("Order", s TR (index)) job_name = get_element_name (browser, job_name_pattern) Job_duration_time = Durationtime (g                Et_element_name (Browser, job_duration_pattern)) If Len (job_name) > Ten and Durationtime = = 0: Zombie_job_list[tmp_list].append (job_name) time.sleep print "ZOMBIE JOB LIST:", Zombie_ Job_list job_name_elements_list = Browser.find_elements_by_xpath (jobs_name_pattern_0) job_length = Len (job_name_elem ents_list) for index in range (1, job_length+1): # print "index:", index Job_name_pattern = Jobs_name_pat Tern.replace ("Order", StR (index)) Job_kill_pattern = Jobs_kill_pattern.replace ("Order", str (index)) job_name = Get_element_name (brow   SER, Job_name_pattern) if job_name in zombie_job_list["List1"] and job_name in zombie_job_list["List2"] and Job_name in zombie_job_list["List3"]: job_to_be_kill_indicate = 1 print "This JOB was should be Kil LED: ", job_name kill_button_element = Browser.find_element_by_xpath (job_kill_pattern) kill_button_e Lement.click () time.sleep (1) confirm_kill_button_pattern = "//tbody/tr/td[1]/button[@class = ' Gwt-butt On '] "confirm_kill_button_element = Browser.find_element_by_xpath (confirm_kill_button_pattern) confi Rm_kill_button_element.click () Time.sleep (2) time.sleep (2) sceen_shot_name = Screenshotpath + "/A Fter_kill_jobs_screen_shot.png "Browser.save_screenshot (sceen_shot_name) browser.quit () return Job_to_be_kill_ind Icate def get_element_name (Browser, element_pattern): Element_name = "" Try:element = Browser.find_element_by_xpath (element_patte        RN) Element_name = Element.text except Exception, e:print "element not exist no more!!!!!"     Element_name = "" Return element_namedef Durationtime (TIMESTR): If Timestr is None or timestr = = "": Return 0 If Re.match (r "\d{2}:\d{2}:\d{2}", Timestr) is none:return 0 timesec = Int (timestr[0:2]) * 3600 + int (timest R[3:5]) * + int (Timestr[6:8]) return timesecdef send_kill_jobs_mail (mailer, Screenshotpath, URL, indicator): # JO BS screen before and after kill Mailtitle = "Jobs_on_%s_hanging"%url screenShotFile1 = Screenshotpath + "/before_k Ill_jobs_screen_shot.png "ScreenShotFile2 = Screenshotpath +"/after_kill_jobs_screen_shot.png "LogFile = screenshot Path + "/nodes_hanging.log" command = ' mail-a ' + screenShotFile1 + '-a ' + ScreenShotFile2 + '-s ' + Mailtitle + ma   Iler + ' < ' + LogFile Print "command:", Command Os.system (command) return 0def Monitor (): # kill exist PHANTOMJS command = "Killa ll Phantomjs "Print" Kill all existing Phantomjs: ", Command Os.system (command) parser = Argparse. Argumentparser () parser.add_argument ('-url ', action= ' store ', dest= ' url ', help= ' data url ', required=true) parser.add_  Argument ('-screenshotpath ', action= ' store ', dest= ' Screenshotpath ', help= ' the screen shot path ', required=true) results = Parser.parse_args () print ' datarush url = ', results.url url = results.url print ' screen Shot Path = ', Resul Ts.screenshotpath Screenshotpath = Results.screenshotpath Mailer = Getmailreceiver () print "Start:monitor DataRu SH starting ... "job_killed_inicate = Kill_zombie_jobs (screenshotpath, URL) if Job_ki, ....... ...". ".".. "        Lled_inicate = = 1:print "Zombie jobs has been killed!!!!!!!" Send_kill_jobs_mail (mailer, Screenshotpath, URL, 1) else:pass print"End:monitor finished .... ..." if __name__ = = ' __main__ ': Monitor (), ............ ... "if" = "="            

　　

Python simulation Browser implements user response