Python 爬蟲執行個體(12)—— python selenium 爬蟲

來源:互聯網
上載者:User

標籤:www   htm   data   try   range   parser   lbs   exec   amp   


# coding:utf-8from common.contest import *def spider():

  url = "http://www.salamoyua.com/es/subasta.aspx?origen=subastas&subasta=79"
  
   chromedriver = ‘C:/Users/xuchunlin/AppData/Local/Google/Chrome/Application/chromedriver.exe‘    chome_options = webdriver.ChromeOptions()   
  #使用代理  # proxies = r.get(‘4‘) # chome_options.add_argument((‘--proxy-server=http://‘ + proxies)) os.environ["webdriver.chrome.driver"] = chromedriver driver = webdriver.Chrome(chromedriver, chrome_options=chome_options) for i in range(1,100): print "正在爬取第" + str(i) + "頁的資料" if i ==1: # 請求url driver.get(session_url) result = driver.page_source else: try: # 將頁面捲軸拖到底部 js = "var q=document.documentElement.scrollTop=10000" driver.execute_script(js) driver.find_element_by_id(‘ctl00_phContenidos_lbSiguiente‘).click() # 得到爬取頁面的結果 result = driver.page_source time.sleep(3) except: result = "" soup = BeautifulSoup(result, ‘html.parser‘) result_div = soup.find_all(‘figure‘, attrs={"class": "Lotes fade"}) # print len(result_div) for i in result_div:
      
         result_replace = replace(i)                print result_replace                item_url = re.findall(‘<figure class="Lotes fade"><a href="(.*?)" id=‘,result_replace)[0]                item_url = "http://www.salamoyua.com/es/" + item_url.replace(‘‘,‘‘)                item_imgurl = re.findall(‘<img id=".*?" src="..(.*?)" style="border-width:0px‘, result_replace)[0]                item_imgurl = "http://www.salamoyua.com" + item_imgurl.replace(‘‘, ‘‘)                if "Remate" not in result_replace:                    sold_price = ""                else:                    sold_price = re.findall(‘<p><strong>Remate:(.*?)</strong></p></figcaption>‘, result_replace)[0]                    sold_price = sold_price.replace(‘ ‘,‘‘)                try:                    item_lotnum = re.findall(‘title="Lote vendido"><span id=".*?">(.*?)</span>‘, result_replace)[0]                    item_lotnum = item_lotnum.replace(‘Lote‘,‘‘).replace(‘ ‘,‘‘)                except:                    item_lotnum = re.findall(‘<span id=".*?">(.*?)</span></header>‘,result_replace)[0]                    item_lotnum = item_lotnum.replace(‘Lote‘, ‘‘).replace(‘ ‘, ‘‘)                print item_url                print item_lotnum                print item_imgurl                print sold_price              

 

  

spider()

 

Python 爬蟲執行個體(12)—— python selenium 爬蟲

聯繫我們

該頁面正文內容均來源於網絡整理,並不代表阿里雲官方的觀點,該頁面所提到的產品和服務也與阿里云無關,如果該頁面內容對您造成了困擾,歡迎寫郵件給我們,收到郵件我們將在5個工作日內處理。

如果您發現本社區中有涉嫌抄襲的內容,歡迎發送郵件至: info-contact@alibabacloud.com 進行舉報並提供相關證據,工作人員會在 5 個工作天內聯絡您,一經查實,本站將立刻刪除涉嫌侵權內容。

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.