python3 學習（2）：在網站地圖爬蟲時的cannot use a string pattern on a bytes-like object 問題的解決方案

標籤：因此   使用   print   設定   對象   add   ring   lib   use   

python3.6.5 +  pycharm

注意：

一、python3裡的 urllib2 已經沒有了，改為了 urllbi.request，因此，直接匯入 import urllib.request 即可。

二、必須對Regex裡的引用變數進行格式變換  .decode(‘utf-8‘)，否則會報錯說 不能在一個位元組類的對象上使用字串格式。

 

如下代碼所示。

### 一、 網站地圖爬蟲，控制使用者代理程式的設定，可捕獲異常、重試下載並設定使用者代理程式。wswp: web scraping with pythonimport urllib.request  ## -- written by LiSongbodef Rocky_dnload(url,user_agent=‘wswp‘,num_retries = 2):    print(‘Downloading:‘,url)    LiSongbo_he={‘User-agent‘:user_agent}    request = urllib.request.Request(url, headers=LiSongbo_he)    try:  ## -- written by LiSongbohtml = urllib.request.urlopen(request).read()    except urllib.request.URLError as e:  ## -- written by LiSongboprint(‘Download error:‘,e.reason)        html = None        if num_retries > 0:  ## -- written by LiSongboif hasattr(e,‘code‘) and 500 <= e.code < 600:                return Rocky_dnload(url,user_agent,num_retries-1) ## retry 5xx HTTP errorsreturn htmlimport re  ## -- written by LiSongbodef Rocky_crawl_sitemap(url):  ## -- written by LiSongbositemap = Rocky_dnload(url)  ## download the sitmap file    # sitemap = sitemap.decode(‘utf-8‘) ## must add this .links = re.findall(‘<loc>(.*?)</loc>‘, sitemap)  ## extract the sitemap links from flag locfor link in links:  ## download each linkhtml = Rocky_dnload(link)  ## crape html hereRocky_crawl_sitemap(‘http://example.webscraping.com/sitemap.xml‘)

運行結果報錯：

Downloading: http://example.webscraping.com/sitemap.xml
Traceback (most recent call last):
  File "C:/Users/klooa/my_env/book9/test.py", line 25, in <module>
    Rocky_crawl_sitemap(‘http://example.webscraping.com/sitemap.xml‘)
  File "C:/Users/klooa/my_env/book9/test.py", line 22, in Rocky_crawl_sitemap
    links = re.findall(‘<loc>(.*?)</loc>‘, sitemap)  ## extract the sitemap links from flag loc
  File "C:\Users\klooa\AppData\Local\Programs\Python\Python36\lib\re.py", line 222, in findall
    return _compile(pattern, flags).findall(string)
TypeError: cannot use a string pattern on a bytes-like object

必須在 sitemap的下一行加上  

sitemap = sitemap.decode(‘utf-8‘)

修改後的運行結果為：

Downloading: http://example.webscraping.com/sitemap.xml
Downloading: http://example.webscraping.com/places/default/view/Afghanistan-1
Downloading: http://example.webscraping.com/places/default/view/Aland-Islands-2
Downloading: http://example.webscraping.com/places/default/view/Albania-3
Downloading: http://example.webscraping.com/places/default/view/Algeria-4
Downloading: http://example.webscraping.com/places/default/view/American-Samoa-5
Downloading: http://example.webscraping.com/places/default/view/Andorra-6
Downloading: http://example.webscraping.com/places/default/view/Angola-7
Downloading: http://example.webscraping.com/places/default/view/Anguilla-8
Downloading: http://example.webscraping.com/places/default/view/Antarctica-9
Downloading: http://example.webscraping.com/places/default/view/Antigua-and-Barbuda-10
Downloading: http://example.webscraping.com/places/default/view/Argentina-11
Downloading: http://example.webscraping.com/places/default/view/Armenia-12
Download error: TOO MANY REQUESTS
Downloading: http://example.webscraping.com/places/default/view/Aruba-13

Download error: TOO MANY REQUESTS

……

## -- written by LiSongbo

python3 學習（2）：在網站地圖爬蟲時的cannot use a string pattern on a bytes-like object 問題的解決方案