Python3 crawler (Urllib+beautifulsoup) BeautifulSoup automatically detects coding errors

Version: python3.x

Operating system: Win7

Editor: Pycharm

Crawl page: One of Ctrip's pages (Korea Seoul 6th 5-night semi-self-guided tour • Nonstop + ski Resort or Nami Island + Rakuten World + 1 Day free event-"Ctrip travel")

 #!/usr/bin/env python3#-*-coding:utf-8-*-from urllib.request Import Urlopenfrom Urllib.error Import httperrorfrom BS4 import beautifulsoupdef getcomment (URL): try:html = Urlopen (URL) excep T httperror as E:return none #网页在服务器上不存在 if the server does not exist directly return None Try:soup = BeautifulSoup (Html.read (), "lxml"        ) Comment = Soup.body.find ("ul", {"Class": "Detail_comment_list"}). Find ("Li") except Arithmeticerror as E: return None return commentcomment = Getcomment ("Http://vacations.ctrip.com/grouptravel/p11504202s32.html#ctm_ref=va _hom_s32_prd_p1_l2_2_img ") If comment = = None:print (" comment could not being found ") else:comment 1 = comment.get_text ( ) Print (comment 1)  
 But output garbled 
 Solution: 
 (1) Use third-party libraries requests+beautifulsoup,requests better processing capacity for encoding 
 (2) Direct code detection, found that the page label is UTF-8 written, but in fact, is the GBK encoded code 

 from urllib.request import urlopen 
 Import chardet 
 A = Urlopen (' http://vacations.ctrip.com/grouptravel/ P11504202s32.html#ctm_ref=va_hom_s32_prd_p1_l2_2_img '). Read () 
 B = Chardet.detect (a) 
 print (b) 
 #{' Encoding ': ' GB2312 ', ' confidence ': 0.99} 
 Use BeautifulSoup to confirm Web page encoding 

#!/usr/bin/env Python3
#-*-Coding:utf-8-*-
From urllib.request import Urlopen
From Urllib.error import Httperror
From BS4 import BeautifulSoup
def getcomment (URL):
    Try
        html = urlopen (URL)
    Except Httperror as E:
        Return None    #网页在服务器上不存在 If the server does not exist directly return none
    Try
        Soup = BeautifulSoup (Html.read (), "lxml", from_encoding= ' GBK ')
        Comment = Soup.body.find ("ul", {"Class": "Detail_comment_list"}). Find ("Li")
    Except Arithmeticerror as E:
        Return None
    return comment
Comment = getcomment ("http://vacations.ctrip.com/grouptravel/p11504202s32.html#ctm_ref=va_hom_s32_prd_p1_l2_2_img")
If comment = = None:
    Print ("Commmnt could not being found")
Else
    Comment1 = Comment.get_text ()
    Print (COMMENT1)

Python3 crawler (Urllib+beautifulsoup) BeautifulSoup automatically detects coding errors