Python. beatifulsoup, pythonsoup

Python. beatifulsoup, pythonsoup

First, I want to try the python crawler function. So I installed beautifulsoup with easy_install. below is the demo code I wrote. Let's take a look.

# Coding = utf-8import urllib2from BeautifulSoup import BeautifulSoup as bsurl_addr = 'HTTP: // your save_to_file (content, fileName): with open (fileName, 'W + ') as f: if not f: return Falsef. write (content) f. close () def parse_content (content): soup = bs (content) tds = soup. findAll ('td ') for I in tds: print I. textdef is_unicode (c): if isinstance (c, unicode): return Truereturn Falsedef is_asii (c): if isinstance (c, str ): return Truereturn Falseif _ name _ = '_ main _': content = urllib2.urlopen (url_addr ). read () # str in any format is converted to the intermediate encoding unicode # Call decode to convert it to the unicode encoding [according to its own encoding] # Then call encode to convert unicode to the specified encoding # convert content [encoding for gb2312] to unicode encoding c = content. decode ('gb2312') print ('C is unicode: ', is_unicode (c) # print (c) save_to_file (c. encode ('gb2312'), 'd:/content.html ') parse_content (content)