The first write, python crawler image, excel operation .,

The first write, python crawler image, excel operation .,

The first time I wrote a blog, I had already registered a blog store. I had the idea of writing a blog, that is, I had no action. I always forgot, forgot, and finally did nothing, the computer is scattered, looking for something in the east and looking for something in the West. Today, I realized the importance of blog writing.

Recently, I watched the online live course of Tanzhou education, which is quite practical for teachers. We all know that learning a programming language is based on your own notes. Here we will follow the instructor's instructions and copy them for learning.

I. Python crawls watercress pictures.

Tool: python3.6.0; bs4.6.0; xlwt (1.2.0) requires a corresponding version. bs4 has been installed before, but the prompt version does not match during running. Online upgrade: pip install update buautifulsoup4

1. pip list, which can be viewed locally.

　　

1. Crawl the 文 graph and find its address. url = 'HTTP: // www.dbmeinv.com /? Pager_offset = 1 '.

2. view the webpage source code, F12, and network. Find the webpage information captured on the left and find the User-agent. The main purpose is to simulate browser logon and prevent anti-crawler attacks.

Find the element. What we need is the image information in the img tag and the src connection.

 

Write all the code according to the instructor

1 import urllib 2 import urllib. request 3 from bs4 import BeautifulSoup 4 url = 'HTTP: // www.dbmeinv.com /? Pager_offset = 1' 5 x = 0 6 # Get source code 7 # custom function 8 # User-Agent simulates browser access, anti-crawler 9 def crawl (url ): 10 headers = {'user-agent': 'mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) chrome/60.0.3088.3 Safari/537.36 '} 11 req = urllib. request. request (url, headers = headers) # create object 12 page = urllib. request. urlopen (req, timeout = 20) # set timeout 13 contents = page. read () # obtain the source code 14 # print (contents. decode () 15 soup = BeautifulS Oup (contents, 'html. parser ') # html. parser is mainly used to parse webpages. 16 my_girl = soup. find_all ('img ') # locate all img labels 17 #5. get picture 18 for girl in my_girl: # traverse 19 link = girl. get ('src') # get src20 print (link) 21 global x # global variable 22 #6. download urlretrieve23 urllib. request. urlretrieve (link, 'image \ Hangzhou s.jpg '% x) # download, urlretrieve (path to be downloaded) 24 x + = 125 print ('% s downloaded' % x) 26 #7. page 27 for page in range (): # range automatically generates an integer sequence and crawls multi-page images. 28 # page ++ = 129 url = 'HTTP: // www.dbmeinv.com /? Pager_offset = {} '. format (page) #30 # url = 'HTTP: // www.dbmeinv.com /? Pager_offset = % d' % page31 crawl (url) 32 33 print ('image download completed ')

The final running result is saved in the image folder.

 

2. Capture comments and import them to excel. Source code of Zhao Ben's declaration.

Import requestsfrom bs4 import BeautifulSoupimport xlwtdef get_content (url, headers = None, proxy = None): html = requests. get (url, headers = headers ). content return htmldef get_url (html): soup = BeautifulSoup (html, 'html. parser ') shop_url_list = soup. find_all ('div ', class _ = 'tit') # class is a keyword in Python, # list derivation return [I. find ('A') ['href '] for I in shop_url_list] # product details, names, comments, def get_detail_content (html): soup = BeautifulSoup (html, 'html. parser ') price = soup. find ('span ', id = 'avuplicetitle '). text evaluation = soup. find ('span ', id = 'comment _ score '). find_all ('span ', class _ = 'item') # There are multiple find_all. Here there are three # for I in evaluation: # print (I. text) the_star = soup. find ('div ', class _ = 'Brief-info '). find ('span ') ['title'] title = soup. find ('div ', class _ = 'breadcrumb '). find ('span '). text comments = soup. find ('span ', id = 'reviewcount '). text address = soup. find ('span ', itemprop = 'street-address '). text print (u'shop name: '+ title) for I in evaluation: print (I. text) print (price) print (U' comment quantity: '+ comments) print (U' address:' + address. strip () print (U' rating Star: '+ the_star) print (' = ') return (title, evaluation [0]. text, evaluation [1]. text, evaluation [2]. text, price, comments, address, the_star) if _ name __= = '_ main _': items = [] start_url = 'https: // www.dianping.com/search/category/344/10/'base_url = 'https: // www.dianping.com 'headers = {'user-agent': 'mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3088.3 Safari/537.36 ', 'cookies':' _ hc. v = tenant; _ utma = tenant; _ utmc = 1; _ utmz = 1.1496365678.1.1.utmcsr = baidu | utmccn = (organic) | utmcmd = organic; PHOENIX_ID = tenant; s_ViewType = 10; JSESSIONID = E815A43E028078AFA73AF08D9C9E4A15; aburl = 1; cy = 344; cye = changsha; _ mta = pai'} start_html = get_content (start_url) # One page # url_list = get_url (start_html) # multipage url_list = [base_url + url for url in get_url (start_html)] for I in url_list: detail_html = get_content (I, headers = headers) item = get_detail_content (detail_html) items. append (item) # Write excel, txt difference, Excel: xlwg newtablew.'dzdp.xls 'wb = xlwt. workbook (encoding = 'utf-8') ws = wb. add_sheet ('test1') headData = ['Merchant name', 'Taste rating', 'environmental rating', 'service rating', 'per capita price', 'number of comments ', 'address', 'Merchant rating'] for colnum in range (): ws. write (0, colnum, headData [colnum], xlwt. easyxf ('font: bold on ') index = 1 lens = len (items) for j in range (0, lens): for I in range (): ws. write (index, I, items [j] [I]) index + = 1 wb. save (newTable)

I really like locking goddess. I forgot my teacher's explanation and learned a lot. Although I don't know much about it in some places, I believe it will be faster through continuous learning and blog writing.